Lessons of Scale at Facebook

Here is the link for one hour video.

From the 2010 USENIX Annual Technical Conference Keynote Address, Bobby Johnson, Director of Engineering, Facebook, Inc. discusses how in just over six years Facebook has grown from an idea in a dorm room to one of the most visited sites on the Internet. This explosive growth has created enormous technical challenges. He talks about some specific technical challenges Facebook has faced and the general principles they employ when addressing problems of scale. He also discusses how they structure their engineering process and culture to stay on top of unceasing growth while still moving fast to build new products.

20:00

Memcached vs my SQL

graph data - just right

Set operations
Append
Remove
Intersect

Client performance


34:00
Control and responsibility go together
Centralized - a variety of model - things can be done faster, gate keeper, distributed out.
Hybrid model - one person knows production team
Make things a lot of faster.

38:00
Lessons learned

Federate everything
Keep failures contained
Dark launch new features
New code is slow
Give developers room to try things
Nobody's job is to say no

Questions
40:00
1. Engineer culture?
Feel responsible but no control

2. TCP, application level, socket , Linux, what change?

Server side, custom kernel?

3. Memcached, all data are in memcached?

Photo id, ...

video, in disk

Web server -> data

4. New code is slow?

PHP profile -> work load

5. Data consistency?

6. Roll out new feature? One week to find out about memory leak, one instance mentioned

Answer:

7. Long term data storage?