Facebook trapped in MySQL ‘fate worse than death’

Giga OM ^ | July 7, 2011 | Derrick Harris

[TenthAmendmentChampion]

According to database pioneer Michael Stonebraker, Facebook is operating a huge, complex MySQL implementation equivalent to “a fate worse than death,” and the only way out is “bite the bullet and rewrite everything.”

Not that it’s necessarily Facebook’s fault, though. Stonebraker says the social network’s predicament is all too common among web startups that start small and grow to epic proportions.

During an interview this week, Stonebraker explained to me that Facebook has split its MySQL database into 4,000 shards in order to handle the site’s massive data volume, and is running 9,000 instances of memcached in order to keep up with the number of transactions the database must serve. I’m checking with Facebook to verify the accuracy of those numbers, but Facebook’s history with MySQL is no mystery.

The oft-quoted statistic from 2008 is that the site had 1,800 servers dedicated to MySQL and 805 servers dedicated to memcached, although multiple MySQL shards and memcached instances can run on a single server. Facebook even maintains a MySQL at Facebook page dedicated to updating readers on the progress of its extensive work to make the database scale along with the site...

[NVDave]

Sounds like someone finally cottoned onto what Honeywell did on their drive systems under CP-6.

[antiRepublicrat]

I don’t really like RAID 6 though. For a small installation of five drives, you might as well add one more and get RAID 10. for larger installations the big problems comes in drive rebuild times. With very large drives, the rebuild of one drive of a RAID 5 can take so long the risk of failure during the rebuild becomes more and more significant. RAID 6 just pushes the problem off a bit.

[Cronos]

ok — I work with datawarehouses so most of our stuff is RAID 1 or sometimes 5

[djf]

A growing number of shops are dumping Wintel server farms (and even IBM RS6000 type farms) for Linux instances running on mainframes.

[andyk]

You have users, threads, posts, mails and the connections between them.

LOL

[tfecw]

Feel free to disagree, but I think you need to improve your knowledge on Facebook's internal systems. There are several papers/preso off of infoq.com and highscability.com if you're so inclined. Both sites also have architecture information on other web scale companies.

I'm sure at one time hundreds of gigs per day was impressive. A year ago, Facebook had roughly 130TB of logs per day. I'm sure that's gone up.

[mad_as_he$$]

Once controlled structure is achieved housekeeping is pure execution. I think the FB problem is that it was started by a dorm room operation not professional computer guys and then never upgraded to deal with large amounts of data. I have two guys that spend all day every day looking at architecture and DB design. They are worth their weight in gold since they are true system guru's. FB seem to have missed that.
Probably didn't make my point as clear as I could of. In data management foundation is everything and worth the initial pain. We start a project today with a very large customer still trying to run a relatively complex operation on Access of all things. Our biggest obstacle is the IT Dept that wrote the application years ago and has been nursing it all this time. Should be real interesting in a painful sort of way!