Facebook Down

Facebook was down yesterday morning for about 2.5 hours. From around 9:30 AM eastern standard time to about 12:00 PM most people reported not being able to access Facebook. Most received a DNS failure message similar to this one:

Service Unavailable – DNS failure

The server is temporarily unable to service your request. Please try again later.

This was Facebook's longest outage in about 4 years and was the result of the database being overwhelmed by requests. This was due to a new software update that handled certain error conditions. The software update incorrectly found an invalid configuration value when someone connected to their Facebook account. This error would result in numerous queries to the database. When you times the number of database queries by the number of people who use Facebook everyday worldwide, this error quickly brought Facebook's database servers to their knees. The problem also compounded itself because, if the same person tried to visit Facebook again, the whole process would start over again.

To fix the problem, Facebook's engineering team had to shut down the site. Once the database servers stopped receiving requests the reason for the problem was discovered and then fixed. Once completed, Facebook then slowly allowed people back on the site and monitored the situation closely. The system that checks for errors in configuration values is turned off at the moment. Engineers are going over new designs for the system to make sure this problem doesn't happen again.

For more details on the problem Facebook experienced, read Robert Johnson Blog.