Facebook Blames “Faulty Configuration Change” for Nearly Six-hour Outage

Facebook Inc has cited ‘faulty configuration changes’ on its routers as the root cause of the nearly six-hour outage, depending on your time zone, that prevented the company’s 3.5 billion users from accessing its social media and messaging services such as WhatsApp, Instagram, and Messenger.

Others have provided a bit more detail on why Facebook vanished from the internet. Cloudflare – which had its own recent internet outage issues – has provided a detailed explanation about what happened. It involves two things that sort out how the internet is the internet – that is Domain Name System (DNS) and Border Gateway Protocol (BGP).

The internet is a lot of connected networks. A lot. So that means to keep order of things, you need something like BGP to tell you where you need to go. DNS is essentially the address system for the location of each website – its IP address – while BGP is the roadmap that finds the most efficient way to get to that IP address.

Cloudflare said Facebook on Monday essentially told BGP through a series of updates that those paths to Facebook no longer existed. But not just for Facebook, everything Facebook runs. That meant people trying to reach Facebook couldn’t find the path to access it.

Why were Instagram, Messenger and WhatsApp down?

All of Facebook’s services were affected, not just Facebook. It included Facebook’s own internal systems, with reports staff were locked out of offices, and could not access their own internal communications platform.

To the huge community of people and businesses around the world who depend on us: we're sorry. We’ve been working hard to restore access to our apps and services and are happy to report they are coming back online now. Thank you for bearing with us.

— Facebook (@Facebook) October 4, 2021

Why did it take so long to fix?

Facebook’s own internal systems are run from the same place so it was hard for employees to diagnose and resolve the problem. Facebook staff were reportedly unable to access their own communications platform, Workplace, and were unable to access their office due to the security pass system being caught up in the outage and it indicated the duration and severity of the outage meant the systems were being brought back to full capacity slowly.

How did they eventually fix it?

Facebook so far has not gone into much detail about what went wrong and how it was fixed, but there were multiple reports the social media giant sent a technical team out to its servers in California to manually reset the servers where the problem originated.

Can this sort of outage be avoided in future?

This one is fairly uncommon but not something that can be completely avoided. However, the Facebook outage, along with others including the Cloudflare outage in 2020, and the Fastly one in June, show the problems with having a single point of failure for a vast number of online services people rely on.

People rely on Facebook not only to connect with friends and family, but businesses use it to log into other services including online sales websites. In some countries, it is the dominant means of communication through services like WhatsApp. That an outage can have such a profound impact on billions of people for several hours will give some pause for thought.

Was my personal data at risk?

No more than when Facebook is up and running.

Isn’t this all weird timing for Facebook?

Yes. Facebook is not having a good time right now. Last week Facebook paused a plan to launch Instagram for kids, after leaked internal research showed the company was aware the app could affect girls’ mental health.

Then on Sunday, the former Facebook civic integrity product manager Frances Haugen went public with explosive allegations that Facebook had prioritised growth and profit over public safety. “The version of Facebook that exists today is tearing our societies apart and causing ethnic violence around the world,” she told 60 Minutes.

How much did it cost?

We haven’t seen a cost estimate yet for how much it cost businesses that rely on Facebook. The outage, along with the whistleblower story on Sunday, prompted Facebook’s share price to drop 4.9% on Monday, causing founder and CEO Mark Zuckerberg’s personal wealth to drop $6bn, according to Bloomberg.