What better way to start off the week than by not being able to use virtually any online service or access half of the internet’s most popular sites? That seems to be the case, as DownDetector (and many tweets) suggest that Google, Amazon, Reddit, and Spectrum — just to name a few — are experiencing issues this morning. Those issues appear to have begun around 6 or 7 AM ET, just as the East Coast was starting its day.
Although some of these connection problems appear to be clearing up as of 8:40 AM (for example, Feedly is finally loading for me after being inaccessible since before 8:00 AM), it’s likely going to take some time before everything is running smoothly again. Reports are still going up on DownDetector as of writing.
It’s unclear what is causing half of the internet to go down, but an ominous message from Discord refers to the issue as a “general internet outage,” which doesn’t sound like something that should be possible:
https://twitter.com/ChrisGSeaton/status/1143136635153977345
About an hour ago, internet service company Cloudflare says that it “identified a possible route leak impacting some Cloudflare IP ranges.” [UPDATE: To be clear, Verizon was responsible for the outage, and Cloudflare was just keeping its customers informed during the recovery process.]
Cloudflare followed up with another update about an hour later explaining that the leak “is impacting many internet services including Cloudflare,” and moments later, announced that the network responsible for the leak had fixed the issues as of 8:42 AM ET. In theory, the worst of the outage is over.
We’ll be keeping an eye out for any residual issues that pop up in the hours to come, but we also hope to get a more detailed explanation for why this happened from the network responsible in the near future.
UPDATE | 3:30 PM: After service was restored, Cloudflare issued the following statement (via TechCrunch):
Earlier today, a widespread BGP routing leak affected a number of Internet services and a portion of traffic to Cloudflare. All of Cloudflare’s systems continued to run normally, but traffic wasn’t getting to us for a portion of our domains. At this point, the network outage has been fixed and traffic levels are returning to normal.
BGP acts as the backbone of the Internet, routing traffic through Internet transit providers and then to services like Cloudflare. There are more than 700k routes across the Internet. By nature, route leaks are localized and can be caused by error or through malicious intent. We’ve written extensively about BGP and how we’ve adopted RPKI to help further secure it.
Cloudflare CEO Matthew Prince also offered a biting take of his own on Twitter:
The teams at @verizon and @noction should be incredibly embarrassed at their failings this morning which impacted @Cloudflare and other large chunks of the Internet. It’s absurd BGP is so fragile. It’s more absurd Verizon would blindly accept routes without basic filters.
— Matthew Prince 🌥 (@eastdakota) June 24, 2019
