Cloudflare Outage Sparks Controversy Over Internal Error

Cloudflare blamed an outage that put major websites and services out of commission for several hours Tuesday on an internal configuration error, highlighting once again the issue of third-party interdependence in the cloud computing ecosystem.

The incident — which occurred beginning 11:20 UTC on Tuesday and affected sites such as X, Uber, Canva, ChatGPT, among others — was initially thought to be due to a distributed denial of service (DDoS) attack, according to the blog post by Cloudflare founder and CEO Matthew Prince, which broke down the technical aspects of the outage. 

However, the company discovered the issue was “a change to one of our database systems’ permissions, which caused the database to output multiple entries into a ‘feature file’ used by our Bot Management system,” Prince wrote. 

As a result to this change, the feature file doubled in size and was then propagated to all the machines in Cloudflare’s network; the software running on these systems reads the file to keep its Bot Management system up to date with constantly shifting threats. 

However, there is a limit on the size of feature files, which the new file surpassed, according to Prince. “That caused the software to fail,” which in turn led to websites running on Cloudflare to deliver “internal server error” messages to users, he wrote.

Related:How CISOs Can Best Work With CEOs and the Board: Lessons From the Field

Technical Details

Cloudflare first noticed an issue in its network when it discovered that the volume of 5xx error HTTP status codes began to spike well above baseline, and subsequently fluctuate wildly. This demonstrated that the system was failing due to loading the incorrect feature file, Prince explained.

What was notable in the system’s behavior is that it would repeatedly fail and then recover, which is “very unusual behavior for an internal error,” he noted. That’s why initially Cloudflare thought it was under DDoS attack.

However, what administrators eventually realized is that the misconfigured file was being generated every five minutes by a query running on a ClickHouse database cluster, which was being gradually updated to improve permissions management. 

“Bad data was only generated if the query ran on a part of the cluster which had been updated,” Prince wrote. “As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.”

This fluctuation made it unclear what was happening until eventually, “every ClickHouse node was generating the bad configuration file and the fluctuation stabilized in the failing state,” he observed. 

Related:Microsoft Exchange ‘Under Imminent Threat,’ Act Now

Once the problem was pinpointed, Cloudflare solved the issue by stopping the generation and propagation of the bad feature file and manually inserting a known good file into its distribution queue, and then forcing a restart of its core proxy.

Resolution and Apology

Cloudflare had mostly resolved the incident by 14:30 UTC, with “core traffic was largely flowing as normal,” according to the post; by 17:06 UTC, all systems at Cloudflare were once again functioning as normal.

Prince acknowledged the importance of Cloudflare to the Internet ecosystem and apologized for the incident, declaring that “any outage of any of our systems is unacceptable.” The company serves as a content delivery network providing security and other services for about 20 percent of all websites on the Internet.

“That there was a period of time where our network was not able to route traffic is deeply painful to every member of our team,” Prince wrote. “We know we let you down today.”

Indeed, the company has not had an outage that caused the majority of core traffic to stop flowing through its network since 2019, though it has had minor incidents in the period between.

Outage Highlights Business Continuity

Related:Grandparents to C-Suite: Elder Fraud Reveals Gaps in Human-Centered Cybersecurity

Unfortunately, network outages that cause downtime on critical business websites are still a fairly common occurrence. Last year a buggy CrowdStrike update knocked offline various systems, including payment and airline reservations systems, offline, costing businesses an estimated $5.4 billion and spurring lawsuits against the company for lost revenue and other downstream effects. And on Oct. 20, AWS suffered a major outage, which stemmed from a DNS issue, that affected cloud service customers for much of the day. 

These incidents call into question once again the vulnerability of organizations’ reliance on the Internet running smoothly and without a hitch for their business survival. Indeed, in the era where artificial intelligence (AI), quantum computing, and other advanced technologies are on the rise, the infrastructure that provides the networks powering these technologies are now as critical as the electrical grid or water supply. 

While the incident was not the result of a cyberattack, it shows the fragility of an Internet ecosystem and demonstrates the need for organizations to understand where interdependence exists. It also emphasizes the continued need for organizations to have business continuity and disaster recovery plans in place to provide back-up for any third-party issues that could disrupt their websites, services or other business activities.

Prince said Cloudflare is already working to harden its networks against future failures. These steps include hardening ingestion of Cloudflare-generated configuration files in the same way that would be done for user-generated input; enabling more global kill switches for features; eliminating the ability for core dumps or other error reports to overwhelm system resources; and reviewing failure modes for error conditions across all core proxy modules.