Cloudbleed

Cloudbleed was a Cloudflare buffer overflow disclosed by Project Zero on February 17, 2017. Cloudflare's code disclosed the contents of memory that contained the private information of other customers, such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. As a result, data from Cloudflare customers was leaked to all other Cloudflare customers that had access to server memory. This occurred, according to numbers provided by Cloudflare at the time, more than 18,000,000 times before the problem was corrected. Some of the leaked data was cached by search engines.

Discovery

The discovery was reported by Google's Project Zero team. Tavis Ormandy posted the issue on his team's issue tracker and said that he informed Cloudflare of the problem on February 17. In his own proof-of-concept attack he got a Cloudflare server to return "private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything."

Similarities to Heartbleed

In its effects, Cloudbleed is comparable to the 2014 Heartbleed bug, in that it allowed unauthorized third parties to access data in the memory of programs running on web servers, including data which had been shielded while in transit by TLS. Cloudbleed also likely impacted as many users as Heartbleed since it affected a content delivery network serving nearly two million websites.

Tavis Ormandy, first to discover the vulnerability, immediately drew a comparison to Heartbleed, saying "it took every ounce of strength not to call this issue 'cloudbleed'" in his report.

Reactions

Cloudflare

On Thursday, February 23, 2017, Cloudflare wrote a post noting that:

The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence.
The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).

Cloudflare acknowledged that the memory could have leaked as early as September 22, 2016. The company also stated that one of its own private keys, used for machine-to-machine encryption, has leaked.

It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.

John Graham-Cumming, Cloudflare CTO, noted that Cloudflare clients, such as Uber and OkCupid, weren't directly informed of the leaks due to the security risks involved in the situation. “There was no backdoor communication outside of Cloudflare — only with Google and other search engines,” he said.

Graham-Cumming also said that "Unfortunately, it was the ancient piece of software that contained a latent security problem and that problem only showed up as we were in the process of migrating away from it." He added that his team has already begun testing their software for other possible issues.

Google Project Zero team

Tavis Ormandy initially stated that he was "really impressed with Cloudflare's quick response, and how dedicated they are to cleaning up from this unfortunate issue." However, when Ormandy pressed Cloudflare for additional information, "They gave several excuses that didn't make sense," before sending a draft that "severely downplays the risk to customers."

Uber

Uber stated that the impact on its service was very limited. An Uber spokesperson added "only a handful of session tokens were involved and have since been changed. Passwords were not exposed."

OKCupid

OKCupid CEO Elie Seidman said: "CloudFlare alerted us last night of their bug and we've been looking into its impact on OkCupid members. Our initial investigation has revealed minimal, if any, exposure. If we determine that any of our users has been impacted we will promptly notify them and take action to protect them."

Fitbit

Fitbit stated that they had investigated the incident and only found that a "handful of people were affected". They recommended that concerned customers should change their passwords and clear session tokens by revoking and re-adding the app to their account.

Remediation

Many major news outlets advised users of sites hosted by Cloudflare to change their passwords, as even accounts protected by multi-factor authentication could be at risk. Passwords of mobile apps too could have been impacted. Researchers at Arbor Networks, in an alert, suggested that "For most of us, the only truly safe response to this large-scale information leak is to update our passwords for the Web sites and app-related services we use every day...Pretty much all of them."

Inc. Magazine cybersecurity columnist, Joseph Steinberg, however, advised people not to change their passwords, stating that "the current risk is much smaller than the price to be paid in increased 'cybersecurity fatigue' leading to much bigger problems in the future."

References

External links

List of domains using Cloudflare DNS on GitHub

Simple website that lets you check for affected domains quickly

A Chrome extension that checks bookmarks against potentially affected domains

Cloudbleed explained-How the biggest web cache leak on internet happened


Quantifying the impact of CloudBleed bug

{{Hacking in the 2010s

Internet security
Software bugs
2017 in computing
Cloud infrastructure attacks and failures
Cloudflare