Cloudbleed was a security bug discovered on February 17, 2017 affecting Cloudflare's reverse proxies, which caused their edge servers to run past the end of a buffer and return memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. As a result, data from Cloudflare customers was leaked out and went to any other Cloudflare customers that happened to be in the server's memory on that particular moment. Some of this data was cached by search engines.
Discovery
The discovery was reported by Google Project Zero team. Tavis Ormandy posted the issue on his team's issue tracker and said that he informed Cloudflare of the problem on February 17. In his own proof-of-concept attack he got a Cloudflare server to return "private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything."
In effects, Cloudbleed is similar to the 2014 Heartbleed bug in allowing unauthorized third parties to access data in the memory of programs running on web servers, including data shielded by TLS. The extent of Cloudbleed also could have impacted as many users as Heartbleed since it affected a security and content delivery service used by close to 2 million websites. Tavis Ormandy, first to discover the vulnerability, immediately drew a comparison to Heartbleed, saying "it took every ounce of strength not to call this issue 'cloudbleed'" in his report.
Reactions
Cloudflare
On Thursday, February 23, 2017, Cloudflare wrote a post noting that:
The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence. The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage.
Cloudflare acknowledged that the memory could have leaked as early as September 22, 2016. The company also stated that one of its own private keys, used for machine-to-machine encryption, has leaked.
It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.
John Graham-Cumming, Cloudflare CTO, noted that Cloudflare clients, such as Uber and OkCupid, weren't directly informed of the leaks due to the security risks involved in the situation. “There was no backdoor communication outside of Cloudflare — only with Google and other search engines,” he said. Graham-Cumming also said that "Unfortunately, it was the ancient piece of software that contained a latent security problem and that problem only showed up as we were in the process of migrating away from it." He added that his team has already begun testing their software for other possible issues.
initially stated that he was "really impressed with Cloudflare's quick response, and how dedicated they are to cleaning up from this unfortunate issue." However, when Ormandy pressed Cloudflare for additional information, "They gave several excuses that didn't make sense," before sending a draft that "severely downplays the risk to customers."
Uber
Uber stated that the impact on its service was very limited. An Uber spokesperson added "only a handful of session tokens were involved and have since been changed. Passwords were not exposed."
OKCupid
OKCupid CEO Elie Seidman said: "CloudFlare alerted us last night of their bug and we've been looking into its impact on OkCupid members. Our initial investigation has revealed minimal, if any, exposure. If we determine that any of our users has been impacted we will promptly notify them and take action to protect them."
Fitbit
Fitbit stated that they had investigated the incident and only found that a "handful of people were affected". They stated that concerned customers should change their passwords and clear session tokens by revoking and readding the app to their account.
Remediation
Many major news outlets have advised consumers of sites using Cloudflare to change their passwords, even for accounts protected by 2-factor authentication as they could be at risk. Passwords of mobile apps too could have been impacted. Researchers at Arbor Networks, in an alert, suggested that "For most of us, the only truly safe response to this large-scale information leak is to update our passwords for the Web sites and app-related services we use every day...Pretty much all of them." Inc. Magazine cybersecurity columnist, Joseph Steinberg, however, advised people not to change their passwords, stating that "the current risk is much smaller than the price to be paid in increased 'cybersecurity fatigue' leading to much bigger problems in the future."
