Once an item is exposed on the internet, it’s there forever. Although content delivery network Cloudflare has fixed the problem in its code that resulted in leaking customer data across the web, the incident is far from over for the millions of websites that rely on the company for security and content optimization services.
What makes this leak so odd and frustrating is that website administrators don’t actually know if their information was leaked or where it wound up. Although Cloudflare knows which customers were affected based on data found in search-engine-cached files, it doesn’t know what information was actually exposed or to whom.
Cloudflare has more than 4 million clients, including governments, e-commerce sites, and financial services organizations, so the ripple effects could be huge. The problem is that no one knows how huge or if it's huge at all.
Although the leakage started in September, the time of the “greatest potential impact” started Feb. 13, said Cloudflare CTO John Graham-Cumming. There's also no evidence so far that anyone had discovered the vulnerability and was exploiting it before the bad code was repaired. Again, no one knows.
What steps you can take to minimize any damage
Still, it’s prudent for IT administrators to assume they have been affected and act accordingly.
One option to consider is forcing a password change for all users. For sites with limited risk to compromised accounts, that may not be worth the effort, but if there is a likelihood that administrator credentials or other sensitive credentials have been exposed, the disruption may be necessary, recommended Ryan Lackey, a security engineer who formerly worked at Cloudflare.
IT administrators should proactively “scour the web” to search for leaked authentication tokens and user credentials, suggested David Weinstein, CTO of NowSecure. If any information is found, sessions should be terminated and passwords reset for affected accounts.
If the application behind Cloudflare used cookies, then get the developers need to be involved in order to understand the risks of having those cookies fall into the wrong hands. “If you are a Cloudflare customer, you should be in full risk assessment mode right now,” said Lori MacVittie, technical evangelist at F5 Networks.
Reissuing certificates and rotating keys aren’t trivial tasks, especially for large organizations. But assuming they haven’t been affected would be a head-in-the-sand reaction, refusing to acknowledge a problem may exist. Thus, IT teams must have a comprehensive understanding of what kind of keys they are using, as well as where and how they’re managed. If they know the damage that could be the result of an exposure, they can take appropriate steps. If not, they have to choose between hoping nothing bad happens or biting the bullet and implementing an expensive, thorough certificate and key reissue.
Mobile applications can also be affected because many of them use the same back ends as web browsers for content delivery and HTTPS termination. There have been reports of HTTP header data for apps such as Discord, Fitbit, and Uber found in search engine caches. NowSecure has listed 200 iOS apps using Cloudflare services. Sites should invalidate authentication credentials for mobile applications and force users to re-enroll apps and devices.
Although all of that sounds potentially alarming, we know at least one area is still safe. Cloudflare's Graham-Cumming said that SSL private keys belonging to customers could not have been leaked because the heap handling SSL keys was entirely separate from the one used by the buggy code.
What happened to cause the Cloudflare leak
How did all this information leak in the first place? Tavis Ormandy from Google’s Project Zero stumbled on the fact that sensitive information was leaking into some responses returned to users and search engine crawlers, so he disclosed the issue to Cloudflare’s engineers.
Due to a programming mistake affecting three “minor” features—Email Obfuscation, Automatic HTTPS Rewrites, and Server Sides Excludes—for several months, Cloudflare’s systems leaked random chunks of server memory into web pages shown to users and to search engine crawlers, Cloudflare’s Graham-Cumming said in the company’s detailed postmortem.
As a result, a user visiting a website powered by Cloudflare may have wound up getting someone else’s web traffic, even the ones that was supposed to be encrypted, hidden on the page. The information—which includes private messages, session cookies, authentication tokens, POST data, encryption keys, and user credentials—could have been cached by search engines. The data may exist in other caches, such as a local browser cache on a user’s computer, or saved by other privately run web-scraping services.
Cloudflare worked with search engines to scrub the leaked information from cached pages, but it’s anyone’s guess exactly how widespread or long-lasting the effects of the leak would be.