For every website and service simplifying how we get information, complete a transaction, or communicate with others, there's a growing number of web-based threats intent on compromising user safety and privacy. Securing all web content over HTTPS is now a necessary step as we increase our dependence on the internet.
"There is no longer any justification for any service on the internet to not be secure," said Eric Mill, an engineer at 18F, a team within the United States General Services Administration (GSA) that provides in-house technology services for the federal government. He is working with various federal agencies to shift all government web services through HTTPS and HSTS (HTTP Strict Transfer Security).
The White House OMB (Office of Management and Budget) issued the HTTPS Only Standard directive (M-15-13) on June 8, 2015, requiring that all publicly accessible federal websites and web services be available only over a secure connection. Under the directive, existing websites and services have until Dec. 31, 2016, to make the switch. As of August, 52 percent of federal websites and services are using HTTPS, and Mill was confident the remaining ones will meet the deadline.
The government's decision follows major social media sites like Facebook and Twitter, which have adopted HTTPS to ensure that all user actions occur over a secure connection. Google recently turned on HSTS for its home page to force browsers to always use HTTPS when connecting to www.google.com. The company already favors HTTPS websites in search rankings, and Chrome and Firefox are working on marking plain HTTP sites as nonsecure.
The ultimate goal is to establish encrypted connections -- HTTPS with HSTS -- as the norm all over the internet and phase out unencrypted HTTP connections altogether.
While some organizations have switched their sites to transfer data over encrypted HTTP connections, there are many more that have not yet done so. Switching websites and services from HTTP to HTTPS requires careful planning and execution, but changing an organization's culture and mindset can also be challenging.
The federal government is a large enterprise -- immensely so, with a lot of power, to be sure -- but its technical and political challenges should be familiar to any business embarking on the same journey. Over at the GSA, Mill said there was a lot of pounding the pavement to get staff at each agency on board with the directive and to understand why the switch was necessary. The team sent support emails and regularly visited agencies to talk to staff members about how to protect the users.
"We had to first challenge their assumptions," Mill said. "Agency staff aren't thinking about what it means to click through certificate warnings in the browser."
It's easy to argue that HTTPS should be applied to sensitive information alone, such as login credentials and e-commerce transactions, and deliver the rest of the content as unencrypted HTTP. That argument is based on privacy grounds, but it completely ignores the fact that much of the unencrypted information -- cookies, user agent details, URL paths, form submissions, and string query parameters -- can be used to infer a lot about users and their networks.
There have also been cases where network operators, such as hotel chains and ISPs, inject ads or other promotional content in the middle of a user session. Then there's the attack against GitHub last year, where someone modified JavaScript code used by the Chinese search giant Baidu's analytics and advertising tool to bombard GitHub with junk traffic. The Great Cannon, as the technique was called, and similar tools would have been neutralized if websites sent all of their pages over encrypted HTTPS connections.
Using HTTPS is not about privacy alone, but security, too. "There is no such thing as nonsensitive web traffic, and site owners have to assume the network is hostile," Mill said.
Another important lesson focused on making sure agency staff understood what HTTPS could not do. HTTPS guarantees the integrity of the connection between two systems and the contents have not been subject to tampering. It doesn't guarantee that the systems on either end of the connection have not been compromised. If the user's computer has already been compromised or the web service is exposing user information due to a configuration error, the HTTPS connection won't prevent data leakage or theft. Enterprise security to protect individual systems and servers is still important.
The biggest technical challenge was to identify all the sites and services that needed to be moved to using HTTPS connections. Public websites aren't alone in using HTTP; APIs are as well, along with individual applications residing on various subdomains.
The directive covered all federal sites, which meant more than .gov sites alone. There were easily 11,000 to 12,000 domains, along with "tens of thousands" of services and APIs, and no one had a master inventory listing them all, Mill said.
Knowing what sites and services were used was important -- the last thing anyone wanted was to break user connections, especially in the case of an API.
The GSA was able to use information from Censys.io, a search engine of devices and network on the internet maintained by the University of Michigan. Most organizations ideally should not have to go this route to create the master list.
The directive also required federal sites to use HSTS, which instructs browsers to always use HTTPS when trying to access the site. This means users can no longer click through certificate warnings in supporting browsers, so if the agency is not on top of its certificate management, users would no longer be able to reach the sites. How to manage the certificates and which kinds of certificates to get were among the most frequently asked questions, Mill said.
While free or inexpensive domain-validated certificates acceptable, investing in the pricier extended-validation certificates provides an additional layer of protection.
There's plenty left to do. Even with half of the sites using HTTPS, 37 percent enforce HTTPS, and a mere 14 percent currently have HSTS enabled. Preloading, where the browsers know to use HTTPS to access the URL even if the user has never visited the site before, is the next big hurdle. At the moment, only a few thousand sites worldwide use preloading, so it's an underutilized security feature. Mill suggested pushing top-level domains to require HTTPS by default, saying, "There's so much more HTTP left to get rid of."