The Huge Web Security Loophole That Most People Don’t Know About, And How It’s Being Fixed

Can we trust the organizations in charge of ensuring that encrypted web traffic is trustworthy?

The Huge Web Security Loophole That Most People Don’t Know About, And How It’s Being Fixed
[Photo: Flickr user Bastian]

Hundreds of companies and other entities around the globe, some associated with national governments, occupy a place of power on the Internet that you likely don’t even know exists. Certificate authorities (CAs) sit at the apex of the root of trust that allows the secure web, email, and other connections that underlie commerce, government, online communities, and everything else to function without effective interception by outside parties.


On February 18, it was explained by Kenn White in a series of tweets how Lenovo apparently preinstalled adware that uses the enormous power inherent in certificates to intercept all web interactions by customers on its laptops and rewrite pages and insert advertising in secure sessions. Lenovo bypassed certificate authorities, but took advantage of gaps in the system that checks for legitimacy.

A certificate authority provides outside validation that a security document—a digital certificate—that is sent by a server was properly issued to someone who controls that server’s domain name. The domain name you use in a browser matches the name in the certificate, which matches the verification process that domain owner went through to obtain it. The CAs are part of a chain of trust that includes developers of operating systems and browsers, and represent the weakest link.

A CA that gets hijacked or otherwise misused can issue documents for any domain anywhere that will be accepted as valid. This allows attacks in which it’s impossible to detect anything has gone awry. For average users, a browser lock icon appears; for automated software, no alarms go off. Check your Gmail, and it all looks and acts like normal, but every byte is in the hands of an intermediary.

A typical certificate

The Threat is Real

This isn’t theoretical. Multiple times in the last decade, it’s been discovered that certificates were issued that could or did lead to weaknesses. The most well-known of these is DigiNotar, a Dutch CA that was compromised in 2011. One of the certificates was allegedly used by the government of Iran to intercept sessions of its citizens. (Revoking compromised certificates remains an ugly and broken task as well, although that’s changing.)

Further, many CAs are run by governments or are widely suspected to be under the authority of governments. One of the most recent revelations from documents obtained by Edward Snowden included details about the NSA’s collection of massive numbers of encrypted web sessions, and its ability to crack some set of them. This was likely based on the use of weak encryption algorithms and poor internal security at some CAs.

There is little recourse to change which parties are involved in issuing certificates, because of the messy evolution of CAs and the legacy systems in place that rely on outdated information. Ivan Ristić, the author of Bulletproof SSL and TLS and a researcher at Qualys, says, “There’s thousands of entities that have evolved into designing it, making it, and using it, but you have to get everyone along for the journey.”

Ivan Ristić

“The marketplace in which these institutions exist is structurally dysfunctional,” says Peter Eckersley, the technology products director at the Electronic Frontier Foundation (EFF). He notes that it doesn’t matter which certificate one buys, from pricey, high-end ones to the cheapest out there: “In all of those cases, you get the security of the worst CA,” as every CA can issue a certificate for every domain.

Trusting that certificate authorities are both secure and worthy of trust ranks high among the biggest problems on the Internet, despite the details being seemingly obscure. The good news is that after years of inaction and false starts, several solutions are finally being put into place.

The Root Of Trust

Back when Netscape ruled the roost in the mid-1990s, it developed Secure Sockets Layer (SSL) as a way to answer the overwhelming concerns about data interception. This was before ubiquitous cellular and Wi-Fi networks whizzed data around our ears, but the hardwired Internet wasn’t designed around security, either. In those days, I walked into “meet-me” rooms where routers controlled by different companies interconnected with little oversight or observation.


SSL was later supplanted by an evolutionary improvement called TLS (Transport Layer Security), although the term SSL is often used generically when either SSL/TLS or TLS is accurate. By any name, it defines the kind of security information that needs to be exchanged, and what formats are accepted. These revolve around digital certificates, which are bundles of encryption keys and identifying information. In SSL/TLS, one party initiates a connection—like a browser connecting to a web server—and negotiates what sort of encryption both ends can talk. The server provides a certificate, which is validated, and then the client generates a random number which is employed only for that single session and sent securely to the server. Both client and server derive various task-specific encryption keys from that number.


The certificate contains the public key associated with the server or a set of servers. A public key is a wonderful bit of encryption hocus-pocus that’s in wide use because it solves a basic problem: How do you pass information between two parties without both sharing the same key? After all, if you could securely share the key with each other, it would mean that you already had a secure method of communications, and had no need for the key.

The Two-Key Solution

With public-key (PK) cryptography, an algorithm generates two complementary keys: One is private and you must protect it at all costs. The other is public and may be shared freely without compromising the private key. At this point in time, there’s no practical way to use brute force to determine a private key when one knows the public one.

This public-private key pair can be used in two distinct ways. The first is for what you might think of as regular encryption. Alice wants to send Bob the plans for her proprietary sous vide immersion heater, and she uses Bob’s public key to encrypt it. When Bob receives it, he uses his private key to decrypt it.


An SSL/TLS digital certificate contains the public-key component of a server’s key pair, but it also includes something extra from the CA. This is the second major use of a PK: validating that a document hasn’t been modified, which is known as a digital signature. (This is a generally useful security technology that’s also used outside of SSL/TLS.)

How digital signatures workGraphic by Acdx via Wikipedia

An input document, whether the details of an SSL/TLS certificate or a letter to a friend, has a mathematical summary computed, known as a hash. Hashes are used to produce short sequences of text that vary enormously when even a single character or digit of the input source changes. The hash is then signed by the private key.

Any party receiving the document can be sure it hasn’t been modified since it’s signed by computing the same hash and decrypting the signature portion with the public key, which should result in an identical hash. Bob posts a recommendation of Alice, signed with his private key, and people who want to be sure Alice isn’t fibbing can validate that only Bob signed it.


In case of a CA, the CA signs a document with its private key that contains the public key of the server. The trouble is: How do you trust the CA’s public key belongs to the right party?

That’s the basis of the public-key infrastructure (PKI) for public and private SSL/TLS connections. CAs sign certificates, and their public keys and other data are baked into browsers and operating systems by Apple, Google, Mozilla, Microsoft, and others. The CAs in these lists are called “trusted roots.”

Which provokes the next question: How do you trust the certificate authority? And should the CA be fully trustworthy, how do you know that a certificate issued is one it legitimately intended to, given the cracks and subversion that have happened in the last few years?


The efforts to fix those latter problems of trust are finally bearing fruit.

All Of Us Are As Weak As Some Of Us

There is no master list of root CAs. Each company (like Apple) or organization (such as Mozilla) makes up its own mind, and while the lists largely intersect, there are CAs one finds in one and not another. The list produced by Mozilla’s consensus-driven community process winds up the default choice for organizations that don’t want to exert the effort to build their own.

Richard Barnes, the head of Mozilla’s cryptographic engineering group, says its “certificate root program establishes criteria for what is required for a root to be admitted to the program.” This includes conforming to a set of industry standards that are regularly updated and “ratcheted up” with new requirements. Mozilla and other parties that create their own root lists regularly audit CAs for compliance. CAs are removed or warned on a regular basis. (Mozilla operates both as a “relying” party, which validates CAs for all its products and for the world at large, and as a software maker that makes certificate-related decisions for the software it creates.)


For instance, a few years ago, some authorities issued SSL/TLS certificates for use in corporate data-inspection devices. These signed certificates allowed companies to inspect the data flowing from employees and others through their Internet gateway. While there are benign corporate security reasons for it, the risk of such certificates being misused by others is high. Mozilla and others said this practice would no longer be allowed, and CAs stopped issuing them. Domain-specific certificates can still be used in such devices, but only for those domains owned by the company deploying them.

These policies help good players remain good. But when something goes awry—whether a legitimate mistake, a hack or theft, or a government-backed interception attempt—other tools have to come into play in operating systems, browsers, and via independent global network observers.

Lenovo’s apparent breach of trust uncovered last night appears to have to resulted from the company bypassing the root programs, like Mozilla’s, and installing its own artificial certificate authority directly into Windows, as well as a certificate valid for all domains that was signed by that fake CA. (Mozilla confirmed that Firefox remains unaffected, because it relies on its own set of root certificates.) This alleged subversion by Lenovo reveals the weaknesses in the current systems’ check and balances, which could soon change.


Three approaches are already helping uncover problems, prevent new ones, and deter parties from engaging in certain kinds of observable attacks:

  • Pinning, in which apps indicate that only specific CAs are allowed to issue certificates on their behalf. This requires no changes by CAs, and can be and has been unilaterally added to operating systems, browsers, and individual apps.
  • Certificate transparency, which requires CAs to publish every certificate they issue. CAs have to agree to produce this data, but there is a general move to make it mandatory for at least Extended Validation (EV) certificates, which require more independent proof of identity by a certificate-registering party.
  • Outside secure site-checking projects and tools spearheaded by several parties, including EFF’s SSL Observatory, the Trustworthy Internet Movement’s SSL Pulse, and the University of Michigan’s ZMap. These efforts help make clear what’s going on globally at the server level, but require no participation or approval by anyone else.

(There are two other approaches as well. DANE (DNS-based Authentication of Named Entities), relies on the domain naming system (DNS), but requires the cryptographic infrastructure of DNS to evolve more fully, among other concerns. And certificate notaries, such as Perspectives, aimed to collect certificate information and, via browser plug-ins or support, warn a user when an unknown certificate appeared. This had some traction in 2012, but is more or less moribund now.)

“Each of these technologies shaves off another area of risk—another area of attack surface,” said Barnes.


Pinning appears to have the most short-term impact, and its use dates back several years. Google pinned its own chain of authority within the Chrome browser, which led it to discover a man-in-the-middle attack in Iran with a certificate obtained illicitly from DigiNotar.

Developer Marco Arment (a colleague of this reporter) recently tweeted about pinning certificates in his iOS podcast app, Overcast, which has over 200,000 users who have registered accounts. Arment said via email, “I’ve never heard from a single user who wasn’t able to use Overcast because of my use of SSL pinning.” Overcast doesn’t store financial information and doesn’t have huge numbers of users, so it doesn’t present a particularly tempting target to thieves or governments. But EFF’s Eckersley says best practices for apps would be to pin certificates, regardless of the category.

Mozilla began pinning in late 2014 with version 32 (34 on Android), starting with its own domains and most of Twitter’s, following Google’s lead. It expanded to cover Google and then Dropbox, and will be adding more. Every major domain for which a browser pins certificates reduces enormous parts of the risk of a certificate-based attack for a large portion of Internet users, since so much network use is centered on several key firms.


This will be expanded soon with support in newer versions of Firefox and Chrome for HTTP Public-Key-Pinning (HPKP), which will allow web servers to provide pinning information whenever a browser connects via https. This approach lets a server pin anywhere from a root-level CA all the way to individual certificates issued for the site.

A sort of inverse situation has browsers limiting roots to having authority over certificates only for specific domains. After a kerfuffle with the French government’s authority, ANSSI, Google restricted the CA to only asserting authority over a limited set of top-level country codes, including .fr and other French territories. Mozilla’s Barnes says it may constrain more CAs. The U.S. government wants its CA included in Mozilla’s root list, and Barnes says it may be added, but limited to validating the government’s own .gov and .mil domains “for all the reasons you might expect.” (He declined to elaborate, but in the wake of Snowden, WikiLeaks, and even congressional committee disclosures about NSA and other agencies’ practices, it’s understandable why he might be cautious about providing the feds with an unfettered role in certificate security.)

Photo: Flickr user Tanakawho

Going Transparent

Certificate transparency (CT) is less far along, but relies on a public record of issued certificates that can be independently verified as being issued by CAs. This log can be monitored in two different ways: first, to be sure that a certificate presented by a server is one that was properly issued; second, that a CA didn’t issue a certificate for a domain that didn’t request it. Google is deeply involved in pushing CT forward. (It didn’t respond to a request for comment.)

Security researcher Ristić noted that it would be trivial to operate a monitoring service in which the public log was continuously checked to see if one’s own domain appeared in it from any party or at any time other than those expected.

This ties in neatly with the various global scanning projects, which non-harmfully scan every public website or just every secured one by making a standard web connection, and build a corpus of information that can be summarized or analyzed by outside parties. Barnes said that these efforts help the group identify problems at various levels, including reporting issues to CAs or crafting new policies.

The certificate authority problem isn’t solved by any means, but the opportunities for mischief and abuse are reduced by every effort that limits the scope of what any given CA can do and what any given piece of software or operating system is willing to accept. Ristić said of CAs that once scanning efforts reported on what they found, “Suddenly, they realized, others were watching.”

With the automated eyes of the world finally providing constant scrutiny and under pressure from the groups that control the root CA lists, certificate authorities have been forced to evolve. CAs don’t represent a worldwide conspiracy, but an accidental one. Verifying trust will decrease the risk that every Internet user faces today, without knowing it.


About the author

Glenn Fleishman is a long-time technology reporter based in Seattle. He has written thousands of articles about encryption, cryptocurrency, privacy, satellites, printing history, and more