While the KGB may have collapsed with the Soviet Union in 1991, its long shadow still quietly serves its homeland through new 21st-century digital forms and tactics, especially in cyberspace.
When the Communist Bloc splintered, so did the KGB. Two new agencies were born: the Foreign Intelligence Service of the Russian Federation (SVR), a spy agency tasked with gathering intelligence, and the enforcement arm, the Federal Security Service (FSB), which is charged with security. Both conduct espionage and counterintelligence.
In the waning months of 2020—a particularly arduous and painful year for the entire world—the SVR threw a wet blanket on an already dampened holiday. Headquartered in a sprawling office complex in southwest Moscow, its buildings from the outside look like a typical modern office park in the suburbs. A close equivalent to the U.S. Central Intelligence Agency (CIA) the SVR manages from these buildings its global espionage and external counterintelligence operations, including electronic surveillance in countries around the world. These days, this includes a wide variety of cyber activities. And not unlike the tech companies in similarly sprawling campuses across the United States that employ our best and brightest technology minds, the SVR has a large and experienced staff of talented cyber professionals.
Inside Microsoft we use obscure elements from the periodic table to classify and code-name different nation-state actors that are engaged in cyber activity. Within our own cybersecurity ranks, the SVR is called Yttrium, a metallic rare-earth and toxic mineral found in the Earth’s crust.
Yttrium is one of the most sophisticated cyberthreats for many of our customers, as well as for Microsoft itself. And instead of conducting its clandestine efforts in back rooms and through dead drops, this 21st-century successor to the KBG often does its work by penetrating computer networks owned by private enterprises and citizens around the world.
WHY THE SOLARWINDS ATTACK WAS SO DEVASTATING
Yttrium’s latest threat hit my radar in the form of an instant message on the last day of November 2020, asking if I had “five minutes for something kind of urgent.” The message came from Tom Burt, our vice president responsible for a wide range of cybersecurity issues. I knew from more than two decades of working with Tom that he was typically calm and understated, two key qualities when dealing with crises. I knew that Tom’s “kind of urgent” was likely an alarm bell.
I quickly ended a meeting to talk with Tom. He reported that we had been approached by FireEye, one of the leading cybersecurity firms, for help with what looked like a serious cybersecurity breach it had suffered. As Tom explained, the early indicators pointed to Yttrium, a finding later confirmed by the White House.
In the coming weeks, this initial report would lead cybersecurity experts to pursue a digital trail that uncovered the attempted hacking of dozens of sensitive computer networks around the world, including Microsoft itself. By January 2021, The New York Times reported that “the U.S. government was clearly the main focus of the attack, with the Treasury Department, the State Department, the Commerce Department, the Energy Department, and parts of the Pentagon among the agencies confirmed to have been infiltrated.” But those weren’t the only targets. At Microsoft we identified dozens of impacted organizations, including other tech companies, government contractors, think tanks, and a university. The impacted countries spilled beyond the United States to include Canada, the United Kingdom, Belgium, Spain, Israel, and the United Arab Emirates. In several instances, the network penetrations had lasted for several months.
The attack quickly became the broadest confirmed penetration of U.S. government and tech sector computer networks. While cybersecurity experts would give the attack a variety of names, including Solorigate and Sunburst, the public would mostly read about it with reference to the Texas company whose software was hijacked to stage the initial attacks—SolarWinds.
But what to make of all this? Was it, as some on Capitol Hill suggested, a Russian “act of war” or a “digital Pearl Harbor”? Or was it just “espionage as usual,” as some in the intelligence community countered. In my view, it was neither.
Its importance is difficult to overstate. The cyberattack provided a “moment of reckoning” that demonstrated technology’s inherent strengths and weaknesses and illustrated the degree to which it had become both a defensive tool and an offensive weapon. And perhaps more than anything, it showed the world how much work we must do to manage all the implications of inventions that are remaking the century in which we live.
But all this requires putting things in perspective. Even before the attacker’s identity was apparent, anything that potentially involved FireEye was a big deal. FireEye is one of the world’s most sophisticated cybersecurity firms. Its CEO, Kevin Mandia, is one of the country’s leading cybersecurity experts, having started his career as a computer security officer in the Air Force before founding his own security company that FireEye eventually acquired. If this leading security company had been penetrated, it almost certainly took an incredibly sophisticated attack. And if the SVR had made it through FireEye’s defenses, it likely was succeeding elsewhere.
RESPONDING TO THE ATTACK
Yttrium had long represented an important area of focus for the engineers at the Microsoft Threat Intelligence Center, which we call MSTIC (pronounced “mystic”). An elite unit itself, MSTIC constantly focuses on identifying and combating emerging cybersecurity threats. It relies on a combination of the world’s best technology and engineers to sift through the 6 trillion electronic signals that flow into our data centers every day. It’s a combination that places MSTIC in the top tier of a cybersecurity ecosystem responsible for protecting the security of not just Microsoft’s network but the company’s government and business customers and much of the world’s critical infrastructure.
For centuries, governments have engaged in espionage and counterintelligence operations. One of the unusual hallmarks of our digital era is that much of this daily fencing today also involves tech companies. It’s MSTIC’s job to hunt for new intrusions and cyberattacks from Yttrium and other nation-state actors. Yttrium also occupies a particularly significant position at the top of the cybersecurity arena. Long known not just for technical sophistication but operational persistence, Yttrium has succeeded in a way that few can match in penetrating sophisticated computer networks and operating undetected for prolonged periods of time.
One challenge was that Yttrium had become more difficult to track. As 2020 was winding down, however, it appeared that Yttrium had reemerged with renewed fervor. If Yttrium had broken into FireEye and elsewhere, it would be important to push the intruders out of the affected networks before they could extract more information. And we wanted MSTIC to learn as much as it could about Yttrium’s new methods before it covered its tracks. One of the ironies of this type of cyberattack was that it represented both a successful espionage coup for an attacker and, once identified, a new opportunity for a defender to spot tactics, techniques, and procedures that could help identify and thwart other ongoing or future attacks.
One key was to move fast and with as many responders as possible. Microsoft’s Security Response Center (MSRC) quickly activated its incident response plan. The MSRC is part of Microsoft’s Cyber Defense Operations Center, where a team works 24/7 and can call on security professionals, data scientists, product engineers, and customer support experts throughout the company to respond rapidly to security threats.
Once we started to work with FireEye, it was clear that this was not a typical case of a sophisticated attacker breaking into just a few computer networks. The attackers had installed a small piece of malware into the update code of a network management program called Orion, a product of SolarWinds. The Orion software was used by roughly 38,000 enterprise customers around the world. When customers installed the update on their on-premise servers, the malware installed as well. As FireEye reported, the malware would connect to what is known as a command-and-control (C2) server. The C2 server could then give the connected computer “jobs” that included the ability to transfer files, execute commands, profile a system, reboot a machine, and disable system services. This meant the attackers suddenly had a backdoor into the network of every customer that had updated the Orion program.
This approach put at risk the software supply chain across the economy and around the world. The immediate questions became: How many enterprises had installed the Orion update, and hence the malware, on their networks? And how quickly could this backdoor be closed?
At Microsoft we quickly mobilized more than 500 employees to work full time on every aspect of the attack. Other tech companies scrambled into action as well. Given the potential breadth of the incident, Microsoft CEO Satya Nadella convened a meeting each evening of our most senior security leaders to run through the day’s work, what we had learned, and what we needed to do next.
It didn’t take long to appreciate the importance of effective technical teamwork across the industry and with the U.S. government. Engineers at SolarWinds, FireEye, and Microsoft immediately began working together. The teams at FireEye and Microsoft knew each other well, but SolarWinds was a smaller company dealing with a huge crisis, and the teams had to build trust quickly if they were to be effective. The SolarWinds engineers shared the source code for their update with the security teams at the other two companies, which revealed the source code of the malware itself. The technical teams in the U.S. government swung into fast action, especially at the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA) at the Department of Homeland Security.
The digital nature of software and the global reach of the internet quickly came into play. Like a pendulum, they swing in both directions. The attackers had turned code into a weapon that was distributed globally through the internet via the SolarWinds update. But with the source code for the specific malware identified, we had a signature, like a digital fingerprint, that we could look for on desktop and server computers. FireEye published this signature for organizations around the world to access.
At Microsoft, we added this signature in an update to our Windows Defender Antivirus program, which customers use to monitor, identify, and remove malware across an organization’s network and computers.
Many customers also share their Defender data with us, which provides an ongoing picture of where specific malware is installed. Within 48 hours we created a map of the world that lit up every location where SolarWinds’ Orion program had been updated with the malware. The map revealed the broad vulnerability around the world and especially in the United States.
The teams at FireEye and Microsoft worked a bit like 21st-century counterparts to Sherlock Holmes. Each day they added digital clues that could be used to uncover Yttrium’s trail. And with each step they took, they picked up more information to uncover Yttrium’s bad acts and how we might thwart them.
Each evening, John Lambert, MSTIC’s leader, reported on the day’s findings. As he explained, the defensive response to a successful cyberattack always needs to answer two initial questions: How did the attacker gain entry, and what network credentials did the attacker obtain? Until we had the answers to both questions, there was no way we could push the attackers out of the affected networks.
THROUGH THE BACKDOOR
As the security teams studied the infection, they discovered that the malware in the Orion software initially created a backdoor in a company’s network but otherwise lay dormant for two weeks, so as not to create any network log entries that would call attention to itself. It then reported information about the network it had infected to a command-and-control server that the attackers had registered and was being hosted in the United States by the tech firm GoDaddy.
If a target network was of interest to Yttrium, the attackers then took a second step. They entered through the backdoor and installed additional code on the organization’s server, in effect opening a window to connect to a second command-and-control server. This second server, unique to each victim to help evade detection, was registered and hosted in a second data center, often on Amazon Web Services’ (AWS) cloud. As John Lambert explained, Yttrium carefully “cleaned up after itself,” closing the backdoor to GoDaddy and using the open window connecting to a service such as AWS instead. As we identified the customers impacted by these second-stage attacks, teams of specialized Microsoft engineers—our cybersecurity hunters—worked to help customers search for and close these windows.
As the security teams at FireEye and Microsoft studied the source code shared by SolarWinds, they discovered that the code installed on the initial command-and-control server at GoDaddy had a “kill switch” that would automatically shut off the malware on an organization’s server under specific conditions. Armed with this knowledge, the security teams worked together to transfer control of the C2 server from GoDaddy to Microsoft, activate the kill switch to turn off any ongoing or new uses of the malware, and identify any organizations that had computers that continued to ask the server for instructions.
This effectively stopped the attackers from using their malware to enter additional networks. While this type of action often marks the climax of cyber battles, in this case the attackers’ technical sophistication meant that it was just the start of some of the most frenetic work. Because Yttrium had already entered multiple networks and opened new windows, we still needed to identify additional impacted networks, learn what Yttrium was doing inside them, find and close the open windows, and force out the attackers.
As the hunting teams pursued Yttrium, they learned that the attackers typically looked for new ways to drill deeper into the impacted networks. Like intruders inside a house, they turned off the equivalent of any security cameras, such as event logging tools and in some cases antivirus software.
They then began scanning the network for the software keys that would give them access to the home’s most precious possessions. Most often, the attackers looked for the accounts of network administrators who had elevated privileges, meaning access to information across an organization’s network.
They then looked for the passwords for these accounts, which unfortunately some customers had stored in an insecure way that was easy for the attackers to find.
With password in hand, the attackers could move from a server located on premises, such as in an organization’s server room, to its other network assets, including in the cloud. Yttrium typically looked for information that would advance its espionage efforts, including an organization’s email, documents, and other digital assets such as source code or the tools that security experts use to identify and combat potential network intrusions. For some organizations, this included emails and documents in Microsoft’s Office 365 cloud service.
Once MSTIC identified the tactics, techniques, and procedures used by the attackers as they accessed Office 365, our threat hunters could scan our cloud services to identify the telltale patterns that a customer had been compromised. Using this method, we identified 60 customers that had been victimized by the attacks.
DIGITAL FIRST RESPONDERS
This led to the next phase of our response, as security experts worked as digital first responders to help the victims. Microsoft employees notified each of these 60 customers and offered information about the attack and the technical indicators we had identified that would help them start their own investigations.
Given the importance of the SolarWinds intrusions, the responsibility for our largest enterprise customer notifications fell on Ann Johnson, a longstanding cybersecurity leader who previously had led our Detection and Response Team (DART). We referred to DART as “the Microsoft cybersecurity team we hope you never meet” because it contacts customers if they have suffered a cyberattack. In this case, some of our smaller customers took that attitude to heart a bit too literally. After answering the phone and hearing someone explain they were from Microsoft calling about a cyberattack, some responded angrily to what they believed was a hoax.
Naturally, our engineers put an extremely high priority on investigating any potential intrusions into Microsoft’s own network. While at first there was no indication that we had been targeted, deeper digging by more people identified that Microsoft, too, had been a target.
This work revealed a limited presence of malicious SolarWinds code on Microsoft’s internal network, as well as other attempted activities. For example, we found unusual activity with a small number of internal accounts and we discovered one account had been used to view source code in a number of source code repositories. This did not enable the attackers to change any source code, and we found no evidence of access to production services or customer data. The investigation also found no indications that our systems were used to attack others. Ultimately, the implementation of cybersecurity best practices had limited the impact, but the intrusion was sobering nonetheless.
In many respects, Yttrium’s work represented one of the most sophisticated cyber intrusions we had ever seen. The attackers in many instances wrote customized code for a specific network and went to great lengths to cover their tracks. As our experts noted, the work reflected a high level of technical expertise and execution.
But equally notable was the operation’s scale. Yttrium had deployed large teams of engineers who acted with patience and persistence. In many instances, they recognized that the ultimate target, such as a U.S. agency, likely had strong security protection in place. So they began by targeting trusted third parties of these agencies, such as an IT service company that might have access to an agency’s network. Once inside this company’s network, they could seek to identify the account of an employee who had access to a government network and try to obtain that individual’s password. And once equipped with that password, they could then look to jump into a government network itself.
The attackers shrewdly used American data centers to help cloak the attacks. Because the NSA has the authority to scan only foreign activity but not computers in the U.S. itself, we surmised that Yttrium used U.S.-based servers at GoDaddy, AWS, and other smaller U.S. providers to host its command-and-control operations and evade detection.
Put together, all this illustrates the degree to which cyberthreats have intensified around the world. Ultimately, the new attack illustrated publicly what can be accomplished if a government builds a large organization that attracts top-tier technical talent and uses that capability to launch a sustained cyberattack. And it showed how much technology has changed not just the relations between nations but the nature of the tensions—and even hostilities— this can create.
THE HARD WORK AHEAD
As the digital trail led to more information about what had happened, even bigger questions emerged. What did the attack say about network vulnerabilities and the global state of cybersecurity protection? How could the tech sector and the government better protect the country and the world?
By February 2021, congressional committees had summoned witnesses to Washington, D.C., to answer these and similar questions. The Senate Intelligence Committee, led by Senators Mark Warner and Marco Rubio, led off with a lengthy Tuesday afternoon session, where I sat at the witness table next to FireEye’s Kevin Mandia and SolarWinds’ new CEO, Sudhakar Ramakrishna. (They also summoned Amazon, but it refused to participate.) The three of us took turns answering questions in person, an exercise we repeated three days later in a virtual joint hearing before the House Homeland Security and Oversight committees. Both hearings examined not only what had happened but also what steps needed to be taken to prevent such attacks in the future.
An initial conclusion is that the world needs to modernize technology infrastructure and broaden the use of cybersecurity best practices. This includes work by the companies that create software to better harden the software “build process” and every part of the software supply chain, to help prevent the insertion of malware into a software update.
The hearings on Capitol Hill hammered home that it will take governments and tech companies working together to secure the world’s digital infrastructure. While this can start with stronger protection for government networks themselves, it must reach well beyond the public sector. We need to broaden awareness and encourage expanded adoption of cybersecurity precautions, and tech companies like Microsoft must make it simpler and easier for customers to understand and use the security protections we create.
As we look to the future, it’s apparent that the next decade will be defined in part by issues like international norms for governments and practical steps to strengthen our cybersecurity defenses. Technology in some respects has created a more dangerous world. A country like the United States can no longer rely on large oceans to separate it from its rivals. The internet has made everyone each other’s next-door neighbor. And software that can be used for espionage can equally be used as ransomware or a weapon to disable a nation’s electrical grid or water supply. Ultimately, it’s easier to send code into battle than troops and missiles.
None of this changed overnight or because of a single development. But with successive changes over the past decade, two things have become clear: We live in a world remade by technology, and we must grapple with the consequences for the new world we have created.
This excerpt is adapted from the new paperback edition of The New York Times bestseller Tools & Weapons: The Promise and Peril of the Digital Age, available Sept. 7, 2021.
Recognize your brand’s excellence by applying to this year’s Brands That Matter Awards before the early-rate deadline, May 3.
