This was no ordinary botnet.
On a February day in 2017, Tamer Hassan was going about his typical work of monitoring advertising-buying and -selling software for potential security issues when he noticed something strange. Hassan, the cofounder and then CTO of ad-fraud detection and prevention company White Ops, had been tracking a smallish botnet—the term for a network of private computers infected with malicious software and controlled by criminals without the owners’ knowledge—and realized that it had suddenly transformed into a hydra that simply wouldn’t die.
When deployed in advertising, a botnet (“robot” + “network”) creates fake websites and uses automated software to pose as real humans and simulate real traffic—siphoning money from companies such as P&G, Unilever, and other big marketers that spend more than $250 billion annually on digital advertising globally, much of it mediated by software that places ads programmatically.
This one had looked normal enough at first. But any time that Hassan and his fellow engineers attempted to block a fraudulent site from attracting programmatic ads—or restrict any IP addresses that appeared to generate fake clicks on the site—they saw the same activity pop up somewhere else. And there wasn’t a pattern: One day the botnet would use someone’s computer for malicious activity, but the next, that same PC would act normally. Worse, it all now seemed to be accelerating and growing more powerful.
Brands are cheated out of somewhere between $6.5 billion and $19 billion annually in digital ad-fraud schemes, but the criminals are rarely held to account, making these scams some of the lowest-risk, highest-reward forms of crime in the world. The consequences, though, extend much further than marketers’ ad budgets. In an era when everything from dishwashers to presidential elections relies on digital integrity, fighting ad fraud is about reinforcing trust in the internet itself. The money these criminals steal “is funding the spread of malware,” says Hassan. It creates “platforms for many other types of cybercrime,” such as identity theft, ransomware, spyware, computer viruses, and more—the type of IT security threats that caused massive data breaches at Equifax, Marriott, and Yahoo in recent years. A 2018 report by McAfee and the Center for Strategic and International Studies estimated that cybercrime costs businesses as much as $600 billion annually.
Raised in the Bay Area, Hassan taught himself to code after receiving a Tandy computer when he was 8. While he was studying engineering at the Air Force Academy in Colorado, the 9/11 terrorist attacks happened, and “I knew that I had to use everything I learned to protect others,” he says. He spent the next several years as a combat search-and-rescue pilot flying Pave Hawk helicopters to save military and civilian lives behind enemy lines in Iraq and Afghanistan, and the experience “shaped how I approach any challenge,” he says. “The only thing certain in a rescue mission in hostile territory is the objective. The rest is fraught with ever-changing variables, opportunities, and trade-offs.”
After 12 years in the Air Force, Hassan decided to combine his coding background with his military experience, and launched White Ops in 2012 with two cofounders. The company has grown to more than 100 people, with headquarters in New York and offices around the world, and it has raised $33 million to date. Its technology operates like a burglar alarm for clients such as The Trade Desk, a company that helps marketers manage and serve programmatic digital advertising. White Ops alerts clients to any new fraud attempts and immediately prevents criminals from collecting on bogus ad views. The company was founded on the thesis that security should be about more than just building a wall to protect clients; it should work to make crime tougher and more expensive to pull off. This is why instead of simply asking a human to solve a captcha (typing numbers and letters to prove you’re not a robot—a tool that can be scammed at scale), White Ops created tech that interrogates a bot a thousand different ways, such as subtle timing differentials when code is executed that can reveal whether it’s being directed by human or machine.
And yet this mysterious botnet—which Hassan and his White Ops programmers had dubbed “3ve” (pronounced Eve), in part because of its three primary attributes of speed, scale, and sophistication—was yielding no answers. In just a few months it had grown increasingly monstrous, creating more than 10,000 counterfeit websites and churning out up to 40,000 new IP addresses per day to generate fake traffic and reap the ad revenue. In addition, 3ve had infected and compromised more than 1.7 million devices owned by everyday users and corporations, the equivalent of enlisting the population of Phoenix in its criminal enterprise.
3ve, it turned out, was bypassing White Ops’s security software altogether. The perpetrators were aware of the company’s defensive maneuvers. As Hassan would later learn, “We were actually written in the [3ve] code,” meaning that the criminals had programmed their botnet to avoid it. Each time he and his team tried to bolster a client’s defenses, the botnet seemed to be learning more about how White Ops worked—and masking its own moves accordingly.
Typically, Hassan says, tech companies are averse to working in concert with law enforcement, even when contending with obvious potential crimes, because they don’t want to be seen as an agent of the government. In addition, tech companies are usually too competitive to seek help from one another. But given the nature of 3ve, White Ops reached out, rounding up a posse of industry players—including Google, which it realized was having its own problems with 3ve—to work with the FBI to bring it down. “Tamer was the linchpin of the operation,” says Verizon Media’s head of trust and safety, Bennet Manuel, who was part of this coalition, referring to the “unprecedented collaboration” among parties.
As the task force would discover, 3ve was able to detect whether computer owners had anti-malware or antivirus software activated—and then avoid those devices. It shunned computers in regions like San Francisco, because of the higher odds of detection by concentrations of tech-savvy locals. Once it found vulnerable computers, only then would it connect with its mission control—more than 1,900 computer servers housed in commercial data centers in Dallas and other locations. At this point, 3ve would open a hidden browser to the unsuspecting computers and start its work by loading ads on more than 5,000 fake websites it created. The servers were programmed with bots to mimic human behavior, using a simulated mouse to scroll down phony web pages being viewed in the fake browser. It would start videos and stop them midway through. The only thing that was real were the ads, and the marketers were losing.
Hassan tried to cut off 3ve wherever he could. He worked with The Trade Desk to check every biddable ad impression in real time to make sure a human would actually see the ad that was served. He armed leading providers for video, mobile, native, and even connected-TV advertising with White Ops’s own pre-bid tech, called Media Guard. The more places Hassan could use White Ops’s predictive analytics to prevent an ad from being delivered to a bot, the more he could frustrate 3ve’s proliferation.
Early on the morning of October 22, 2018, after more than 18 months of observing 3ve, documenting its behavior, and helping the FBI build a case by tracking it (all while trying to block it where he could), Hassan got the call: The first arrest had been made. Hassan sent excited, emoji-filled messages to his other partners on an encrypted text channel. Over the next month, the FBI charged three of the eight alleged perpetrators—located in Malaysia, Bulgaria, and Estonia—with crimes including wire fraud, computer intrusion, aggravated identity theft, and money laundering. The other five remain at large.
Hassan, who became CEO of the company in April, has spent the past several months working with his team to dismantle 3ve’s tech infrastructure and learn more about it in advance of the next battle, which will inevitably come. He can’t delineate specific lessons he’s learned from 3ve because of the ongoing criminal proceedings, but when pressed, he’ll note that the experience confirms White Ops’s philosophy. “You can’t play defense. You have to play offense,” he says. “This makes the fight actually winnable.”