Sweating In the Hot Zone

Imagine what life would be like if your product were never finished, if your work were never done, if your market shifted 30 times a day. The computer-virus hunters at Symantec don't have to imagine.

Patrick Martin opens a door marked "Response Lab" and enters what looks like a typical computer room, full of tall black racks, blinking lights, industrious system administrators, and the wash of white noise generated by whirring computer fans and intense air-conditioning.

But this room, located in an office complex in Santa Monica, California, nowhere near the beach, is no benign, garden-variety data center. It may be the nastiest, scariest room in the computer industry. It's where Symantec Corp. tests out every update to its antivirus software to make sure it will block newly discovered security threats. To do that testing, Symantec not only needs copies of all the software the company produces for every sort of computer, but it also needs a sample of nearly every virus, Trojan horse, and worm--nearly 200,000 of them--that has ever crawled across the Internet.

What the Plum Island Animal Disease Center is to virulent pathogens, Symantec's Response Lab is to computer viruses. The software update being tested today for eventual distribution will protect Symantec customers from a new variant of the W32.Sober mass-mailer worm, which travels by email attachment and sends itself to all of the people listed in the recipient's address book. It has been labeled a category-three threat (five is the highest).

"This is the dirtiest of all of our networks at Symantec," says Martin, a senior product manager. "There are special firewalls that protect these machines." And by the door, there's a Hazmat box marked danger. It's for disposing of disks, tapes, and even hard drives, so any viruses they may contain aren't inadvertently released. Explains Martin: "No storage media ever comes out of this room. It can go in, but it can't come out."

The Response Lab is part of Symantec's Security Response Center, which houses a collection of investigators who collect viruses and other malicious code, autopsy them to figure out how they work, and then develop updates to the Symantec software that protects computers at homes and in workplaces around the world.

In Symantec's business, a product is never finished--not after the development team hits its last deadline, not after the quality-assurance crew has hammered on it, not after the manuals have been written and the CDs cranked out. "You can't go out and tell all the bad guys, 'Stop developing nasty stuff, because we don't have another product release for six months,' " says Dave Cole, director of product management. "You've got to be nimble. You have to respond fast." Symantec may be an extreme case, but it also offers a vision of the future, even for seemingly more placid businesses. Given the rapid pace of technological change, the quickly shifting nature of consumer expectations, and the constant emergence of new global competitors, product life cycles have shortened in many industries. (Apple, for instance, has rolled out eight versions of its iPod in the four years since its introduction.) And that means many companies could someday find themselves living in this state of perpetual, real-time product development.

Here's the frenetic pace at which that happens at Symantec. A new set of "signatures"--essentially, the company's version of the Most Wanted list that tells computers how to identify and block incoming security threats--is created about 30 times a day. That means an hour doesn't go by when Symantec's products aren't evolving to try to better protect users. And when new threats emerge--like the Zotob worm that struck in mid-August and shut down computers on Capitol Hill and at The New York Times and CNN, among others--Symantec goes into what Martin calls "adrenaline mode." Staffers at the Security Response Center and Symantec locations around the world--from Sydney to Tokyo to Dublin to Taipei to Calgary--race to figure out how the threat works and then create a signature as quickly as possible. "There's no question," says Symantec chairman and chief executive John Thompson, "that you're only as good in business as your last response to an attack. You always have to be one step ahead of the bad guys."

And the bad guys have been pretty busy.

Average time to recover fully from a virus disaster in 2004: 31 days
Average time to recover fully from a virus disaster in 2003: 24 days*

It wasn't always this way. When Vincent Weafer, a soft-spoken and imperturbable native of Ireland, took the reins in Santa Monica in 1999, "there were less than two dozen people, and the group was this nice little research group, looking at the future of security," he says. "Nothing really happened. We'd see maybe five new viruses a day, and they would spread in a matter of months, not minutes."

Then a new generation of online epidemics, such as Melissa, Slammer, Nimda, and Code Red, began to spread more quickly, forcing Symantec to adopt more of a firefighting mentality and begin to establish offices around the world so it could respond at any hour of the day. The response center's staff grew to hundreds; Weafer estimates that his staff is about 10 times what it was in 1999. ("We don't give out the actual number, because you've got people who'll try to use it for intelligence," he says cryptically.)

The average dollar cost of a virus disaster in 2004:: $130,000
The average cost of a vrius disaster in 2003: $99,000*

Symantec's fast responses have helped it corner the market for antivirus software sold to consumers; according to the NPD Group, a research firm, it had nearly 85% of the market earlier this year, compared to runner-up McAfee's 12%. But Microsoft announced plans to start competing with Symantec in May and began unveiling an antivirus offering of its own in July. And things are more competitive in the corporate world, where, according to a 2003 report by another research firm, IDC, Symantec has 28.5% of the market, compared to McAfee's 23.9%. IDC expects the $8 billion market for security software--corporate and consumer--to double by 2008.

These days, about 20,000 virus samples--not all of them represent unique viruses--come in to Symantec every month. New strains propagate in clever ways: over instant-messaging software, peer-to-peer file-sharing systems such as LimeWire, and even wireless Bluetooth connections between cell phones. And where yesterday's security threats were nasty enough when they erased hard drives and crashed Web sites, now the creators of malicious code are often hunting for credit-card numbers and other personal information they can use in criminal enterprises. Symantec also expanded the scope of the response center's responsibilities to include spam, pop-up ads, spyware, and adware.

The people who work on the response-center team are an eclectic group, and they weren't easy to find. "It's not as if colleges are creating thousands of anti-malware or security experts every year that we can hire," Weafer says. "If you find them in any part of the world, you just go after them." One senior researcher, Peter Szor, came from Hungary; Sarah Gordon, who profiles virus writers to try to understand their motivations, works out of her home in Melbourne, Florida; Peter Ferrie, an expert at disassembling viruses to see what makes them tick, came to Santa Monica from Iceland. "The people we look for are the kind of people who aren't necessarily creating new products, but they like to take things apart and break them," Weafer says. "Give them a Rubik's Cube and they'll have it disassembled in five minutes. They're motivated by solving problems."

Weafer and six of his lieutenants are sitting around an oval Formica conference table for a weekly security briefing; several remote locations are looped in by speakerphone. There's a rapid rundown of the threats everyone's dealing with, and that provides an opportunity for Weafer to ask lots of questions and make sure the various sites are acting in sync.

Denise Bellotti at Symantec's anti-spam unit in San Francisco reports that the lab has identified a new tactic among phishers--con artists who send emails that link recipients to an official-looking Web site in an attempt to elicit credit-card numbers and passwords. "They're using a scout message first to establish credibility, and then they're sending a second message with the attack," Bellotti says from the speakerphone. In other words, a seemingly innocuous first contact--or scout message--is followed by the con. "That was a new one that came out over the last couple of days," she says.

Javier Santoyo, a senior researcher, brings up the subject of "kernel mode root kits," a particularly insidious Trojan horse that burrows so deeply into a computer that the operating system itself can't see it. (A "Trojan horse" is a program that masquerades as a helpful application so users will install it, but then creates a secret backdoor that allows the sender to access the computer.)

"Are we still at the leading edge of this threat?" Weafer asks. "I mean, is it a fringe thing, or are we seeing it accelerate?"

"I think it's going to become more and more common," Santoyo says. "These guys are good. They know the quirks of the operating system."

Mark Kennedy, a software architect, chimes in: "And our problem is that when you go that deep to try to extract it, you can render the machine unusable. Then you get blamed, rather than the bad guy." Though Weafer estimates that root kits constitute less than 1% of viruses out there, he says, "the bad actors are getting really bad."

The day never ends for Symantec employees charged with outsmarting those bad actors. Every afternoon at 5 p.m., the crew in Santa Monica passes the baton to colleagues in Tokyo, meaning that they become responsible for new threats that appear--and for taking the lead on lingering older threats. "From 5:00 to 5:30, it's the U.S. team's job to brief the Tokyo team," Weafer says. "And in the second half of the hour, the Tokyo team is effectively in control, but they can draw on the U.S. team." At the end of the Japanese workday, Tokyo hands off to Dublin, and at 8 a.m. in California, the baton returns to Santa Monica.

Work in progress shifts smoothly from one continent to the next, as in August when the Santa Monica team was investigating an issue for customers in the United States and Europe. "Keylogger" software had been capturing information typed by computer users and sending it to an unknown source. "When we transferred the issue to Japan, that handover included the technical knowledge of what our researchers in Santa Monica had found, the tools we were developing, and the contact points with our customers," Weafer says. "Our job is not to drop anything."

Other companies, such as McAfee, Sophos, and Kaspersky Lab, have their own antivirus troops, of course, and they're avid rivals. "We compete on response time--who saw a virus first and how fast did you get the solution to customers," says Vinny Gullotto, vice president of McAfee's antivirus and vulnerability emergency-response team, its counterpart to Symantec's response center. But he notes, too, that these virus hunters also cooperate to address fast-moving, global threats. In that sense, they resemble a community of scientific researchers, says Shane Coursen, a senior technical consultant at Moscow-based Kaspersky. (Coursen works in Nevada.) "We talk about things that are happening in the virus and malware world, like the best way to counter a particular threat," Coursen says. Security researchers at rival companies even maintain mailing lists--some of which are kept private to prevent virus writers from infiltrating them--that they use to exchange ideas and even virus samples among themselves, he says. "But we don't get down and dirty and talk about individual lines of code in our products."

When malicious code could cause serious damage, Symantec and its peers often provide information to law-enforcement agencies such as the FBI, the U.S. Secret Service, and the Royal Canadian Mounted Police. "If they need an analysis done on a new threat, we'll help them," Weafer says. "We provide them with intelligence that we have, but we don't chase criminals."

Symantec does spend a lot of time trying to figure out just what makes those criminals tick. Understanding their motivations and personalities can give an edge to researchers responsible for dismantling new viruses and predicting what may be coming. "Usually, the virus writer is a young person who doesn't recognize the impact of what they're doing," says Gordon, Symantec's profiler and senior research fellow. "The motivation varies with the individual. It can be revenge, the technical challenge, or the desire for notoriety. There's a generational problem here, where a lot of young kids don't realize that what they're doing when they're on the computer can have an effect on the real world."

That was exactly what happened in August 2003, when an 18-year-old set the Blaster-B worm slithering across the Internet. This worm caused computers to launch an attack on a Microsoft Web site and also created a backdoor for stealing information from infected machines.

It was part of what Weafer refers to as the "week from hell," when three major threats surfaced simultaneously to test the response center's mettle. While Blaster-B, a variant of an earlier worm, was instructing computers to crash a Microsoft Web site, the Welchia worm sought out computers that were infected with Blaster, deleting the file, repairing the operating system, and rebooting the computer. But in the process, Welchia created disruptive traffic on the Net. SoBig.F was a mass-mailer worm that looked for email addresses in the recipient's address book and then sent copies of itself to others, with subject lines such as "Re: Details" and "Re: Your application."

"It was the first time we'd seen three back-to-back category three or above threats hit us," says Alfred Huger, Symantec's senior director of engineering for security response. "The possible number of machines that could've been affected was astronomical. That was one of the things that drove us fairly hard."

Symantec researchers hurried to understand what made the threats tick, develop signatures for them, and educate customers about the importance of updating their antivirus software. But the particular nature of the viruses presented challenges. "Blaster forced systems to reboot, and if your system keeps rebooting, it's difficult to update your antivirus software," Huger says. "Welchia saturated networks, pinging computers all over the place, which added insult to injury. SoBig was an extraordinarily prolific mass mailer that turned your computer into an unwitting spam machine, and it spread tremendously quick." Many staffers remained at work around the clock for two weeks, Weafer says, and all the more-routine functions of the office--staff meetings, mandatory reports--were waived.

August 2003 forced Symantec to reevaluate its assumption that it would need to deal with only one threat at a time. That led to significant changes in the company's thinking about the depth and quality of its staffing. "You have to make sure you have competent, trained reserve staff waiting on the bench," Huger says, "because the frontline analysts can only work so long before they start to burn out."

With a deep bench, security-response managers now rotate people from the front lines, where they're responsible for responding to new security threats that crop up, into groups where they can help with new-product development, for example. Others write internal research papers. Still others are assigned to develop new tools that will help their colleagues battle the next wave of threats. "There are lots of opportunities for people to do something different and contribute in another way," says Martin.

When Weafer describes his long-term objectives at the response center, he uses phrases like "trying to take the chaos out" and "making the exciting boring." That means spreading work evenly to facilities around the world to reduce the number of all-nighters at Santa Monica or another site, along with a predictable and well-defined process for responding to threats.

But employees might be loath to work in an environment entirely drained of excitement. It's a great responsibility to be deconstructing threats on the fly and instantly devising shields.

The team in Santa Monica has lately been speculating about the ways that bad guys might attack game consoles such as Microsoft's Xbox or Internet-connected set-top boxes. "That means there's always something new to learn. That continuous learning forces you to stay sharp. And I think it's a big part of what people like about working here," Martin says.

Sidebar: Rating Critical Security Threats

(On a scale of 1 to 10, with 1 being "no concern at all" and 10 being "extremely concerned")

  • Viruses and worms: 7.6
  • Outside hacking and cracking: 7.1
  • Identity theft and phishing: 7.0
  • Spyware: 6.8
  • Denial of services:6.6
  • Spam: 6.3
  • Wireless and mobile device viruses: 6.2
  • Insider threats: 6.2
  • Cyberterrorism: 5.6

Gartner Information, Security and Risk Research (May 2005)
Survey of IT managers and departments at North American organizations with global operations and revenue exceeding $750 million.

Scott Kirsner (skirsner@fastcompany.com) is a Fast Company contributing writer who covers technology from San Francisco.

Add New Comment

0 Comments