Sweating In the Hot Zone

Patrick Martin opens a door marked "Response Lab" and enters what looks like a typical computer room, full of tall black racks, blinking lights, industrious system administrators, and the wash of white noise generated by whirring computer fans and intense air-conditioning.

But this room, located in an office complex in Santa Monica, California, nowhere near the beach, is no benign, garden-variety data center. It may be the nastiest, scariest room in the computer industry. It's where Symantec Corp. tests out every update to its antivirus software to make sure it will block newly discovered security threats. To do that testing, Symantec not only needs copies of all the software the company produces for every sort of computer, but it also needs a sample of nearly every virus, Trojan horse, and worm--nearly 200,000 of them--that has ever crawled across the Internet.

What the Plum Island Animal Disease Center is to virulent pathogens, Symantec's Response Lab is to computer viruses. The software update being tested today for eventual distribution will protect Symantec customers from a new variant of the W32.Sober mass-mailer worm, which travels by email attachment and sends itself to all of the people listed in the recipient's address book. It has been labeled a category-three threat (five is the highest).

"This is the dirtiest of all of our networks at Symantec," says Martin, a senior product manager. "There are special firewalls that protect these machines." And by the door, there's a Hazmat box marked danger. It's for disposing of disks, tapes, and even hard drives, so any viruses they may contain aren't inadvertently released. Explains Martin: "No storage media ever comes out of this room. It can go in, but it can't come out."

The Response Lab is part of Symantec's Security Response Center, which houses a collection of investigators who collect viruses and other malicious code, autopsy them to figure out how they work, and then develop updates to the Symantec software that protects computers at homes and in workplaces around the world.

In Symantec's business, a product is never finished--not after the development team hits its last deadline, not after the quality-assurance crew has hammered on it, not after the manuals have been written and the CDs cranked out. "You can't go out and tell all the bad guys, 'Stop developing nasty stuff, because we don't have another product release for six months,' " says Dave Cole, director of product management. "You've got to be nimble. You have to respond fast." Symantec may be an extreme case, but it also offers a vision of the future, even for seemingly more placid businesses. Given the rapid pace of technological change, the quickly shifting nature of consumer expectations, and the constant emergence of new global competitors, product life cycles have shortened in many industries. (Apple, for instance, has rolled out eight versions of its iPod in the four years since its introduction.) And that means many companies could someday find themselves living in this state of perpetual, real-time product development.

Here's the frenetic pace at which that happens at Symantec. A new set of "signatures"--essentially, the company's version of the Most Wanted list that tells computers how to identify and block incoming security threats--is created about 30 times a day. That means an hour doesn't go by when Symantec's products aren't evolving to try to better protect users. And when new threats emerge--like the Zotob worm that struck in mid-August and shut down computers on Capitol Hill and at The New York Times and CNN, among others--Symantec goes into what Martin calls "adrenaline mode." Staffers at the Security Response Center and Symantec locations around the world--from Sydney to Tokyo to Dublin to Taipei to Calgary--race to figure out how the threat works and then create a signature as quickly as possible. "There's no question," says Symantec chairman and chief executive John Thompson, "that you're only as good in business as your last response to an attack. You always have to be one step ahead of the bad guys."

And the bad guys have been pretty busy.

Average time to recover fully from a virus disaster in 2004: 31 days
Average time to recover fully from a virus disaster in 2003: 24 days*

It wasn't always this way. When Vincent Weafer, a soft-spoken and imperturbable native of Ireland, took the reins in Santa Monica in 1999, "there were less than two dozen people, and the group was this nice little research group, looking at the future of security," he says. "Nothing really happened. We'd see maybe five new viruses a day, and they would spread in a matter of months, not minutes."