Sweating In the Hot Zone

Symantec researchers hurried to understand what made the threats tick, develop signatures for them, and educate customers about the importance of updating their antivirus software. But the particular nature of the viruses presented challenges. "Blaster forced systems to reboot, and if your system keeps rebooting, it's difficult to update your antivirus software," Huger says. "Welchia saturated networks, pinging computers all over the place, which added insult to injury. SoBig was an extraordinarily prolific mass mailer that turned your computer into an unwitting spam machine, and it spread tremendously quick." Many staffers remained at work around the clock for two weeks, Weafer says, and all the more-routine functions of the office--staff meetings, mandatory reports--were waived.

August 2003 forced Symantec to reevaluate its assumption that it would need to deal with only one threat at a time. That led to significant changes in the company's thinking about the depth and quality of its staffing. "You have to make sure you have competent, trained reserve staff waiting on the bench," Huger says, "because the frontline analysts can only work so long before they start to burn out."

With a deep bench, security-response managers now rotate people from the front lines, where they're responsible for responding to new security threats that crop up, into groups where they can help with new-product development, for example. Others write internal research papers. Still others are assigned to develop new tools that will help their colleagues battle the next wave of threats. "There are lots of opportunities for people to do something different and contribute in another way," says Martin.

When Weafer describes his long-term objectives at the response center, he uses phrases like "trying to take the chaos out" and "making the exciting boring." That means spreading work evenly to facilities around the world to reduce the number of all-nighters at Santa Monica or another site, along with a predictable and well-defined process for responding to threats.

But employees might be loath to work in an environment entirely drained of excitement. It's a great responsibility to be deconstructing threats on the fly and instantly devising shields.

The team in Santa Monica has lately been speculating about the ways that bad guys might attack game consoles such as Microsoft's Xbox or Internet-connected set-top boxes. "That means there's always something new to learn. That continuous learning forces you to stay sharp. And I think it's a big part of what people like about working here," Martin says.

Sidebar: Rating Critical Security Threats

(On a scale of 1 to 10, with 1 being "no concern at all" and 10 being "extremely concerned")

• Viruses and worms: 7.6
• Outside hacking and cracking: 7.1
• Identity theft and phishing: 7.0
• Spyware: 6.8
• Denial of services:6.6
• Spam: 6.3
• Wireless and mobile device viruses: 6.2
• Insider threats: 6.2
• Cyberterrorism: 5.6

Gartner Information, Security and Risk Research (May 2005)
Survey of IT managers and departments at North American organizations with global operations and revenue exceeding $750 million.

Scott Kirsner (skirsner@fastcompany.com) is a Fast Company contributing writer who covers technology from San Francisco.