Can Microsoft Finally Kill All The Bugs?

Triage Your Failures

Do we ship, or do we slip? It's a question that preys on every product manager as the coding approaches its final months of development and testing and the do-or-die deadline looms large. What causes a product release to slip? In a word, bugs. It's simple probability: Bugs will inevitably infest a product like Exchange, with its 6 million lines of code. During the final stretch, when Exchange 2003 was being readied for shipment to manufacturers and was supposedly at its most stable, testers were fixing roughly 500 bugs a month. DeMichillie reports that during a product's three-year development cycle, "tens of thousands" of bugs are typically discovered in testing and deployment.

Not all bugs are created equal. When a bug is discovered, it is dropped into a database, called Product Studio (previously known by the more colorful moniker "RAID") that enables program managers to track it and ensure that it's routed back to the appropriate developer. Once it's entered into the database, a bug is sorted and evaluated. Bugs that have a high likelihood of crashing the product are given the highest priority rating.

The goal is to zap every single "priority one" bug and kill off as many low-priority bugs as possible. But the bug count will never hit zero. Not every bug that's found can be fixed because there's not enough time. "As the ship date looms, you've got to stop making changes and let the product settle down," says DeMichillie.

"The pressure builds to postpone bugs. You start to tell yourself, 'This bug's not that bad. We'll get it the next time.' "

The Data Does the Talking

On June 30, the 500 members of the Exchange development team threw themselves a party in the parking lot behind Building 34 on the Redmond campus. Exchange 2003 was out of the building and on its way to the manufacturers, the culmination of a three-year effort. Gates dropped by and thanked people for doing a great job; the pervasive feeling was one of exhilaration, not exhaustion. "We can say with tremendous confidence that this is the highest-quality release of Exchange to date," says Missy Stern, a product manager for the Exchange marketing team. "The reliability of this product is unparalleled. We're feeling extremely good about it."

Stern is staking her claim on the data. Microsoft had over 50 partner companies run Exchange 2003, deploying the software on 170,000 workstations over five-week periods. Within Microsoft, Exchange nailed the "three nines" before it was shipped: 99.95% of uptime for six weeks, running on 100% of Microsoft's mail servers. The company ran weekly surveys of its dog fooders, the first time it had polled users during development. The operations and technology group continued testing right up until the code was released to manufacturing. As far as Microsoft is concerned, Exchange 2003 has passed its quality test.

Or has it? Just 16 days after the release party, Microsoft issued a patch for a critical flaw that was discovered in Exchange 2003. Windows XP, 2000, and NT 4 were also affected, which led analysts to conclude that the squirrelly code went undetected for years, despite the most intensive round of testing in Microsoft's history. Then came mid-August's Blaster, another computer pox that attacked a different vulnerability in Windows and proceeded to slow worldwide Internet traffic. When security sleuths dug into the code, they discovered a taunt: "billy gates why do you make this possible? Stop making money and fix your software!"

Taken together, the flaws show that "Microsoft's marketing claims might be setting up false expectations in people," says DeMichillie. "There's no silver bullet for eliminating every mistake, and Microsoft's developers would be the first to admit this." Still, there are signs that Microsoft's drive for quality, even at the expense of deadlines, is in earnest. In late April, the company said it would delay the launch of Office 2003, originally set for this past summer. It now comes out October 21. The reason: The folks in Redmond had taken an extra three months to finish testing the software with users.

Bill Breen (bbreen@fastcompany.com) is a Fast Company senior writer.