Can Microsoft Finally Kill All The Bugs?

The Process Is in the Plan

For most of its history, Microsoft wanted to be the anti-IBM -- a nimble, creative place where programmers could have a neat idea one day and get it into the product the next, without cumbersome approvals and documentation. But there was a problem with that loose-limbed approach. Says DeMichillie: "As its products grew more complicated, Microsoft discovered that even though IBM had its failings, its processes were there for a reason -- to ensure predictability, stability, and quality."

Microsoft's development teams are now adopting some of the IBM-like processes that they once abhorred. Case in point: the product blueprint -- detailed design documents that programmers now work from. The blueprint for Office 2003 consists of roughly 4,000 product specs, each of which amounts to a 30-to-50-page document that describes in rigorous detail a spec's features, how it works, and how it must be tested. In theory at least, such rock-solid product design results in cleaner code and fewer bugs. "The blueprint keeps us from getting random," says Eric Levine, group program manager and an 11-year veteran of the Office team. "We create off of it, we code off of it, we cool down, and we test. That's the process."

But it's still unclear whether developers rigorously stick to the plan. On the one hand, in the latter stages of Exchange's development, programmers must file a change request when they want to rejigger the code, and managers meet daily in a war room to make their calls on the requests -- just as at IBM. A deep database underlies Exchange's staggering 6 million lines of code, and records every single alteration -- just as at IBM. But programmers are still encouraged to get creative and take risks. "We give a lot of freedom to our developers for a reason -- they're smart people," says Betsy Speare, the release manager for Exchange. "We're willing to make a mistake and catch it, so long as we don't miss an opportunity."

In other words, Microsoft is trying to be both rigorous and nimble. "They want to be able to react quickly to competitors and market changes," says DeMichillie. "So they're trying to have it both ways. Can they succeed? No one has before."

Make Your Company Your Customer

Given his combination of business smarts and deep technical knowledge, Bill Gates is uniquely positioned to run a hard-hitting product critique. And he does so in something called a Bill Review -- a two-to-three-hour meeting with the leaders of a product team in which he weighs in on the design, features, and strategy for an upcoming release. Still, the agenda for a Bill Review is at least partially driven by what the team expects will catch his interest, such as the overall product architecture. The blocking and tackling of a launch -- testing and tracking bugs -- have never been high on Gates's radar, and a Bill Review can miss potential traps, such as the flaws that were ultimately exploited by Code Red. But his review of Exchange 2003 was different; Gates hammered away at the need to ensure quality and security. As Speare walked out of the meeting, she recalled that Gates's challenge to the team seemed to hang in the air: "Make it work, and make it great."

It's doubtful that anyone takes that mission more seriously than Devenuti, a quietly intense Microsoft lifer. As the boss of Microsoft's own worldwide IT operations, Devenuti must deal with some of the most demanding, technoliterate people on the planet. IT isn't Devenuti's only worry. He also heads up the 3,000-person operations and technology group, which is the first to set up, launch, and weigh in on a new product before it's shipped to the outside world. All of which makes Devenuti Microsoft's first adopter -- and at times, the most unpopular man in Redmond.

Here's why: When Exchange 2003, code-named Titanium and still in beta, rolled out of the development and testing lab in September 2002, it rolled right into Devenuti's shop. His team put the not-ready-for-prime-time software up on a group of servers that were isolated, or "forested," from the rest of Microsoft's global network. The wobbly code crashed servers, temporarily shutting off many users' email. At Microsoft, as elsewhere, tempers go up when servers go down. "People start screaming, 'What the hell's wrong with you guys?' " says Devenuti.

When new code runs amok, a beta version is pushed back to the lab, where its bugs are zapped, and it's then redeployed to an ever widening number of servers. The process is called "dog fooding" -- as in eating your own dog food. Before Exchange is shipped to more than 100 million users, Microsoft will use it on itself. Says Devenuti: "Microsoft's employees are Microsoft's first and best customers."

Other high-tech companies dog-food their wares, but it's un-likely that any company does it on Microsoft's scale. "Right now, I've got 50,000 people using the next version of Office, which lets us say to customers that we've deployed it and we've lived on it," Devenuti says. "Dog-fooding gives the product a level of proof and validity that testing alone could never attain."

Dog-fooding isn't a perfect solution, however. After all, the dog food is consumed in a homogenous environment at Microsoft. Each piece of software is up-to-date and compatible with every other piece of software, rarely the case in the real world. And when Exchange 2003 crashes one of Microsoft's mail servers, Devenuti can summon an Exchange developer to debug it. You can't.

"It's not yet clear that quality really is a top priority across all of Microsoft's lines of business."