Fast Company

Can Microsoft Finally Kill All The Bugs?

Viruses, flaws, and worms, oh my! With PCs crashing and the Internet
wheezing, Gates & Co. are on the quality hot seat. We'll take you inside
Microsoft's effort to get its software right, right from the start.

From: Issue 75 | October 2003, Page 82 By: Bill Breen Photographs by: Dan
Lamont Illustrations by: Fiona Hewitt

It was the worm that turned Microsoft. In early July 2001, a malicious
piece of computer code squirmed into a Web server running on Microsoft's
Internet Information Server software. It quickly propagated across the
Internet and into at least a quarter million other servers, knocking an
untold number of Web sites offline. The computer worm, dubbed Code Red,
slowed Internet traffic to a crawl and cost companies billions in fixes and
lost productivity. Some computer-security analysts called it the most
damaging Web-server virus ever. As it turned out, its symptoms were
fleeting; within a month, the fever had broken.

But because it had exploited a known flaw in Microsoft's Internet server
program, Code Red caused the Redmond, Washington, giant to turn red with
embarrassment, especially when it was revealed that Microsoft's own MSN
Hotmail servers were also infected.

Code Red proved to be a defining moment for Microsoft. Coupled with other
quality concerns and computer viruses, it prompted some industry analysts
to renew their periodic warnings against using newly released (and
therefore risky) Microsoft products. Those warnings came at a time when
Microsoft was going head to head with IBM and Oracle in the rich
server-software market for large corporations, where security and
reliability are deal- breaking priorities.

"Customers were concerned whether Microsoft products could be trusted to be
available whenever they needed them," says Michael Cherry of Directions on
Microsoft, a firm that analyzes the company's products and strategies.
"These trust- related issues threatened to damage Microsoft's OS and Office
business, hamper the company's drive to increase server-product sales, and
impede its expansion into consumer services."

The worries were crystallized in a January 2002 memo that chairman and
chief software architect Bill Gates emailed to Microsoft's 50,000
employees. Gates declared that a companywide initiative, Trustworthy
Computing, would unfold over the next 10 years. The goal: to make computing
as reliable, dependable, and secure as a phone's dial tone. The days of
rushing out imperfect software were over. Cool features and tight deadlines
would no longer drive product rollouts; quality and security were now the
top priorities. "Flaws in a single Microsoft product . . . not only affect
the quality of our platform and services overall, but also our customers'
view of us as a company," Gates wrote. "We can and must do better."

Better late than never, Microsoft is discovering that we are living in a
quality economy, where companies are pouring vast amounts of craftsmanship
into their wares. As it seeks to sign up large corporations for its server
software and shake the bad publicity that comes with chronic attacks from
computer viruses, Microsoft has concluded that its decades-long practice of
putting out "good enough" software is no longer good enough.

This fall, Microsoft is rolling out Exchange Server 2003 and Office 2003.
The new and supposedly improved versions of two of its biggest workhorse
products will be an acid test of whether Gates can make good on his vow to
develop dependable code -- and of whether his quest for quality is for
real. "The joke inside Microsoft has been that quality is job 1.1 -- the
real bugs don't get fixed until after the release, when the service patches
come out," says Greg DeMichillie, a Directions on Microsoft analyst who
spent nine years at the software maker. "It's not yet clear that quality
really is a top priority across all of Microsoft's lines of business."

The results so far are indeed mixed. This past January, Microsoft suffered
a major setback in its quality effort when Slammer, another pernicious
computer worm, attacked a vulnerability in Microsoft's SQL server database
software and spread through network connections, crashing more than 100,000
databases around the world and shutting down at least 15,000 ATMs in the
United States. And in mid-August, thousands of computers were infected by
the Blaster worm, which exploited a hole in Windows operating systems. To
skeptics, Slammer and Blaster prove that Gates's talk of improving the
security and stability of Microsoft's products amounts to little more than
a sales pitch.

Not long ago, I traveled to Redmond to see for myself. I met with CIO Rick
Devenuti and key leaders on the Exchange and Office teams to get a closeup
view of how they're grappling with Gates's new mandate. I found that
Microsoft's quality effort is for real, and that it rests on four simple
propositions. But be forewarned: This initiative is still very much in
development. Call it quality, first release. Patches will arrive over the
next decade.

The Process Is in the Plan For most of its history, Microsoft wanted to be
the anti-IBM -- a nimble, creative place where programmers could have a
neat idea one day and get it into the product the next, without cumbersome
approvals and documentation. But there was a problem with that loose-limbed
approach. Says DeMichillie: "As its products grew more complicated,
Microsoft discovered that even though IBM had its failings, its processes
were there for a reason -- to ensure predictability, stability, and quality."

Microsoft's development teams are now adopting some of the IBM-like
processes that they once abhorred. Case in point: the product blueprint --
detailed design documents that programmers now work from. The blueprint for
Office 2003 consists of roughly 4,000 product specs, each of which amounts
to a 30-to-50-page document that describes in rigorous detail a spec's
features, how it works, and how it must be tested. In theory at least, such
rock-solid product design results in cleaner code and fewer bugs. "The
blueprint keeps us from getting random," says Eric Levine, group program
manager and an 11-year veteran of the Office team. "We create off of it, we
code off of it, we cool down, and we test. That's the process."

But it's still unclear whether developers rigorously stick to the plan. On
the one hand, in the latter stages of Exchange's development, programmers
must file a change request when they want to rejigger the code, and
managers meet daily in a war room to make their calls on the requests --
just as at IBM. A deep database underlies Exchange's staggering 6 million
lines of code, and records every single alteration -- just as at IBM. But
programmers are still encouraged to get creative and take risks. "We give a
lot of freedom to our developers for a reason -- they're smart people,"
says Betsy Speare, the release manager for Exchange. "We're willing to make
a mistake and catch it, so long as we don't miss an opportunity."

In other words, Microsoft is trying to be both rigorous and nimble. "They
want to be able to react quickly to competitors and market changes," says
DeMichillie. "So they're trying to have it both ways. Can they succeed? No
one has before."

Make Your Company Your Customer Given his combination of business smarts
and deep technical knowledge, Bill Gates is uniquely positioned to run a
hard-hitting product critique. And he does so in something called a Bill
Review -- a two-to-three-hour meeting with the leaders of a product team in
which he weighs in on the design, features, and strategy for an upcoming
release. Still, the agenda for a Bill Review is at least partially driven
by what the team expects will catch his interest, such as the overall
product architecture. The blocking and tackling of a launch -- testing and
tracking bugs -- have never been high on Gates's radar, and a Bill Review
can miss potential traps, such as the flaws that were ultimately exploited
by Code Red. But his review of Exchange 2003 was different; Gates hammered
away at the need to ensure quality and security. As Speare walked out of
the meeting, she recalled that Gates's challenge to the team seemed to hang
in the air: "Make it work, and
make it great."

It's doubtful that anyone takes that mission more seriously than Devenuti,
a quietly intense Microsoft lifer. As the boss of Microsoft's own worldwide
IT operations, Devenuti must deal with some of the most demanding,
technoliterate people on the planet. IT isn't Devenuti's only worry. He
also heads up the 3,000-person operations and technology group, which is
the first to set up, launch, and weigh in on a new product before it's
shipped to the outside world. All of which makes Devenuti Microsoft's first
adopter -- and at times, the most unpopular man in Redmond.

Here's why: When Exchange 2003, code-named Titanium and still in beta,
rolled out of the development and testing lab in September 2002, it rolled
right into Devenuti's shop. His team put the not-ready-for-prime-time
software up on a group of servers that were isolated, or "forested," from
the rest of Microsoft's global network. The wobbly code crashed servers,
temporarily shutting off many users' email. At Microsoft, as elsewhere,
tempers go up when servers go down.

"People start screaming, 'What the hell's wrong with you guys?' " says
Devenuti.

When new code runs amok, a beta version is pushed back to the lab, where
its bugs are zapped, and it's then redeployed to an ever widening number of
servers.

The process is called "dog fooding" -- as in eating your own dog food.
Before Exchange is shipped to more than 100 million users, Microsoft will
use it on itself. Says Devenuti: "Microsoft's employees are Microsoft's
first and best customers."

Other high-tech companies dog-food their wares, but it's un-likely that any
company does it on Microsoft's scale. "Right now, I've got 50,000 people
using the next version of Office, which lets us say to customers that we've
deployed it and we've lived on it," Devenuti says. "Dog-fooding gives the
product a level of proof and validity that testing alone could never attain."

Dog-fooding isn't a perfect solution, however. After all, the dog food is
consumed in a homogenous environment at Microsoft. Each piece of software
is up-to-date and compatible with every other piece of software, rarely the
case in the real world. And when Exchange 2003 crashes one of Microsoft's
mail servers, Devenuti can summon an Exchange developer to debug it. You can't.

"It's not yet clear that quality really is a top priority across all of
Microsoft's lines of business."

Triage Your Failures Do we ship, or do we slip? It's a question that preys
on every product manager as the coding approaches its final months of
development and testing and the do-or-die deadline looms large. What causes
a product release to slip? In a word, bugs. It's simple probability: Bugs
will inevitably infest a product like Exchange, with its 6 million lines of
code. During the final stretch, when Exchange 2003 was being readied for
shipment to manufacturers and was supposedly at its most stable, testers
were fixing roughly 500 bugs a month. DeMichillie reports that during a
product's three-year development cycle, "tens of thousands" of bugs are
typically discovered in testing and deployment.

Not all bugs are created equal. When a bug is discovered, it is dropped
into a database, called Product Studio (previously known by the more
colorful moniker "RAID") that enables program managers to track it and
ensure that it's routed back to the appropriate developer. Once it's
entered into the database, a bug is sorted and evaluated. Bugs that have a
high likelihood of crashing the product are given the highest priority rating.

The goal is to zap every single "priority one" bug and kill off as many
low-priority bugs as possible. But the bug count will never hit zero. Not
every bug that's found can be fixed because there's not enough time. "As
the ship date looms, you've got to stop making changes and let the product
settle down," says DeMichillie. "The pressure builds to postpone bugs. You
start to tell yourself, 'This bug's not that bad. We'll get it the next time.'"

The Data Does the Talking On June 30, the 500 members of the Exchange
development team threw themselves a party in the parking lot behind
Building 34 on the Redmond campus. Exchange 2003 was out of the building
and on its way to the manufacturers, the culmination of a three-year
effort. Gates dropped by and thanked people for doing a great job; the
pervasive feeling was one of exhilaration, not exhaustion. "We can say with
tremendous confidence that this is the highest-quality release of Exchange
to date," says Missy Stern, a product manager for the Exchange marketing
team. "The reliability of this product is unparalleled. We're feeling
extremely good about it."

Stern is staking her claim on the data. Microsoft had over 50 partner
companies run Exchange 2003, deploying the software on 170,000 workstations
over five-week periods. Within Microsoft, Exchange nailed the "three nines"
before it was shipped: 99.95% of uptime for six weeks, running on 100% of
Microsoft's mail servers. The company ran weekly surveys of its dog
fooders, the first time it had polled users during development. The
operations and technology group continued testing right up until the code
was released to manufacturing. As far as Microsoft is concerned, Exchange
2003 has passed its quality test.

Or has it? Just 16 days after the release party, Microsoft issued a patch
for a critical flaw that was discovered in Exchange 2003. Windows XP, 2000,
and NT 4 were also affected, which led analysts to conclude that the
squirrelly code went undetected for years, despite the most intensive round
of testing in Microsoft's history. Then came mid-August's Blaster, another
computer pox that attacked a different vulnerability in Windows and
proceeded to slow worldwide Internet traffic. When security sleuths dug
into the code, they discovered a taunt: "billy gates why do you make this
possible? Stop making money and fix your software!"

Taken together, the flaws show that "Microsoft's marketing claims might be
setting up false expectations in people," says DeMichillie. "There's no
silver bullet for eliminating every mistake, and Microsoft's developers
would be the first to admit this."

Still, there are signs that Microsoft's drive for quality, even at the
expense of deadlines, is in earnest. In late April, the company said it
would delay the launch of Office 2003, originally set for this past summer.
It now comes out October 21. The reason: The folks in Redmond had taken an
extra three months to finish testing the software with users.

Bill Breen ([log in to unmask] ) is a Fast Company senior writer.


VICUG-L is the Visually Impaired Computer User Group List.
To join or leave the list, send a message to
[log in to unmask]  In the body of the message, simply type
"subscribe vicug-l" or "unsubscribe vicug-l" without the quotations.
 VICUG-L is archived on the World Wide Web at
http://maelstrom.stjohns.edu/archives/vicug-l.html