“Most people think of us as a
financial institution, but the network is the brand,” says Rick Knight, Visa’s head of global
systems operations and engineering. “If it goes down, lives are on the line.”
in a briefing room, its walls opaque like any other’s. But with the push of a button, they become
transparent glass, revealing what’s beyond–a NASA-like command center with a 40-by-14-foot wall of
screens, including Visa’s network overlaid on a world map. The network’s vital signs are constantly
tracked, showing, at the moment, 8,000 transaction messages a second.
This is Visa’s OCE, or
Operations Center East, the biggest, newest, and most advanced of its U.S. data centers. It is a
data-security heaven–and Visa’s acknowledgment that hackers are increasingly savvy, that data is
an ever-desirable black-market commodity, and that the best way to keep Visa (and its 150 million
daily transactions) safe is to ensconce its network inside a heavily fortified castle that
instantly responds to threats.
The OCE’s 130 workers have two jobs: Keep hackers out and
keep the network up, no matter what. That’s why rule No. 1 for visitors is: Never reveal its
location. “On the eastern seaboard” is as specific as Visa will allow.
Somewhere On The Eastern Seaboard
Hydraulic bollards lurk beneath the road outside the OCE, which can
rise fast enough to stop an intruding car going 50 miles per hour. If the car exceeds that, it
won’t be able to make a vicious hairpin turn built into the road and will then careen into a
drainage pond, a modern-day moat.
Invited guests who pass the gauntlet have their photo and
index fingerprint encoded on a badge. Entering the data center means first passing a “mantrap”
portal. With the doors locked on either
side, you put your badge on a reader that compares it with the real you for a few seconds. Next,
the badge on another reader and then put your finger on a fingerprint detector.
The portal leads you into the network-operations center, where workers in business casual
monitor the wall of screens, plus four monitors at their desks. Three Visa security gurus sit in a
room behind the main center. One has Sun Tzu’s The Art of War, which might as well be
required reading here. They’re monitoring networks across Visa’s operations, looking for malware,
for odd behavior. Knight says about 60 incidents a day warrant attention.
might turn out to be a few employees in a Visa office watching a YouTube video at the same time, or
it could be somebody trying to penetrate a network. Less likely is someone trying to hack Visa’s
core transaction network–which is private, free from Internet hazards like the denial-of-service
attacks that groups like Anonymous love. When hackers took down Visa’s corporate website in 2010,
it had zero effect on the core network.
Backups For Backups
Hackers are one thing, but Knight also loses sleep over network capacity. Visa
and IBM routinely run tests at an off-site lab to simulate maximum network capacity–currently just
over 24,000 transaction messages per second, about double what Visa expects for this year’s
Christmas season. At some point over that 24,000-message limit, “the network doesn’t stop
processing one message. It stops processing all of them,” Knight says.
This last happened in 2005, when Visa had a system failure while updating some software and was
down for the longest eight minutes of Knight’s life.
That event is one reason why the OCE was built to meet the Uptime Institute’s definition of a
“Tier 4” center, which means every major system–mainframes, air conditioners, batteries–has a
backup. It’s Visa’s first such center and is strong enough to withstand
California-style earthquakes and Midwestern-size super-tornadoes. For example: If power goes out,
banks of batteries kick in, followed by diesel generators that also have backups. If a water main
bursts, there’s a 1.5-million-gallon water tank (if that runs dry, Visa dug three wells around the
The only thing that might take down the OCE, it seems, is us. Transaction growth has averaged 7%
to 17% annually. Visa expects the OCE to handle growth into the 2020s, but at some point, of
course, something will give.
Inside “The Pod”
A second mantrap portal leads into the hub of the data center. Its main
corridor runs nearly three football fields, linking seven 20,000-square-foot rooms called pods. Two
pods house Visa’s core network, one has its corporate networks, and one handles development work. A
fifth pod was built out this year to handle Visa acquisitions like Fundamo, a South African maker
of mobile payment software. Two pods await future expansion.
We enter Pod 4, part of the network center. This is it–the heart, the brain. It’s loud inside
from the hum of spinning hard drives and the whir of fans inside rows of top-of-the-line IBM
mainframes, EMC storage arrays, and Cisco switches. They’re connected by miles of cabling–the
center has enough to run along I-95 from Maine to Key West, Florida.
Though this room is key to a network that makes modern life possible, it seems designed to
separate us fallible humans from the network. People largely stay out. Even the air conditioners
sit just outside the pod, so repair people are kept away from the computers.
Scrawled on a wallboard inside the OCE is the motto 7x24xforever. Even in the event of the
Apocalypse, “we could run for at least a week,” Knight says. Though he acknowledges that after the
Apocalypse, credit-card usage might drop.
A version of this article appears in the November 2011 issue of Fast