"Most people think of us as a financial institution, but the network is the brand," says Rick Knight, Visa's head of global systems operations and engineering. "If it goes down, lives are on the line."
He's talking in a briefing room, its walls opaque like any other's. But with the push of a button, they become transparent glass, revealing what's beyond--a NASA-like command center with a 40-by-14-foot wall of screens, including Visa's network overlaid on a world map. The network's vital signs are constantly tracked, showing, at the moment, 8,000 transaction messages a second.
This is Visa's OCE, or Operations Center East, the biggest, newest, and most advanced of its U.S. data centers. It is a data-security heaven--and Visa's acknowledgment that hackers are increasingly savvy, that data is an ever-desirable black-market commodity, and that the best way to keep Visa (and its 150 million daily transactions) safe is to ensconce its network inside a heavily fortified castle that instantly responds to threats.
The OCE's 130 workers have two jobs: Keep hackers out and keep the network up, no matter what. That's why rule No. 1 for visitors is: Never reveal its location. "On the eastern seaboard" is as specific as Visa will allow.
Somewhere On The Eastern Seaboard
Hydraulic bollards lurk beneath the road outside the OCE, which can rise fast enough to stop an intruding car going 50 miles per hour. If the car exceeds that, it won't be able to make a vicious hairpin turn built into the road and will then careen into a drainage pond, a modern-day moat.
Invited guests who pass the gauntlet have their photo and right index fingerprint encoded on a badge. Entering the data center means first passing a "mantrap" portal. With the doors locked on either side, you put your badge on a reader that compares it with the real you for a few seconds. Next, you put the badge on another reader and then put your finger on a fingerprint detector.
The portal leads you into the network-operations center, where workers in business casual monitor the wall of screens, plus four monitors at their desks. Three Visa security gurus sit in a room behind the main center. One has Sun Tzu's The Art of War, which might as well be required reading here. They're monitoring networks across Visa's operations, looking for malware, for odd behavior. Knight says about 60 incidents a day warrant attention.
Those incidents might turn out to be a few employees in a Visa office watching a YouTube video at the same time, or it could be somebody trying to penetrate a network. Less likely is someone trying to hack Visa's core transaction network--which is private, free from Internet hazards like the denial-of-service attacks that groups like Anonymous love. When hackers took down Visa's corporate website in 2010, it had zero effect on the core network.
Backups For Backups
Hackers are one thing, but Knight also loses sleep over network capacity. Visa and IBM routinely run tests at an off-site lab to simulate maximum network capacity--currently just over 24,000 transaction messages per second, about double what Visa expects for this year's Christmas season. At some point over that 24,000-message limit, "the network doesn't stop processing one message. It stops processing all of them," Knight says.
This last happened in 2005, when Visa had a system failure while updating some software and was down for the longest eight minutes of Knight's life.
That event is one reason why the OCE was built to meet the Uptime Institute's definition of a "Tier 4" center, which means every major system--mainframes, air conditioners, batteries--has a backup. It's Visa's first such center and is strong enough to withstand California-style earthquakes and Midwestern-size super-tornadoes. For example: If power goes out, banks of batteries kick in, followed by diesel generators that also have backups. If a water main bursts, there's a 1.5-million-gallon water tank (if that runs dry, Visa dug three wells around the facility).
The only thing that might take down the OCE, it seems, is us. Transaction growth has averaged 7% to 17% annually. Visa expects the OCE to handle growth into the 2020s, but at some point, of course, something will give.
Inside "The Pod"
A second mantrap portal leads into the hub of the data center. Its main corridor runs nearly three football fields, linking seven 20,000-square-foot rooms called pods. Two pods house Visa's core network, one has its corporate networks, and one handles development work. A fifth pod was built out this year to handle Visa acquisitions like Fundamo, a South African maker of mobile payment software. Two pods await future expansion.
We enter Pod 4, part of the network center. This is it--the heart, the brain. It's loud inside from the hum of spinning hard drives and the whir of fans inside rows of top-of-the-line IBM mainframes, EMC storage arrays, and Cisco switches. They're connected by miles of cabling--the center has enough to run along I-95 from Maine to Key West, Florida.
Though this room is key to a network that makes modern life possible, it seems designed to separate us fallible humans from the network. People largely stay out. Even the air conditioners sit just outside the pod, so repair people are kept away from the computers.
Scrawled on a wallboard inside the OCE is the motto 7x24xforever. Even in the event of the Apocalypse, "we could run for at least a week," Knight says. Though he acknowledges that after the Apocalypse, credit-card usage might drop.
A version of this article appears in the November 2011 issue of Fast Company.