How Visa Protects Your Data

A rare trip inside the network's top-secret security center. Location: We can't say.

The command room inside
Visa's Operations Center East, where your last credit-card purchase was scrutinized. | Photo by
Melissa Golden
The command room inside Visa's Operations Center East, where your last credit-card purchase was scrutinized. | Photo by Melissa Golden

"Most people think of us as a financial institution, but the network is the brand," says Rick Knight, Visa's head of global systems operations and engineering. "If it goes down, lives are on the line."

He's talking in a briefing room, its walls opaque like any other's. But with the push of a button, they become transparent glass, revealing what's beyond—a NASA-like command center with a 40-by-14-foot wall of screens, including Visa's network overlaid on a world map. The network's vital signs are constantly tracked, showing, at the moment, 8,000 transaction messages a second.

This is Visa's OCE, or Operations Center East, the biggest, newest, and most advanced of its U.S. data centers. It is a data-security heaven—and Visa's acknowledgment that hackers are increasingly savvy, that data is an ever-desirable black-market commodity, and that the best way to keep Visa (and its 150 million daily transactions) safe is to ensconce its network inside a heavily fortified castle that instantly responds to threats.

The OCE's 130 workers have two jobs: Keep hackers out and keep the network up, no matter what. That's why rule No. 1 for visitors is: Never reveal its location. "On the eastern seaboard" is as specific as Visa will allow.

Somewhere On The Eastern Seaboard

Hydraulic bollards lurk beneath the road outside the OCE, which can rise fast enough to stop an intruding car going 50 miles per hour. If the car exceeds that, it won't be able to make a vicious hairpin turn built into the road and will then careen into a drainage pond, a modern-day moat.

Invited guests who pass the gauntlet have their photo and right index fingerprint encoded on a badge. Entering the data center means first passing a "mantrap" portal. With the doors locked on either side, you put your badge on a reader that compares it with the real you for a few seconds. Next, you put the badge on another reader and then put your finger on a fingerprint detector.

The portal leads you into the network-operations center, where workers in business casual monitor the wall of screens, plus four monitors at their desks. Three Visa security gurus sit in a room behind the main center. One has Sun Tzu's The Art of War, which might as well be required reading here. They're monitoring networks across Visa's operations, looking for malware, for odd behavior. Knight says about 60 incidents a day warrant attention.

Those incidents might turn out to be a few employees in a Visa office watching a YouTube video at the same time, or it could be somebody trying to penetrate a network. Less likely is someone trying to hack Visa's core transaction network—which is private, free from Internet hazards like the denial-of-service attacks that groups like Anonymous love. When hackers took down Visa's corporate website in 2010, it had zero effect on the core network.

Backups For Backups

Hackers are one thing, but Knight also loses sleep over network capacity. Visa and IBM routinely run tests at an off-site lab to simulate maximum network capacity—currently just over 24,000 transaction messages per second, about double what Visa expects for this year's Christmas season. At some point over that 24,000-message limit, "the network doesn't stop processing one message. It stops processing all of them," Knight says.

This last happened in 2005, when Visa had a system failure while updating some software and was down for the longest eight minutes of Knight's life.

That event is one reason why the OCE was built to meet the Uptime Institute's definition of a "Tier 4" center, which means every major system—mainframes, air conditioners, batteries—has a backup. It's Visa's first such center and is strong enough to withstand California-style earthquakes and Midwestern-size super-tornadoes. For example: If power goes out, banks of batteries kick in, followed by diesel generators that also have backups. If a water main bursts, there's a 1.5-million-gallon water tank (if that runs dry, Visa dug three wells around the facility).

The only thing that might take down the OCE, it seems, is us. Transaction growth has averaged 7% to 17% annually. Visa expects the OCE to handle growth into the 2020s, but at some point, of course, something will give.

Inside "The Pod"

A second mantrap portal leads into the hub of the data center. Its main corridor runs nearly three football fields, linking seven 20,000-square-foot rooms called pods. Two pods house Visa's core network, one has its corporate networks, and one handles development work. A fifth pod was built out this year to handle Visa acquisitions like Fundamo, a South African maker of mobile payment software. Two pods await future expansion.

We enter Pod 4, part of the network center. This is it—the heart, the brain. It's loud inside from the hum of spinning hard drives and the whir of fans inside rows of top-of-the-line IBM mainframes, EMC storage arrays, and Cisco switches. They're connected by miles of cabling—the center has enough to run along I-95 from Maine to Key West, Florida.

Though this room is key to a network that makes modern life possible, it seems designed to separate us fallible humans from the network. People largely stay out. Even the air conditioners sit just outside the pod, so repair people are kept away from the computers.

Scrawled on a wallboard inside the OCE is the motto 7x24xforever. Even in the event of the Apocalypse, "we could run for at least a week," Knight says. Though he acknowledges that after the Apocalypse, credit-card usage might drop.

A version of this article appears in the November 2011 issue of Fast Company.

Add New Comment

7 Comments

  • joseph

    the best place to hide something is in plain sight but don't try to pull my leg and say its top secret when you build some thing in the middle of VA. its not like you built it in area 51. Russell Branch Pkwy, Ashburn, VA is not a top secert location.

  • Sean

    It doesn't take much work to find it.

    44901 Russell Branch Parkway
    Ashburn, VA

    Google Maps: http://g.co/maps/vxv9u

    You can see the "hairpin turn" on the aerial view at the west side of the facility.  It has literally the exact same area to turn as the highway offramp near my apartment, which I've taken at speeds well in excess of 50 MPH numerous times in a car that's nothing special.  If you for some reason can't find a vehicle that handles equal to or better than a decade old 3 series with good tires, there also seems to be nothing stopping anyone from just driving over the grass.

    The "hidden" location, bollards, and such are simply security theater meant to impress those who don't think about it.

    That's not to say that the parts that do actually matter, the redundant everything, separated networks, and to some extent the mantraps aren't worthwhile, but most everything that is claimed as being different from other high-end datacenters is fluff.

  • David

    So everybody should keep the location secret, except the county in which they are in?

    "Recently VISA opened its East Coast processing facility campus in Loudoun County."
    -Loudoun County proposed Fiscal Plan for 2011

  • Dapinski

    Based on the information in the article, it took me about 5 minutes to find the facility. Fun article, but it didn't make me feel more secure about Visa's security. If keeping the location secret is important to security, don't discuss it at all. First rule of fight club, is don't talk about fight club.