When you walk around the offices of Twitter’s engineering department, located on the sixth floor of the company’s downtown San Francisco headquarters, you will see signs counting down the days until the World Cup. The World Cup is the world’s biggest sporting event, and because it consists of fewer matches than the Olympics, it generates more Internet traffic, in shorter bursts. More than 3.2 billion people watched at least a minute of the World Cup live in 2010. For Twitter, Facebook, ESPN, YouTube, and a host of regional social media sites from Brazil to Russia, the World Cup means engineers frantically working overtime to prevent outages and site overloads.
Twitter learned this the hard way in 2010. Facing an unprecedented surge of user traffic, in amounts of more than 150,000 tweets per hour, Twitter crashed repeatedly during the 2010 World Cup. Users constantly encountered the “Fail Whale” (to be fair, Twitter has crashed a lot less so far in 2014) and Twitter’s engineers copped to “periodic high rates of errors.” In a now-deleted post on Twitter’s engineering blog, the popular social networking service said the huge traffic influx from the 2010 World Cup created unspecified network issues.
When World Cup matches take place, entire countries take to the Internet. They tweet, update their Facebook statuses, and watch instant replays at a massive clip. This has meant all sorts of new partnerships designed to capture fans and their wallets. CNN and Facebook are teaming up to use Facebook’s analytics teams to provide real-time information on global sentiment about upcoming matches. ESPN is getting ready for an unprecedented logistics and engineering challenge by livestreaming all World Cup matches to mobile and desktop. Brazil’s mobile carriers and Internet service providers are working frantically to prevent outages. In short: If you work at a digital service that people use during the World Cup, the last few months have meant lots of overtime.
But back at Twitter’s headquarters, the company’s goal is to prevent another replay of the 2010 World Cup tech troubles. They’re a publicly traded corporation now, and one that wants to take on Facebook and even SMS text messages as the way the world communicates. Although Twitter would only speak with Fast Company in general terms about how they keep their network resilient for big events like the Olympics or presidential elections, it’s clear they take the 2014 World Cup very seriously.
Raffi Krikorian is one of Twitter’s main engineers who keeps the service’s backend working. Krikorian, a vice president of platform engineering, helms a team (which encompasses approximately one third of Twitter’s software engineers) responsible for preventing outages and making sure the service is available.
“I’ve been here just shy of five years, and I still have PTSD from the last World Cup at Twitter,” Krikorian told me. “When you come to my floor at Twitter headquarters, we have signs all over the floor with a countdown to the World Cup. Reliability is at the top of our minds, and reliability first is the mantra. Somewhere in the world, there is a sporting event, an election, or an earthquake.”
But it also poses very specific engineering challenges for Twitter. Krikorian’s team plans for the World Cup using worst-case scenarios of extremely high site traffic. One hypothetical he brought up on the phone was a Brazil-Japan match; Twitter’s market penetration in Japan is massive and a Japanese television show holds the record for inspiring the most tweets-per-second.
According to Krikorian, Twitter’s engineering problems are incredibly specific and difficult because they occur in real time. While a service such as Facebook or Reddit can get away with having messages posted 30 seconds or even a few minutes late, Twitter has built their service model (and business model) around real-time communication. This means setting up a system of “shock absorbers”--redundant servers designed only for service when the network is extremely busy--around the world that kick in during extremely high traffic periods, and lots of planning at Twitter headquarters for different scenarios. Twitter performs tests to figure out which servers best serve specific users, and how to scale out their global server infrastructure to serve new markets.
The company faces engineering challenges on two fronts that make planning for the World Cup even harder: An ongoing transition from Twitter being a text-only service to embracing photos, video, and other embedded media, and strong growth in regions like India and Brazil that don’t necessarily have a strong network infrastructure. That led to tech-y changes like Twitter embracing the SPDY transfer protocol, which helps international users get information quicker.
Beyond bracing for an onslaught of traffic, Twitter is also embracing the World Cup and its swarm of users during crucial points in games as a marketing and growth opportunity. To this end, it has implemented new UI/UX touches and updates designed to attract and retain new soccer/football-loving users. For example, Twitter's sign-up process has been retooled for the World Cup, primarily with fans in non-U.S. markets in mind--the sign-up process now includes the option to select your favorite team and choose national flags and patriotic header photos to represent your team on your profile. Twitter rolled out new notification features in the weeks leading up to the games. New, customizable Cup- and game-specific timelines are also being rolled out and individual teams are getting their own Twitter web profiles which will include player and scorecard information. During the last World Cup, Twitter unleashed "hashflags"--hashtags which include a corresponding country flag icon--and is making the feature available during the 2014 games, as well.
There's no doubt that the 2014 World Cup presents Twitter and advertisers with a big opportunity to attract eyeballs. Twitter has been building up to this moment for years. "Our true scaling story started at the last World Cup," Krikorian says. "Everything we have done over the past four years has been done to enhance our users' trust in Twitter and our reliability." With Twitter expecting massive amounts of traffic in just a few days, the microblogging company's preparations will be put to the test by both marketers and users.
Correction: An earlier version of this article misidentified Raffi Krikorian's title. He is vice president of platform engineering.
[Image: Flickr user E.N.K]