Last week Netflix released Ice, a publicly available tool that offers a granular look at Amazon Web Services usage and the associated costs in more detail than Amazon itself provides. Netflix, which accounts for a third of all North American Internet traffic on any given night, relies on the Amazon’s cloud platform because it’s the only company with the capacity to deliver all of that data to customers.
But when AWS runs into problems, like when its Elastic Load Balancer that routes network traffic went down, causing an embarrassing Christmas Eve outage of Netflix’s streaming service, there is no recourse. As a result, Netflix is forced to create its own tools for dealing with problems it encounters in AWS.
Without a legitimate competitor, they don’t have much choice. That’s the message that emerged from the remarks of two Netflix IT gurus who spoke at the Structure 2013 cloud computing conference in San Francisco. “We have a great, great relationship with AWS,” Netflix’s chief cloud architect Adrian Cockcroft insisted, before describing a working relationship that sounded… well, “great” was not the first word that leapt to mind. “They won’t do anything really substantial just for us,” he said on stage.
Cockcroft told FastCompany that Netflix was still one of the top 10 biggest customers of AWS, but that the company’s default mentality was open source. “We’re building an ecosystem with Netflix OSS, and it lets us validate whether an idea is good in public, but we’re also using it as a hiring tool. Really good engineers like @adrianfcole joined us because of open source. Netflix OSS is also great for when you’re hiring, because it means that people already know your tools and architecture.”
He did believe that Netflix’s pestering had eventually led to AWS adopting features like IPV6 support. “We’ll ask them for something, and they’ll think about it a bit, go out and talk to other people and say ‘Maybe it is a good idea.’ Eventually, after much cajoling and twisting of arms, they come out with something.”
Some in the cloud computing world embrace OpenStack, an open source software project that aims to rival AWS. As one of the biggest users of the cloud, Netflix has drawn ire for not throwing more support behind that. In 2011, Cockcroft declared that OpenStack was not ready for prime time, saying that AWS versus OpenStack reminded him of Apple iOS versus Google’s fragmented Android platform. “It’s far harder for developers to build Android apps that work on all devices, and then they usually make much less money from them,” Cockcroft wrote on his blog. “Apple and its ecosystem is dominant, growing fast, and extremely profitable.” Cockcroft longed for the day when cloud service is just a commodity that can be taken for granted, and you don’t need to pay attention to providing it. But that day hasn’t come yet.
That still seems to be Netflix’s take. “We’re far from being in a commoditized cloud market,” said Netflix’s director of cloud solutions Ariel Tseitlin. “It really isn’t a utility like we feel someday it is going to become.”
“Amazon still has a long way to go, but there’s night and day between its feature breadth and feature set and everyone else’s,” said Tseitlin. There’s also the issue of sunk cost, the time and energy they’ve already invested in their relationship with Amazon. “We had to pay a pretty large pioneer tax,” said Tseitlin, and they’re wary of blazing more trails if they don’t have to. Despite the “pains and complications” that Netflix sometimes encounters working with AWS, Tseitlin says that, “we don’t have a particularly strong desire to pay that pioneer tax again with another cloud provider.”
The desire to route around Amazon is part of the impetus behind the Netflix Open Source Software project, which is where Netflix releases its internal tools to the public. And they’ve announced a cloud prize offering 10 $10,000 rewards to people outside the company who improve upon Netflix’s tools, similar to the prize offered for the best improvements to Netflix’s recommendations engine.
“If you look at how much infrastructure we built, the huge amount of extra glue and services and tooling we’ve invested in, that gives you an indication of what could be offered in the future,” said Tseitlin. “We want to stimulate that ecosystem by open-sourcing our work as part of our Netflix OSS platform.”
Tseitlin adds that Netflix’s infrastructure costs currently represent a “tiny little sliver” compared to what the company spends on marketing and securing content. Cockcroft adds that the tools released through Netflix OSS are things that can be done by a few people in a few months. Netflix doesn’t want to put huge teams on creating ambitious tools any more than they’d want to shell out for their own data centers.
“We don’t want to do the undifferentiated heavy lifting of owning large buildings, owning all that air conditioning,” Cockcroft said. “If I had a hundred million dollars, I wouldn’t want to spend it on a data center; I’d want to spend it on another House of Cards or on launching in a new country.”
[Image: Flickr user David Smith]