These Engineers Just Built Their Own “Pied Piper” Compression Algorithm

A team of engineers at Dropbox’s Hack Week re-created the fictional compression schema from HBO’s Silicon Valley.

These Engineers Just Built Their Own “Pied Piper” Compression Algorithm
Silicon Valley Season 2 Finale with Kumail Nanjiani, Zach Woods, T.J. Miller, Thomas Middleditch. [Photo: John P. Johnson, courtesy of HBO]

You would be forgiven for thinking that the elevator pitch for Daniel Reiter Horn’s latest project sounds familiar. That’s because there’s a good chance you saw it on HBO. In fact, everything about the project, right down to its name, was ripped right from the network’s hit comedy Silicon Valley. And that was precisely the point.


Horn’s “Pied Piper” is exactly what fans of the TV show would expect, given its name: It’s a media file compression schema designed to compress and decompress files without compromising their quality. Over the course of about a week, Horn and his team have managed to achieve a 22% reduction in file size for JPEG images without any notable loss in image quality. Next, they’re moving onto video files. Eat your heart out, Richard Hendricks.

“The goal for this product is to prove that we can achieve lossless compression for JPEG images and H.264 videos,” explains Horn. “If successful, we can use this technology to save larger files in even less space.”

Pied Piper came out of a “Hack Week” held at Dropbox, where Horn works as an infrastructure engineer. Pied Piper is one of dozens of projects being worked on during the weeklong event, where staffers are encouraged to dream up and build products and features outside of the normal scope of the company’s mission. In total, 10 Dropbox employees are working on the Pied Piper algorithm.

Dropbox engineer Daniel Reiter Horn working on their version of Pied Piper.

So what’s the problem being solved here? As Horn points out, there are still some outdated inefficiencies in the way some files are compressed. “For instance, almost all JPEG files today are Huffman coded, yet it is well known that applying an additional arithmetic coder to existing JPEG files brings a further 10% reduction in file size at no cost to the file,” he says. “Our Pied Piper algorithm aims to go even further with a more efficient encoding algorithm that maps perfectly back to existing formats.”

While it’s certainly a fun experiment to undertake during a company’s hack week, Pied Piper could wind being a bit more significant than that: Its ability to compress file sizes could actually have tangible, real-world benefits for Dropbox, whose core business is storing files in the cloud.

“I view it as building a greener Dropbox,” Horn says. “Using fewer hard drives to save space means mining fewer minerals to build those hard drives and burning less power to run their supporting machinery.”


The team has also open-sourced the code on GitHub, allowing others to take advantage of their work.

While Horn and his colleagues may not make it to the stage of TechCrunch Disrupt with this one, they can rest easy knowing that their algorithm can help save precious hard drive space. And who knows–it might be even help make the world a better place.

About the author

John Paul Titlow is a writer at Fast Company focused on music and technology, among other things.