The desktop file system is no longer suited to storing our digital work and lives, and it’s building toward catastrophe. Desktop file systems evolved in a world of individual non-networked computers between which documents were transferred on physical media. The modern mobile and multi-device world is ending that paradigm, and “sync” services aren’t helping. We need something new.
Even if all of a user’s devices shared a single “synced” file system, problems occur when devices are online at different times, separated by long distances, or when files are edited simultaneously on different devices. In a global, collaborative, multi-device world, the file system falls apart completely. Then there are other factors: The “Internet of things,” or more specifically, the increase in the number of networked sensors and services that inhabit our lives, exacerbates the problem by generating tremendous amounts of data, which in raw form is often incomprehensible to most humans. Files and folders made sense when most data was created by humans for humans as documents. That is not the world we live in today.
The digital world is now made of big platforms trying to solve different parts of this problem. Google, Facebook, Apple, Twitter, and Dropbox, among others, have all attacked different parts of this challenge and encouraged many developers to do the same. We need a new paradigm for data storage that fits our new lifestyle and embraces real-time collaboration across many apps, users, and devices: a stream of events, not static documents.
At their best, platforms can empower their users and independent developers. They allow application developers to focus on their strengths: designing applications, rather than building and maintaining disparate back-end systems which can be expensive and difficult to scale. Small application teams, especially the current breed of developer/designer mobile shops, often lack the expertise, experience, resources, or desire to build and operate infrastructure which can quickly become difficult to scale when an application “goes viral.” The costs of running servers and storage month after month often outpace the revenue of fledgling apps. Platforms offer these teams a way out: Build your app on our API and we will take care of the rest. A captive audience of users with complete social graphs and a partner who assumes all responsibility for ops is a difficult offer to resist for many developers.
Unfortunately this arrangement is doomed to failure without outside intervention. In the race to monetize (especially for venture-backed and recently public companies) independent developers can quickly become one another’s competition. The balance of power in this arrangement incentivizes platform companies to act poorly. If an app is particularly successful or becomes a core part of the user experience on the service, the platform company has the choice to build or buy. There is nothing wrong with mergers and acquisitions, but when the platform company controls the API, the balance of power is always in their favor. In the case of Instagram, this worked out splendidly for the developers. Instagram became a major attraction for Twitter users, so the creators were able to leverage that position to produce a $1B buyout from Facebook, which essentially bought a large part of Twitter’s audience. On the other hand, when Snapchat attracted a large audience of Facebook users, Facebook created Poke, a near feature clone, which it could then promote officially. Companies that depend on platforms for new users can find themselves in a dangerous situation if their products become too popular.
Users also engage in a complex value trade with platforms. Users’ data is often inextricable or inseparable from the platform where it was created or stored because of restrictive APIs and terms of service. Most users split their time between multiple platforms (Facebook, Dropbox, Google) requiring them to maintain separate social graphs and duplicate data between services, both of which need to be manually synced regularly. Most applications work with only one platform and rarely communicate with other applications, which is often a limit of platform APIs themselves. The lack of user control in this space led to the rise of automated scripting applications and platforms in their own right like Yahoo Pipes, IFTTT, and Zapier that help users move, sync, and transform data between apps and platforms. Unfortunately, even these bolted-on solutions require platform support, and those which give users too much power or portability have had their access credentials terminated.
Ultimately all platform companies fall prey to the same fallacy, that they can build the API to rule them all: a single platform for all the world’s users, applications, and data. This goal and their position of immense power incentivizes many platform companies to become bad actors in their ecosystems, hurting developers and users alike. Platforms act as intermediaries between users and developers, depriving both of a direct relationship with each other and an open market. This consolidation and potential for abuse of power is unsustainable for the ecosystem as a whole. Systems of significant value to society always becomes commodified eventually. Initial development of technology is often centralized, closed, and proprietary, because this allows for more experimentation and faster iteration–which is beneficial to consumers. Once a product permeates the market, competing alternatives emerge. Eventually these systems need standardization, interoperation, and in some cases, disruptive intervention.
It is unimaginable that the world could have parallel telephony providers, each with proprietary hardware and the inability to call customers of another network, but this is exactly the situation today in the digital world. Each platform has a separate, isolated social graph which it fights to keep cut off from the outside world and competitors. I don’t need separate phones (and accompanying service plans) so I can speak to friends on each of the carriers, why do I need different accounts to communicate with users on different platforms? Today’s platforms are starting to feel like the dialup services of the ’90s. Users are forced to become a citizen of only one platform, cut off from their relationships, data, and applications that are the exclusive citizens of another. Just as telephone systems are interoperable and the World Wide Web won over dialup walled gardens, it’s time for an open solution to data storage, sync, and communication.
All the technological building blocks needed to create a comprehensive solution exist already. Many emerged from the failures and successes of major platform companies.
The basic technical philosophy of platforms is sound: Users have a single service provider and the choice of many applications created by independent developers. Twitter at its best (before recent API changes) exemplified this system. Twitter was a back-end provider with a bare-bones web interface. Users had the choice of hundreds of clients and thousands of other applications, each providing a different experience. That ecosystem made Twitter a more powerful tool and expanded its utility to new groups of users. Tweetie, iOS developer Loren Brichter’s Twitter client, was a completely different application than TweetDeck and served a different audience. If Twitter had one official iOS client from the outset (like it now does) those who would have used other specialized applications would be poorly served. The mobile ecosystem gave birth to a new class of small designer-developer teams who excel at creating beautiful user interfaces. Those teams should be free to explore, experiment, and push technology forward. Divorcing service providers from apps gives them this freedom, but doesn’t provide protection.
Applications need to be portable between service providers and accounts on different service providers need to be interoperable. It should not matter what provider you use–you should be able to communicate with anyone on any provider, using any app, and be able to take your apps with you when and if you change providers. This model protected and ensured the success of another critical technology for the past several decades: email. The experience for a Gmail user who send a message to another Gmail user is the same as if she sent a message to a Yahoo Mail user. Similarly, she can use Sparrow, Mail.app, Outlook, Thunderbird, or any other mail application with her Gmail account. Despite three companies (Gmail, Yahoo, and Microsoft) dominating email for the last decade, this balance of power remains to the immense benefit of users, developers, and service providers. The reason for this balance is clear: SMTP and IMAP.
SMTP and IMAP are both protocols–a standard set of formats and rules for exchanging messages between two computer systems. SMTP specifies how two service providers communicate and IMAP specifies how mail applications communicate with service providers. The web, Internet, and telephone systems are also built on open protocols which have powered a generation of innovation and prosperity for their respective industries. Protocols aren’t easy to build well. Distributed systems are complex and need to be 100% reliable. Both SMTP and HTTP had strong competitors in their early days. The challenge in designing protocols stems from their longevity. Protocols stay in use forever. Protocol designers need to anticipate the needs of potential users half a century in advance. While the software implementations of protocols evolve to become more efficient, updates to the protocols themselves come very slowly after the protocols have been widely adopted. A good protocol needs to be lightweight, extensible, and robust. Most protocols themselves are built on top of other, earlier protocols making the reliability of every component extremely important.
For the past year and a half, I have worked full time with a team on an open-source protocol that I think can fill this need. It’s called Tent. It describes real-time evented data store with two APIs: server-server and application to server. Tent is built on top of battle-tested HTTP, TLS, and JSON and can be used to model nearly every use case from decentralized microblogging to photosharing to real-time collaborative document editing to cloud-backed file sync. With Tent, users choose what information they share and with whom. (Docs and open-source integrations are available for reference here.)
A protocol-driven, post-file-system world would look very different to users and developers from the platforms of today, although many of the companies would remain the same. For users the greatest change overwhelmingly would be simplicity. Instead of dozens of accounts and login credentials, each user would have one provider (although they would be free to have more to maintain multiple identities, like having one home and one work email address). They could choose their service provider based on their personal preferences. Just like many users are comfortable using Gmail for free knowing the content of their messages are being searched and mined for ads, many users may choose a free, ad-supported service provider. Privacy enthusiasts might choose a paid service provider, or even choose to host their own server (just as many people self-host WordPress or email today). Businesses could also self-host or choose a highly secure business-focused SaaS.
In this world, every service provider would support every application by default, since they both implement the same protocol–just as most email apps can be used with most providers or any web browser can load any website. All of a user’s data would be stored in one place, accessible to all their applications. Just like desktop applications have access to a user’s entire file system, now web and mobile applications could access and modify any data to which the user gave them permission. The protocol would need to store information generically with some analog to file types. Not every application needs or would want to open every file, but common types would emerge for common needs (photos, documents, and messages, for example).
The choice of applications and service providers would return control to the user–where it belongs. The problems we worry about with web, mobile, and platform-apps today didn’t exist back when users were in control: privacy; security; data ownership; apps being stabbed in the back by their platforms. Users would not be faced with a choice between restrictive terms of service and losing access to their online relationships and the apps they love. If a service provider changed their terms or was shutting down, a user could move their content, relationship, and apps to another provider, like plugging their phone or television into a different service provider. Most importantly, service providers would be heavily commodified, eliminating the power imbalance between users and platforms today.
Developers benefit tremendously as well. As with platforms, developers would not need to maintain servers for many types of applications. The user’s server would act as the primary data store for all their content; it would send and receive posts to and from other users’ servers. For a broad class of applications, this means developers don’t need to worry about privacy or many security concerns. In the case of native or stateless web apps, developers would ship applications that communicate exclusively with users’ servers, which never “phone home” to a developer-controlled server.
This means no more leaked password databases or compromised user data. It also frees developers from the challenges of scaling a social service. Handling increased storage and communication between users would be a challenge left to Internet service providers, not individual dev teams. Similarly, users would decide which permissions to assign to which applications, so in many cases no sensitive data will ever be available to the application. Just like the web, email, and telephones, a protocol for the post-file system lets everyone focus on their strengths. Designers and app developers can build the best UIs, service providers can maintain reliability and security, and users can create and consume content. Let’s hope that’s the future.
[Explosion: Vasily Smirnov via Shutterstock]