At this year’s WWDC, Apple delighted some users with a new feature in macOS Mojave. Called “Stacks,” the new functionality automatically sorts your desktop clutter into neat stacks of file types at the blink of an eye. Stacks aren’t really new, though. Like many other aspects of the technology we use today, they evolved from the work of Apple’s research center: the Advanced Technology Group.
The ATG was founded in 1986 by Larry Tesler, a computer scientist who had previously worked at Xerox Palo Alto Research Center–aka PARC, the birthplace of the graphic user interface–before moving to Apple. The group’s mission was to create breakthrough technologies that didn’t need to be products. The theory went that the ATG’s computer scientists, shielded from the company’s day-to-day grind, would have the creative and professional freedom to spark the Next Big Thing in consumer tech.
From 1986 to 1997, isolated from the fray at One Infinite Loop, Apple engineers and scientists crafted breakthrough technologies like HyperCard, QuickTime, QuickTime VR, and Apple Data Detectors. These inventions–even if they don’t exist today in their original form–shaped how computers, smartphones, and even the web itself works today.
The early days of interface design
Though they were introduced onstage at WWDC as “Stacks,” they were once known as “Piles.”
In May 1992, ATG engineers Richard Mander, Gitta Salomon, and Yin Yin Wong presented the concept at the Conference on Human Factors in Computing Systems in a paper titled “A ‘Pile’ Metaphor for Supporting Casual Organization of Information.” It extended the desktop metaphor even further by allowing users to organize their files in stacks of papers, images, or videos, leaving folders for more permanent archival purposes–just like real life. These manual piles were drawn in an isometric perspective and looked slightly disheveled, just like the real thing, and to develop a way look through them, the ATG studied the way people browsed paper files in real office settings. For instance, you could temporarily spread the files out on the desktop by drawing a zig-zag cursor gesture over the pile. Another method, called “edge browsing,” let you hover the cursor up and down the pile’s edge to display a miniature version of each file.
The ATG also came up with a way to create Piles automatically, following user-defined rules, like “pile up all graphic files of more than 640 x 480 pixels.” The operating system would automatically stack up those files in a neat and straight pile, a visual cue that differentiated them from the manual piles. Danish user experience expert Jakob Nielsen–who attended the conference–liked the idea: “Given the way my desk looks right now,” he wrote back in the day, “I certainly sympathize with the idea of allowing users to collect documents in piles rather than file folders.”
In Mojave, Apple has clearly implemented the concept for the first time in a real product. Though it hasn’t used the isometric perspective that the 1992 design envisioned, which instantly gave you an idea of how many documents made up the pile, it’s virtually the same thing. Despite not being as flexible and powerful as Piles, today’s Stacks are a cool way to clean up desktop clutter–a gift from the past, but a tiny one compared to the global impact of other inventions that emerged from the ATG during the same years.
A Fateful LSD Trip
HyperCard was a programming tool invented by legendary computer engineer Bill Atkinson, the main graphic user interface developer of the Lisa, Apple’s first GUI machine, and the Macintosh. Atkinson, who was in the process of getting his neurochemistry PhD when Steve Jobs convinced him to join Apple, created HyperCard during an LSD trip in 1985. The idea was simple: HyperCard allowed computer users to put together information, like text, images, audio, and more on different visual “cards,” and navigate through these cards using hot spots called hyperlinks. The software was released in 1987 for free–something Atkinson insisted on–and became an instant hit. Millions used it to create their own programs. From multimedia databases to dentist blogs, there was a HyperCard stack–as the programs were called–for everything.
Sounds familiar? HyperCard was the World Wide Web–except it didn’t run over the TCP/IP protocol that powers the internet. In fact, it wasn’t networked at all. In 2002 Atkinson regretted not having implemented HyperCard over a network protocol but, in the mid ’80s, the idea of a permanent network between consumer computers was implausible, so Bill shouldn’t be too hard on himself.
Building A Software System That Can Play Video
The Advanced Technology Group was also responsible for digital video as we know it today. Quicktime was introduced at the 1991 edition of Apple’s Worldwide Developers Conference with a 320 x 240-pixel copy of Ridley Scott’s famous 1984 Super Bowl commercial playing on a Macintosh. It was an amazing feat, despite the fact that some critics laughed at the “postage stamp-sized” resolution. Yet they were witnessing the first consumer computer playing digital video as we know it today. QuickTime made the Macintosh the first computer with a video architecture, starting the digital video revolution that ended in trillions of hours of Netflix and YouTube streams. To give you an idea of how this early digital video looked, here’s a silly demo video included in the QuickTime 1.0 CD-ROM for developers, running at 152 x 116 pixels.
Through the years that followed that little demo, it was Apple QuickTime and its extensible architecture–which kept growing to accept multiple tracks of video and audio, digital music, and subtitles–that shaped every video standard, architecture, and codec that came to be on the internet. All the way to the 4K Ultra-HD movies you can stream through the web today.
Obviously, QuickTime deeply affected consumers but also professionals, who discovered a way to create and edit video without having to spend hundreds of thousands of dollars in proprietary equipment. It truly ignited a video authoring revolution, the same way QuickDraw and Adobe PostScript started the digital printing revolution.
The Birth of VR Video
If you’ve ever enjoyed Google Streetview or any virtual reality video that lets you look around in 360 degrees, you have the Advanced Technology Group to thank. Developed in 1994, QuickTime VR was a QuickTime extension that allowed people to create and view photographic panoramas with total freedom of movement.
The ATG came up with the idea of stitching multiple photographs into a single enveloping photo. This initially required more power than was available–in fact, the group bought a Cray supercomputer to stitch together the early demos–but soon, they had developed the tools to allow photographers to do it on their desktop Macs. During playback, viewers would only need to move the cursor around to look at different views of a room or a landscape. The software would deform the stitched photo, which was usually wrapped on a cylinder, a sphere, or a cube, to give the illusion of perspective.
QuickTime VR also came with hotspots for navigation between panoramas–an invention you use every time you navigate through Streetview, too. It also let people view objects from any angle. If you’ve ever shopped on the web and rotated a car or Lego toy in 3D space, you’ve used QuickTime VR–or a copycat technology.
A UI Trick Found Everywhere
Today, when you get an email on your phone with a telephone number, an email, or a street address, this info gets automatically highlighted so you can carry out tasks with it. This may seem like a very simple thing, but it wasn’t when the ATG came up with the idea in the mid-’90s. Until then, computers didn’t interpret text at all.
Introduced with Mac OS 8 in 1997, Apple Data Detectors parsed plain text to identify specific structures that it could understand. Then, ADD would change the text’s appearance to indicate that you can do stuff with it. The technology only existed in Apple computers, though, and was not widely used at the time because, aside from starting a new email from an address, they didn’t have much practical use. Sure, you could highlight a phone number and add it to a contact to your address book, but you couldn’t call it with a click of a mouse.
Then the iPhone came along, and Apple Data Detectors made a comeback in a huge way. All of the sudden, every piece of meaningful was clickable and actionable. For instance, a text message containing a physical address could automatically give you the option to open Maps. It was so useful and wildly popular that Google and Samsung copied it, which resulted in a lawsuit from Apple–a patent mess that is still going on today.
The end of an era
Despite its innovative work, the Advanced Technology Group was a mess. There was a multitude of projects that weren’t going anywhere, like OpenDoc, a Frankenstein technology that was supposed to allow you to make documents using software components from different manufacturers. When Steve Jobs came back to Apple in the late ’90s, to a company in a deep crisis without a clear mission, there was no immediate value in the experimental stuff the ATG was developing.
In 1997, Jobs put “a bullet through its head,” as he said at the time. He redistributed some of its engineers to other product groups around the company, which was put in emergency mode as he tried to raise it from the ashes. Its spirit, however, survived at Apple for a few years. The company kept breaking ground with new inventions and patents, which became revolutionary products like the iPod and the iPhone. And while it’s been some time since Apple revolutionized anything, the legacy of the ATG survives–on our desktops, in videos, and through the internet in general.