Apple, Microsoft, And The 3-D Desktop Of Your Dreams

Soon you won’t have to open up a window on your PC. Rather, you’ll dive through it into a 3-D world of icons and files. We’re not joking.


The movies, game consoles, and smartphones have already gone 3-D crazy. Soon we may routinely print out things in 3-D tangible form. But thus far there’s one aspect of consumer tech that’s remained largely 2-D, no matter how excited sci-fi movies special effects guys get about ’em: PC desktops. That’s about to change.

Both Microsoft and Apple are busy working on banishing your flat, boring old array of 2-D icons and windows with faux dropped-shadows and replacing them with a 3-D interface. It’s about time.

On the eve of releasing a publicly downloadable early build of Windows 8, which leverages the decidedly two-dimensional look and feel of its touch-based Windows Phone 7, Microsoft teased the world with a couple of visionary ideas from its research labs. The most eye-grabbing was its 3-D Windows environment.

It’s a radical reimagining of the design of a computer–with the user’s hands operating behind the screen on a traditional keyboard and trackpad. The screen’s a special transparent OLED unit made by Samsung, though, so this is no rendered flight of fancy. By placing the screen in front of the user’s digits it means their fingers are visually immersed in the displayed icons–with windows and other UI objects literally at their fingertips. That’s an inversion of the way almost every desktop UI has worked since Xerox PARC came up with the Windows/Icons/Mice/Pointers concept that inspired a young Steve Jobs, because here your fingers are translated into a virtual interaction with the desktop via pointers and arrows and so on.


The advantage should be immediately obvious from the video: You want a file from your collection? You reach for it in 3-D space. You want to resize an image or reposition a desktop element? You push it, pull it, or use one of the commonly accepted moves like pinch. It’s a 3-D extenstion of the way you interact directly with 2-D desktop items on an iPad, iPhone, or an Android-powered device, and you can bet that using it leads to a very profound sense of actually working within with your machine, compared to the slightly distant “tapping at your box of tricks to get it to do something with your files” we all do nowadays.

But Apple, whom the history books tell us did the most to innovate the 2-D desktop computing meme we’re all familiar with, is also investigating 3-D. A recent patent award, as noted by, even gives the firm what may be a “foundational” patent in 3-D desktop organization.

In Apple’s imagined 3-D desktop, the user’s files are organized in piles in 3-D space like in a real office. Some may be pinned to the wall, some are alone, others are in stacks (and there’s no reason that expert users may choose to keep things “real,” as in this virtual 3-D environment you could stack things on the ceiling, or leave them floating in the air). Though this sounds like a simple reorganization of the way we all work with files in folders now, the actual innovation is that instead of having to do something arcane like double-tapping on a file to use it, Apple’s guessing it may be more intuitive to gesture at the file. Something like zooming it into the center of the field of view to work on it, then pushing it away while you consult another document. Graphical transitions would reposition and resize the image representing the flle in real time, making interacting with this kind of desktop feel even more fluid and natural than current tricks like symbolically dropping an unwanted file in the trash.

Are you blinded by science and highly suspicious that this sort of interaction would be confusing and pointless? Set aside that NIMBY-ism now, and do an experiment with your actual desktop. Where’s the Word file you’re working on? Right there in the middle of your laptop screen, front and center. Look for that Post-it note about the slightly important meeting next Tuesday: I bet it’s either on the edge of your monitor or perhaps slapped on your desk itself, a little out of reach so you don’t hide it and prominent enough you don’t forget it. If you’re a bit old-school you may even have a paper desk calendar handy just underneath your laptop. Remember where you’ve dropped that folder of files you need to keep handy, but don’t need right this moment–it’s probably right on the edge of your desk, possibly in a pile of things that have similar priority. Your system for doing this may be more organized than your cubicle-mate’s chaotic and messy stacks of things, but in use it’s structurally pretty similar (and some theories even suggest the messiest desks are the most efficient).

Basically we control our use of our physical space in direct reaction to what we’re doing, with importance and attention space directly mapped onto where we put things on our desk. As more and more of our work and play moves into the virtual world of a computer, it makes sense to build the user interface in a way that makes the most of our long-honed physical desk management skills–and turbo-boost them with new features (like zoom, interactive video-chats with colleagues, and so on) that you just can’t do in real life. With brains like MS and Apple working on this, the learning curve may be a bit steep…but the product’s utility is going to be fabulous, and you can bet that once you’ve used it you’ll consider 2-D desktops quaint.

Clever stuff, and we made it all the way to the end without a single reference to Minority Report. Darnit.


[Image: ZMKstudio via Shutterstock]

Chat about this news with Kit Eaton on Twitter and Fast Company too.


About the author

I'm covering the science/tech/generally-exciting-and-innovative beat for Fast Company. Follow me on Twitter, or Google+ and you'll hear tons of interesting stuff, I promise.