Apple Imagines A World You Digitize By Touch, And They Aren’t Alone

As creative apps go mobile, companies are developing new tools to capture the world.

Apple has filed a patent application for a stylus that you can touch to any surface, take a photo, and store the texture and color data into a 3-D image file. Easy to imagine: you’re sitting at Starbucks, drawing on your iPad. You need to add a nice wood grain to a render that you’re working on, and instead of searching for one via online databases, you simply tap your pen to the table, digitize the walnut finish, and add it to your scene.


Will Apple will ever release this stylus? Who knows. But the patent is part of a larger trend in creative tools: as apps like FiftyThree’s Paper and Adobe Comp make tablets a reasonable substitution for desktops, our peripherals are evolving along with them, creating a future where, instead of just digging through image databases at our desks, we’ll be able to hunt, scan, and catalog the colors, textures, objects, and art in the world around us, as if we’re collecting specimens for our illustrative mad science experiments.

At SXSW, when I met with Scott Belsky, the founder of Behance (who now leads mobile app development at Adobe), he showed off a series of new apps Adobe had been developing around the idea of grabbing media in the wild. One app can identify any font your camera sees; another can grab a texture, like fabric or grass. One can build color palettes from your surrounding, so if you’re inspired by a piece of interior decorating, it can inform a menu you’re producing for a client.

In each of these cases, a camera snaps a piece of media, uploads it to Adobe’s Creative Cloud, and is available for use immediately. “We had a brainstorm recently about the marketing angle, and explaining the zen of what we’re creating,” Belsky said. “And one person was like, “Get out.’ I was like, ‘What do you mean, get out?’ and he said, ‘All I’m imagining is all these work stations with empty chairs.'”

Designers leave the office to be inspired. It’s why designers from automobile companies and design firms often travel the globe for project research, to be influenced by both global aesthetics and cultures, to see familiar materials used in new ways. What’s different now is that we can bring our computers with us. They’re smaller than ever–tiny enough to slide into our pockets–and work with media acquisition peripherals loaded with new sensors, made cheap by the economies of smartphone scale. It’s a mix that’s going to lead to an explosion of new media acquisition devices.

Adobe BrushAdobe

Georg Petschnigg, founder and CEO of FiftyThree, currently sells the most popular tablet stylus in the world, the Pencil. Pencil is purely an input device, focused on stretching the bounds of what you can do on a touchscreen. But Petschnigg is bullish on the impact of new, differently shaped sensors in our pockets. He points to the smartphone camera as an early disruption in this space. When everyone had a camera in their pocket, everyone became a photographer, and apps like Instagram allowed us to further modify and share our photos. But now, we’re on the precipice of the next step in this evolution.

“There’s a huge trend where people are tricking out their mobile devices,” says Georg Petschnigg, founder and CEO of FiftyThree. “The emergence of the selfie stick…who would have guessed? But it’s really a modification on the cellphone to take a better photo.”


The smartphone camera has been the most popular modifier to date. From wide angle attachments, to microscopes, to this Super8 camera body that’s raised $182,000 on Kickstarter and counting. Google’s Project Tango puts two cameras on the phone, to create stereoscopic, 3-D maps of our environments. In fact, there’s a whole field of these object and room mapping startups out there like Scanify, Structure, and Sense.

Truth be told, much of this innovative quirk may be born from the market’s sheer desire to experience new, never-seen tech. “Consumers [in the U.S. and U.K.] have had smartphones for 8+ years and are familiar to the point of boredom with their smartphones, so they’re looking for products that extend what their iPhone or Android can do,” explains Djordje Dikic, CEO of Palette (formerly Swatchmate), which just closed $1 million in series A funding to expand the sale of their Cube digitizer, a pocketable box that accurately scans any color in your environment with an internal LED. He also points to a looming technology that will only increase our appetite for digitization tools: virtual reality. Digitizing the natural world becomes of huge commercial interest when companies like Facebook want to create an entire virtual one.

But the camera and various imaging technologies are only a tease of things to come. Petschnigg points to microphones and even GPS as other sensors that are in ready supply but barely tapped in the creative process. “All of those wearable devices and internet of things stuff i think is really interesting,” he says. “They give the human more senses, the ability to perceive the world around you.”

The possibilities of our sensors fit to new styles of industrial design will only push the explorer movement of media acquisition, allowing us to think less about how we acquire media, and think more about media as a byproduct of our own journeys. Take how New Deal Design designed the original Lytro–a camera that could take a photo that you could refocus after the fact. Rather than being shaped like a normal dSLR or film camera, New Deal modeled it after a kaleidoscope.

“The trend is very clear–if you’ve got a good analog tool, why not add some function and making it digital, using ages of use-culture to deepen the object’s cultural grounding?” Amit says. “Plus, many traditional tools had an exceptional ergonomics and usability.”

Which brings us full circle to Apple’s patent for a world-scanning stylus. Maybe Apple will release it. Maybe they won’t. But either way, Apple is just one of a number of companies that’s designing the future of mobile creative tools that will untether us from desktop scanners in windowless rooms, and allow us to mine and digitize the real world for inspiration.



About the author

Mark Wilson is a senior writer at Fast Company who has written about design, technology, and culture for almost 15 years. His work has appeared at Gizmodo, Kotaku, PopMech, PopSci, Esquire, American Photo and Lucky Peach