Current Issue
This Month's Print Issue

Follow Fast Company

We’ll come to you.

2 minute read

Microsoft Interns Create Ultimate Photo-Tagging Spy App

TagSense, a prototype app designed by two Microsoft interns, can automatically tag a picture with a person's name, physical activities, facial expression, and exact physical location—all without human input.

TagSense

A new, creepily awesome Android application developed by two Microsoft interns turns Android smartphone cameras into full-on spy machines. The app, called TagSense, relies on smartphone sensors to automatically tag photographs with the identities and activities of whoever's in them. No human input is required.

Although it also makes use of facial recognition technology, TagSense mainly relies on the myriad peripherals integrated into the average smartphone. Android phones' built-in accelerometers are used to determine the exact physical activity a picture subject is involved in, while light sensors in the camera are used to determine weather conditions outside in conjunction with location data obtained via GPS. The phones' microphones can then determine whether the picture subject is talking, laughing, crying, or silent. All this information is then tagged onto the photos.

But the kicker for TagSense is that it is designed to tap into information stored on other, nearby smartphones. The application will be able to interact with data stored on adjacent phones that are also using the app, enabling collaborative tagging and verification of photo subjects' identities. The application's creators claim that there will be a strict opt-in requirement for use of that particular feature.

According to co-creator Chaun Qin, "phones have many different kinds of sensors that you can take advantage of. [...] They collect diverse information like motion, orientation, location, sound and light. By putting all that information together, you can sense the setting of a photograph and describe its attributes."

Both Qin and co-creator Xuan Bao are currently summer interns for Microsoft Research. Bao and Qin are working on computer science PhDs at the University of South Carolina and Duke University respectively. They both collaborated on the project with Romit Roy Choudhury, a professor at Duke University.

Qin and Bao's application also pioneers technology that could be of interest to everyone from tech firms to the media to the military. An excerpt from their technical paper gives an idea of the possibilities:

One may imagine improved image search in the Internet, or even within one’s own computer—Bob may query his personal photo collection for all pictures of Alice and Eve playing together in the snow. Another application may tag videos with important event/activity markers; a user of this application may be able to move the video slider to the exact time-point where President Obama actually walks up to the podium, or starts speaking. Today, such functionalities may be available in select images and videos, where some humans have painstakingly tagged them. TagSense aims to automate this process via sensor-assisted tagging.

The research behind TagSense was partially funded by the National Science Foundation, and Choudhury's Systems Networking Research Group receives support from Microsoft, Nokia, Verizon, and Cisco. The application itself was tested on eight Google Nexus One phones that snapped more than 200 photographs on the Duke University campus.

TagSense is still in prototype mode, but the app's creators believe a commercial product based on it could be available in a few years. The application debuted at the recent MobiSys convention in Washington.

In the short term, apart from the app's creators having secured themselves very lucrative careers, all this will only further the photo-tagging arms race between Facebook, Apple, Google, and Yahoo!/Flickr. While deeply creepy, TagSense is a prime example of where the big names will be headed in their quest to index and monetize the world's drunken party photos.

[Image: Flickr user SanchoM]

For more stories like this, follow @fastcompany on Twitter. Email Neal Ungerleider, the author of this article, here.