Click here to preview the new Fast Company

Want to try out the new

If you’d like to return to the previous design, click the yellow button on the lower left corner.

Microsoft Interns Create Ultimate Photo-Tagging Spy App

TagSense, a prototype app designed by two Microsoft interns, can automatically tag a picture with a person's name, physical activities, facial expression, and exact physical location—all without human input.


A new, creepily awesome Android application developed by two Microsoft interns turns Android smartphone cameras into full-on spy machines. The app, called TagSense, relies on smartphone sensors to automatically tag photographs with the identities and activities of whoever's in them. No human input is required.

Although it also makes use of facial recognition technology, TagSense mainly relies on the myriad peripherals integrated into the average smartphone. Android phones' built-in accelerometers are used to determine the exact physical activity a picture subject is involved in, while light sensors in the camera are used to determine weather conditions outside in conjunction with location data obtained via GPS. The phones' microphones can then determine whether the picture subject is talking, laughing, crying, or silent. All this information is then tagged onto the photos.

But the kicker for TagSense is that it is designed to tap into information stored on other, nearby smartphones. The application will be able to interact with data stored on adjacent phones that are also using the app, enabling collaborative tagging and verification of photo subjects' identities. The application's creators claim that there will be a strict opt-in requirement for use of that particular feature.

According to co-creator Chaun Qin, "phones have many different kinds of sensors that you can take advantage of. [...] They collect diverse information like motion, orientation, location, sound and light. By putting all that information together, you can sense the setting of a photograph and describe its attributes."

Both Qin and co-creator Xuan Bao are currently summer interns for Microsoft Research. Bao and Qin are working on computer science PhDs at the University of South Carolina and Duke University respectively. They both collaborated on the project with Romit Roy Choudhury, a professor at Duke University.

Qin and Bao's application also pioneers technology that could be of interest to everyone from tech firms to the media to the military. An excerpt from their technical paper gives an idea of the possibilities:

One may imagine improved image search in the Internet, or even within one’s own computer—Bob may query his personal photo collection for all pictures of Alice and Eve playing together in the snow. Another application may tag videos with important event/activity markers; a user of this application may be able to move the video slider to the exact time-point where President Obama actually walks up to the podium, or starts speaking. Today, such functionalities may be available in select images and videos, where some humans have painstakingly tagged them. TagSense aims to automate this process via sensor-assisted tagging.

The research behind TagSense was partially funded by the National Science Foundation, and Choudhury's Systems Networking Research Group receives support from Microsoft, Nokia, Verizon, and Cisco. The application itself was tested on eight Google Nexus One phones that snapped more than 200 photographs on the Duke University campus.

TagSense is still in prototype mode, but the app's creators believe a commercial product based on it could be available in a few years. The application debuted at the recent MobiSys convention in Washington.

In the short term, apart from the app's creators having secured themselves very lucrative careers, all this will only further the photo-tagging arms race between Facebook, Apple, Google, and Yahoo!/Flickr. While deeply creepy, TagSense is a prime example of where the big names will be headed in their quest to index and monetize the world's drunken party photos.

[Image: Flickr user SanchoM]

For more stories like this, follow @fastcompany on Twitter. Email Neal Ungerleider, the author of this article, here.

Add New Comment


  • Sam Duck

    A 'porn' industry, famers, (like Hollywoodites), and politicians, worst case scenario of a bad day, and availability of addresses, or the ability to attain such, goodbye rights to privacy, or as Jimmy Durante' used to state, "Goodnite Mrs. Calabash, where ever you are". A true nightmare on elm street for all the criminal mindtrenders.

  • TomMariner

    Of course -- and a harbinger of things to come -- as soon as everyone understands that smartphones are really walking sensor farms that we can harvest to do amazing things, we'll really take a big leap forward. For the smartphone designers -- folks, you are doing great with the sensors, but keep it up -- more properties, more sensitivity, more precision -- pretty soon you'll launch a thousand companies that make a StarTrek Tri-corder look like Bell's first phone.

    Oops, did I just give away the business plan -- sorry partners.

  • nealu

    Agreed. The utilization of all the technologies integrated into a smartphone is still an idea that's still in its infancy. I would be eager (and slightly scared) to see where this goes over the next few years.

  • Martin Fitz

    note to self...file under:
       ..."more reasons to never buy a smartphone..or be around people that use them"

    wait,,wait  instead, file under...

    ..."more reasons why the NSA's server budget will increase exponentially".

  • Richard Alexander

    The ability of devices to interact with each other--especially each other's data--to accomplish some goal without human intervention is full of potential, above and beyond its application to cameras or smart phones. It is interesting, however, that smart phones have so much capability within them that could be used in this way.

    I suppose the pair's next project should be a pair of goggles that would provide this information to a wearer as he is walking down the street, important people and places identified in real time by the system. Perhaps a person would be identified at a distance if he has a pre-existing relationship or is of particular interest to the wearer, or in close proximity because he interacts directly with the wearer in some way? Now, no man need cry from the gutter, "Did anyone get the license number of that truck?" Perhaps the system could even remember where one leaves their keys? Need help with a file or an application? This system could guide the wearer through their tasks (maybe not like Clippy).

    Ubiquitous photography is the next step after ubiquitous computing. These applications seem likely to eclipse conventional photography, or discrete photography. Images of the future may not be the result of explicit effort on the part of a photographer, but, rather, culled from image streams that might not even have been created by a single person. I'm sure that Canon and Nikon and the other camera manufacturers are interested in these developments. I think this is their future.

  • nealu

    They are. So are a lot of other people - these two interns tied together a series of technological developments which are of interest to many, many groups.

    Now the real question will be how to balance technological innovation with the privacy issues that will inevitably arise.