• 10.04.11

The New iPhone’s Face Recognition Capabilities Could Redefine Privacy

Along with a laundry list of other new features, Apple’s iPhone 4S will sport advanced facial recognition that’ll even find its way into apps…and the everyday life of tens of millions of iPhone owners.

The New iPhone’s Face Recognition Capabilities Could Redefine Privacy

First there were rumors based on some patent applications and thus it seemed Apple was going to use face recognition tech from its acquisition of Polar Rose in future iPhones. Then hackers digging through the code inside a beta test version of iOS 5 found hooks that confirmed face recognition was going to be a big part of it. Apple then officially confirmed it. Alongside new iPhones, iOS 5 is now imminent. And since the first rumors we’ve also learned that Apple is putting deep Twitter integration as well as a slightly artificially intelligent system called Assistant that also does speech recognition in the mix. The result is a perfect storm of technology that will bring fast, integrated face recognition tech to every hour of millions of iPhone users’ lives. Things will change.


Apple’s placed face recognition into its iPhoto and Aperture desktop apps for sometime now, allowing users to quickly and semi-automatically tag people in the photos they add to the photo library so they can be easily viewed by person category, but a patent application unearthed in March showed Apple had big plans. These included what was imagined as “iPhoto for iPhones.” The patent showed auto-cropping of faces, and was specifically about implementation of an automated slideshow on a portable device like an iPhone. The idea makes sense–our smartphones are the camera most of us carry most of the time, making them the most useful camera we own, and we’re all so engrossed in social networking that auto-face IDs seem both handy and useful. 

It turns out this was just the tip of the iceberg. When coders dug through Apple’s beta versions of iOS5 they found what were deemed to be “highly sophisticated” API systems that let an iPhone automatically track eye positions and mouth positions (so the angle to the user, and possibly where their attention is being directed could be calculated) as well as passing key data on to a face recognition algorithm that would be accessible to all apps…not just Apple’s own ones. The tricks are definitely borrowed from a firm called Polar Rose, which Apple bought relatively recently, and are demonstrated in the clip:

Cool stuff. And also, incredible. Because the implications are enormous, far beyond the scope of simple slideshows of your kids, automagically grabbed from your on-iPhone photo reel. Because face ID will be laced throughout the OS, it means social apps like Facebook and perhaps Twitter will be able to be sensitive to the ID tags, embedding them in automatically-generated status updates (should you enable such a feature). The use of face IDs could be a part of any future NFC credit card system, with the iPhone’s front camera taking a quick peek at the user’s face to check the account owner is indeed activating a payment, not some thief it doesn’t recognize. Video calling apps like FaceTime or Skype could get a turbo-boost, with the imaging system automatically tracking, video-stabilizing and perhaps choosing to crop-in to a particular speaker, rather than the typical static webcam image we’re all getting used to. Since iPhones and iPads are finding much use in hospitals, then perhaps we can speculate that medical apps, particularly medical records apps, will be safer and more secure–if there’s a “recognize the patient’s face” feature.

Some thinkers suggest face recognition could even allow Apple to break from its current single-user-log-in system for iPhones and iPads, with the device switching profiles from, say, parents’ to children’s when the device is picked up by a new user (and enabling the relevant parental censorship controls at that point, if you choose). 


Recent super-smart face swapping tools that work in real-time (like a cleverer implementation of some of the newer effects in Apple’s Photo Booth desktop app), as demonstrated recently in the clip below, could arrive in the form of specialist apps, allowing a whole new era of creative user video generation.

And those are just the direct user-facing implementations. Much subtler uses are possible, and some of these will emerge only once thousands of developers get their brains into gear: Consider how a news-summarizing app could look at the user’s face to work out where they’re spending their time looking at the screen…and could thus work out how best to dynamically organize its layout to best suit the user’s habits. How about games that watch where you’re looking on-screen and shape the gameplay dynamically? Will it be possible for a clever video-processing app, which tracks a user’s mouth position and analyzes its shape in real time, be able to do a degree of automated lip-reading?

Then there’s the issue of user privacy. Apple seems to be super-cautious about user privacy and anonymizing data about its users that it collects (as in the overblown “locationgate” scandal), but we have to ponder if it also use face recognition and user-attention-tracking data to inform its advertising efforts. Alongside that thought, the ubiquitous use of face IDs, and deep integration of social networking into iOS 5 will be bound to cause hand-wringing about the erosion of personal privacy, and whether or not people you photograph will object to being automatically ID tagged in Facebook images uploaded on the fly from an iPhone at, say, a party. 

Finally, we think Apple’s voice recognition tech–part of the Assistant feature–will utterly change how personal and enterprise owners interact with their iPhones and iPads. But if face ID and face tracking is built into the system too, then it’s an even more powerful tool.


Of course face ID systems aren’t new, but what Apple’s doing here is what Apple often does: Take an existing tech, wrap it up in a smarter more intuitive wrapper, integrate it thoroughly in a deeper, more sophisticated system, and sell it to tens of millions of users. With the 8-megapixel camera in the new iPhone 4S, plus all the other optical tweaks Apple included to kill off the use of your usual point-and-shoot camera, face ID should also be possible in many more images than with lower-resolution cameras. Perhaps even more than Facebook’s slightly controversial use of face ID technology, Apple may be about to change how the connected world thinks about privacy and information sharing. 

[Image: Flickr user avaskeg]

Chat about this news with Kit Eaton on Twitter and Fast Company too.

About the author

I'm covering the science/tech/generally-exciting-and-innovative beat for Fast Company. Follow me on Twitter, or Google+ and you'll hear tons of interesting stuff, I promise.