The New iPhone's Face Recognition Capabilities Could Redefine Privacy

First there were rumors based on some patent applications and thus it seemed Apple was going to use face recognition tech from its acquisition of Polar Rose in future iPhones. Then hackers digging through the code inside a beta test version of iOS 5 found hooks that confirmed face recognition was going to be a big part of it. Apple then officially confirmed it. Alongside new iPhones, iOS 5 is now imminent. And since the first rumors we've also learned that Apple is putting deep Twitter integration as well as a slightly artificially intelligent system called Assistant that also does speech recognition in the mix. The result is a perfect storm of technology that will bring fast, integrated face recognition tech to every hour of millions of iPhone users' lives. Things will change.

Apple's placed face recognition into its iPhoto and Aperture desktop apps for sometime now, allowing users to quickly and semi-automatically tag people in the photos they add to the photo library so they can be easily viewed by person category, but a patent application unearthed in March showed Apple had big plans. These included what was imagined as "iPhoto for iPhones." The patent showed auto-cropping of faces, and was specifically about implementation of an automated slideshow on a portable device like an iPhone. The idea makes sense—our smartphones are the camera most of us carry most of the time, making them the most useful camera we own, and we're all so engrossed in social networking that auto-face IDs seem both handy and useful. 

It turns out this was just the tip of the iceberg. When coders dug through Apple's beta versions of iOS5 they found what were deemed to be "highly sophisticated" API systems that let an iPhone automatically track eye positions and mouth positions (so the angle to the user, and possibly where their attention is being directed could be calculated) as well as passing key data on to a face recognition algorithm that would be accessible to all apps...not just Apple's own ones. The tricks are definitely borrowed from a firm called Polar Rose, which Apple bought relatively recently, and are demonstrated in the clip:

Cool stuff. And also, incredible. Because the implications are enormous, far beyond the scope of simple slideshows of your kids, automagically grabbed from your on-iPhone photo reel. Because face ID will be laced throughout the OS, it means social apps like Facebook and perhaps Twitter will be able to be sensitive to the ID tags, embedding them in automatically-generated status updates (should you enable such a feature). The use of face IDs could be a part of any future NFC credit card system, with the iPhone's front camera taking a quick peek at the user's face to check the account owner is indeed activating a payment, not some thief it doesn't recognize. Video calling apps like FaceTime or Skype could get a turbo-boost, with the imaging system automatically tracking, video-stabilizing and perhaps choosing to crop-in to a particular speaker, rather than the typical static webcam image we're all getting used to. Since iPhones and iPads are finding much use in hospitals, then perhaps we can speculate that medical apps, particularly medical records apps, will be safer and more secure—if there's a "recognize the patient's face" feature.

Some thinkers suggest face recognition could even allow Apple to break from its current single-user-log-in system for iPhones and iPads, with the device switching profiles from, say, parents' to children's when the device is picked up by a new user (and enabling the relevant parental censorship controls at that point, if you choose). 

Recent super-smart face swapping tools that work in real-time (like a cleverer implementation of some of the newer effects in Apple's Photo Booth desktop app), as demonstrated recently in the clip below, could arrive in the form of specialist apps, allowing a whole new era of creative user video generation.

And those are just the direct user-facing implementations. Much subtler uses are possible, and some of these will emerge only once thousands of developers get their brains into gear: Consider how a news-summarizing app could look at the user's face to work out where they're spending their time looking at the screen...and could thus work out how best to dynamically organize its layout to best suit the user's habits. How about games that watch where you're looking on-screen and shape the gameplay dynamically? Will it be possible for a clever video-processing app, which tracks a user's mouth position and analyzes its shape in real time, be able to do a degree of automated lip-reading?

Then there's the issue of user privacy. Apple seems to be super-cautious about user privacy and anonymizing data about its users that it collects (as in the overblown "locationgate" scandal), but we have to ponder if it also use face recognition and user-attention-tracking data to inform its advertising efforts. Alongside that thought, the ubiquitous use of face IDs, and deep integration of social networking into iOS 5 will be bound to cause hand-wringing about the erosion of personal privacy, and whether or not people you photograph will object to being automatically ID tagged in Facebook images uploaded on the fly from an iPhone at, say, a party. 

Finally, we think Apple's voice recognition tech—part of the Assistant feature—will utterly change how personal and enterprise owners interact with their iPhones and iPads. But if face ID and face tracking is built into the system too, then it's an even more powerful tool.

Of course face ID systems aren't new, but what Apple's doing here is what Apple often does: Take an existing tech, wrap it up in a smarter more intuitive wrapper, integrate it thoroughly in a deeper, more sophisticated system, and sell it to tens of millions of users. With the 8-megapixel camera in the new iPhone 4S, plus all the other optical tweaks Apple included to kill off the use of your usual point-and-shoot camera, face ID should also be possible in many more images than with lower-resolution cameras. Perhaps even more than Facebook's slightly controversial use of face ID technology, Apple may be about to change how the connected world thinks about privacy and information sharing. 

[Image: Flickr user avaskeg]

Chat about this news with Kit Eaton on Twitter and Fast Company too.

Add New Comment

5 Comments

  • Sarah Tags

    I personally don't think that face recognition technology on the iPhone, and any facet or use of that technology, have more implication for "bad" uses versus "good" uses than any other technology. There will always be hackers that will manipulate technology, and there should be, as I believe they can spur innovation. 
    Some of the points against this sound like fear of all technology, just like the fear of the internet, the fear of computer viruses, etc. It's like saying I am afraid of electric nail guns because, although they were intended for making houses, they could be used as weapons available without license at any local hardware store.
    By the way, the 800M Facebook users tend to imply that our faces and our lives are already broadcasted all over the world. It should be the broadcast part, not the capture part, that should be the issue.

  • Rick Noel, eBiz ROI, Inc.

    Aside from the many legitimate privacy concerns, a positive
    application for facial recognition technology is to provide more robust authentication
    making it harder for pedophiles to impersonate children in their online predatory
    activities.

  • Kristen Chalker

    Wow, the possibilities in technology are truly endless today.  The fact that the face recognition can do so many "cool" things is interesting, and fascinating, but for me it stops at that.  First of all, I think the iPhone is technologically advanced enough.  In a world of smart phones and on the go everything, where do we draw the line?  There are so many apps and extra features that it wouldn't surprise me if they got rid of the ability to make a phone call at some point.  It seems as if the focus is only on everything else, not the actual purpose of a phone - which is to make phone calls.  Or at least it was at one point.  It's hardly fair to say that that is still the case today.  I feel as if the face recognition technology will end up being more trouble than it's worth.  It already weirds me out that my phone and iPod can be located if I lose them or if there is an emergency.  Who's to say someone isn't tracking me all the time?  The same goes for the face recognition.  It almost seems like it is smarter than the smart phone.  And in the wrong hands the technology could be abused.  I can see it being helpful for patient recognition in hospitals and what not, but if people could already hack and find out that this technology existed in the first place, chances are they will be able to take that same technology and use it in ways that it should not be used.  I know that Apple is always trying to do the latest and greatest, and in many ways they are creating smart and useful products, but sometimes I feel as if they are more concerned with the money that will be made and less with the practicality or safety of the user.  Especially when it comes to privacy.  Our face and our lives could be broadcast all over the internet or other places without our consent thanks to what Apple considers a good idea.  I am interested to see how this saga plays out.

  • Ian Watson

    I agree, everyone is very focused on the hardware but the face recognition technology, combined with the cloud is very powerful. Siri is exciting as well. The last time Apple took a technology out of SRI it was the mouse and that had a huge impact. Read more on my blog http://universal-tool.blogspot...