For years, Facebook has been assembling a team of AI experts–led by industry legend Yann LeCun–and much of that team’s energy has gone into understanding what’s going on in still images and videos. I wrote about the group’s efforts in 2015, and my colleague Daniel Terdiman dug much deeper last year.
On an intellectual level, I always understood why this initiative mattered. But I’m not sure if I fully appereciated what the upshot would be–as seen in the company’s products and services–until Tuesday’s keynote at the company’s F8 developer conference. Much of the vision that Mark Zuckerberg and others outlined, including an ambitious gambit to turn the camera inside the Facebook app into an augmented-reality platform, clearly used the creations of the AI lab as a primary ingredient.
Mike Schroepfer, Facebook’s CTO, appeared onstage to talk about the company’s research progress, giving a peek at technologies that allow the company to do things like identify people, pets, and products in a photo with a high degree of confidence or erase a moving sailboat from a video.
When I chatted with Schroepfer after the keynote, I asked him whether such capabilities were the result of sudden, specific breakthroughs or slow, steady work. “You look back,” he told me. “It’s like the movies. That moment. That afternoon. I don’t think there was that. It’s just been a concerted push over a number of years.”
The company began delving into AI in earnest in 2013, initially with its applications in advertising in mind, he says. “It takes some time to get the teams formed and up to the right size and get them working on things.”
Back when Facebook began investing in machine vision, it was obvious that people liked to share photos and videos, and that understanding those media would help the company wrangle them. But it wasn’t yet apparent that one of Facebook’s most important competitive battles would end up being with Snapchat creator Snap, which likes to declare that it’s a camera company. That visual-centric rivalry presumably influenced the camera-and-AR announcements that Facebook has lately been making, including the reveals at F8.
Schroepfer says that Facebook’s AI advances over the years have helped prepare it for these new initiatives, but also that the company’s product focus brings clarity to its research directions. “All this cool tech stuff is neat and all, but there’s always ‘What do people want to do with this?,'” he explains. “That’s where Mark [Zuckerberg] sketches out a vision.”
The days are long gone when that vision was primarily about one product. The AI effort for Oculus VR is noteworthy because it involves collaboration between Facebook researchers and Oculus researchers on challenges such as Simultaneous Localization and Mapping (SLAM), the science of determing both the real-world physical surroundings of someone inside a VR world and that person’s position within them. “The authors of those systems all work at Facebook and Oculus,” says Schroepfer, looking understandably proud.
Other Facebook properties, such as Instagram and Messenger, are more dependent on centralized Facebook AI research, but are powerful ways to turn new technologies into experiences that can reach large numbers of people. “They’re pulling the rest of us in a lot of ways,” Schroepfer says of the newly fast-evolving Instagram.
At first, much of the AI involved in mobile services such as most of Facebook’s offerings was conducted in the cloud, where it’s possible to throw entire server farms at computationally intensive tasks. More recently, however, it’s been possible to perform advanced AI directly on smartphones. That’s both because mobile hardware is getting better and because researchers are getting better at wringing as much number-crunching efficiency as possible out of their computer science. “It’s not just that the CPUs are getting faster,” Schroepfer says. “We make better algorithms.” And he adds that it’s a crucial advance.
“Who wants to record a video, upload it, have awesome effects applied, download it, and say ‘That’s no good?,'” he asks. “It’s just not fun.”
The Smartphone And Beyond
One of the things that was striking about the AR announcements at F8 was that they were centered around a device that’s already pervasive: the smartphone. That’s not to say that Facebook isn’t planning for a future involving more out-there gadgets with an AI angle, as Microsoft is doing with HoloLens. It is. But Schroepfer says that consumerizing AR now in a way that’s eminently doable with mainstream hardware is a good way to get the world ready for the out-there stuff to come.
When the iPhone arrived a decade ago, he notes, it seemed familiar despite all its departures from past devices because it let people do things they were already comfortable with, such as make phone calls, check email, and surf the web. If Facebook’s AR platform helps popularize useful things that can be done by overlapping the digital and real worlds, it will make future AR-ready eyeglasses or contact lenses or other devices feel less foreign, a feat that will be crucial to their adoption. “That’s a really hard consumer proposition, when you’re building a new category,” he says.
Schroepfer doesn’t mention Google’s ill-fated Google Glass–which arrived before consumers had any idea what AR was or why they might care–but he doesn’t have to; the contrast with what Facebook is trying to do is still clear.
And when might AR become magical in the way it only will when you don’t even have to think about the electronics involved? “The actual hardware of making these mythical glasses, we’re way in the ‘R’ part of the ‘R&D’ of that,” he says, briefly removing his own glasses and brandishing them as if they were a futuristic set of goggles with built-in intelligence.
Beyond that, he makes no predictions about timing. “AI and AR,” he says, “are two places where multiple smart people disagree.”
Recognize your brand’s excellence by applying to this year’s Brands That Matter Awards before the early-rate deadline, May 3.
