2017 will be remembered as the year in which the wall between real and fake video began to crumble. Thanks to neural networks, we can synthesize fake humans from a few frames of video or get any person to say whatever we want. And now, it’s possible to turn any still photo of a person’s face into a moving, expressive video that can be controlled by another person. It’s just like the iPhone X’s animojis–but using a human face instead of a 3D animal. Watch the incredible video below:
Previous facial animation software required video of the target face to capture enough frames to show different expressions and emotions in the resulting synthetic video. But this new algorithm–presented at Siggraph Asia 2017–completely bypasses the need for a video source and works strictly from one single still photo of the target face.
To bring this still photo to life, the software tracks the facial expressions of one person–using a normal video feed, no need for a fancy 3D camera–capturing points that the algorithm uses to animate a still photo of another person using those points. The neural network corrects any visual artifacts that may arise in the transformation of the photo and adds hidden detail that doesn’t exist–like the interior of the mouth–to complete the illusion.
There are many potential real world uses of this technology. The most obvious is an animoji-style feature that transforms a selfie into a facial puppet for hours of karaoke using another person’s face. A company like Snapchat could also implement a feature in which someone could impersonate a friend to send a message to another friend–which sounds like a fun prank, but could quickly turn into a vehicle for abuse.
But there’s one possible use of their tech that may come to your desktop or smartphone sooner than you’d think: The researchers’ demo video shows how they can implement a reactive profile photo on your Facebook wall–which can be animated dynamically in real-time according to people’s reactions to your posts. If someone is visiting your wall and reacts “angry” to one of your wall posts, your photo will animate to make an angry face that you never made. If it’s a happy reaction, your avatar will smile and so on. You can see how this works in this video capture:
Tel Aviv University PhD student Hadar Averbuch-Elor–one of the research paper’s coauthors–says that this is a research project and that “there are no plans to release the software to the public.” However, two other coauthors, Johannes Kopf and Michael F. Cohen, are both engineers at Facebook Research’s Computational Photography group. “The goal was to show that we can animate a single image,” Hadar tells me in an email. “What Facebook will do with this–I don’t know.”
It’s reasonable to imagine that Facebook would like to incorporate such a fun feature into its platform as soon as possible. But even if that doesn’t happen, you can be sure we’ll see it elsewhere soon. Hadar added that while they “don’t plan to make the software publicly available in the near future, [they] published all the algorithmic details in [their] research paper.”
Less fun are the possible unintentional uses of this technology–like anyone being able to trigger World War III by sending Donald Trump a video of Kim Jong-Un telling him what he thinks of his fake hair.