It’s one more crack in the fabric of reality as we know it: Researchers at the University of Washington and Facebook have described their work on software that can take any image containing a human body—whether in a painting or a photograph—and automatically create an animated character that walks through the still image.
Appropriately, they call it Photo Wake-Up.
[Image: courtesy University of Washington]
On their project page, UW computing scientists Chung-Yi Weng and Brian Curless, along with Facebook’s Ira Kemelmacher-Shlizerman, describe a process that can take one single photo and create a character that walks out of the frame toward the viewer. They can also make the character run, sit, or jump.
The researchers will present their algorithm later this month at the Conference on Computer Vision and Pattern Recognition in Long Beach, California, but the software begins by analyzing a still image to detect a human form and fits a morphable body on it. From there it creates a body map, labeling each of the parts. Once the map is done, the software uses it to construct the three-dimensional mesh of the character, estimating the weight of their body so it can later apply realistic motion. Finally, it takes the original image and uses it to build a texture around the 3D body. The final step is to reconstruct the background that is occluded by the person’s form so they don’t leave a white hole behind.
Their technology works with anything you can throw at it, from a photo of Paul McCartney on the cover of Help to one of Picasso’s famed portraits. You can see it in action here:
In a press release, Kemelmacher-Shlizerman says that this partially solves one of the fundamental problems with computer vision, which researchers previously considered impossible to resolve. “The big challenge here is that the input is only from a single camera position, so part of the person is invisible,” she explains. “Our work combines technical advancement on an open problem in the field with artistic creative visualization.” Curless points out that, while previous efforts used several photos from different points of view, “you still couldn’t bring someone to life and have them run out of a scene.”
[Image: courtesy University of Washington]
As the researchers demonstrate, this technique is particularly impressive when you use an augmented reality headset to view it. They propose a method that will allow you to look at any photo or painting, snap your fingers, and have it step into the real world in front of your eyes. This would be a really cool way to interact with a museum exhibit, for example. According to the scientists, a more immediate application of the technology would be to create game avatars out of still images or drawings—or use it to bring children’s drawings to life in a game.
The software has its limitations. For example, it can’t handle people that have their arms or legs crossed. But this is still an early version. As Weng says, “Photo Wake-Up is a new way to interact with photos. It can’t do everything yet, but this is just the beginning.” Indeed. It seems likely that their research—which was funded by Facebook, Huawei, and Google—is just a glimpse at the way similar AI will rapidly break down the wall between fact and fiction online.
Recognize your brand’s excellence by applying to this year’s Brands That Matter Awards before the early-rate deadline, May 3.
