Remember all the times you snickered when some TV character zoomed in on a photo and said “Enhance!” turning a blurred mess into a highly detailed, razor-sharp image? Now there’s a system–an artificial intelligence-powered image-scaling program called EnhanceNet–that will make that film trope impossible to laugh at, because it’s now a technological reality.
Just look at the images above. They’re not fake. EnhanceNet can take a grossly pixelated, low-resolution photo of a bird and turn it into a crystal-clear image. It even reproduces the camera’s depth of field from the original photo, as you can see in the foreground tree branches.
Its inventors–Mehdi S. M. Sajjadi, Bernhard Schölkopf, and Michael Hirsch at the Max-Planck Institute for Intelligent Systems in Germany–classify their software as a “Single Image Super-Resolution Through Automated Texture Synthesis.” That’s exactly what it does: It scales up very low-resolution photos by synthesizing textures that introduce new detail into the resulting high-resolution image. The synthetic detail makes the resulting pictures so realistic that they are almost indistinguishable from the actual photos. Perhaps for the first time ever, artificial intelligence can now re-create reality with uncanny accuracy–by using fake details.
According to the numerous experiments described in the researchers’ paper, “the outcome achieves a significant boost in image quality at high magnification ratios.” The software creates these images by “feed-forward fully convolutional neural networks in an adversarial training setting.” In other words, like other adversarial neural networks, one system generates results while the other one evaluates the accuracy of the result.
The technology puts current high-end photo scaling methods to shame. You can see the huge difference in this comparison between the top state-of-the-art method (PSNR, or “peak signal-to-noise ratio”) on the left and the result obtained by EnhanceNet on the right:
The first falcon is comparable to the scaling available in Photoshop, which always results in over-smoothed images that inevitably lose definition and don’t pass the human eye test. The boost in quality and detail on the EnhanceNet result is obvious.
Talking over email, Mehdi S. M. Sajjadi claims that “the algorithm could easily be put in commercial software such as Photoshop.” What’s more interesting is that you could actually make it part of the operating system “on smartphones, enhancing the quality of images as you zoom into them to avoid blurriness” in real time, Sajjadi says. There are additional applications for this, he adds:
From upsampling old movies to 4K-quality, restoring old family photographs that are too blurry when you want to get a large print over to more general applications such as improving object detection. This is something we’ve actually studied in the paper. It turns out that using our algorithm on images makes it easier for other neural networks to detect objects in images, which has wide applications from Google image search to detecting pedestrians in self-driving cars.
Imagine that: One neural network enhances the image, then another one uses the result to detect objects. Of course, that also potentially includes law enforcement applications. Sajjadi told me that even while the “details that are reconstructed are not necessarily the true ones [. . .] the techniques can indeed be used in special cases such as license plate recognition.” That means that police would not have to wait for ultra-high resolution cameras to catch people speeding. They could use this technology to do it right now.
You could also use this system, Sajjadi says, to un-pixelate faces that have been pixelated, like those in anonymized photos. And while “the reconstructed face is not necessarily identical to the original one, since most information has been lost during the pixelation process [. . .] it may help with identifying someone.” This would not work as evidence in court, he says, but you could identify people you couldn’t identify before.
This technology could pose a real threat to privacy. Someone could get the code for this and train a neural network to re-build an image that was blurred or pixelated before. If you’ve ever uploaded amateur porn with pixelated faces, prepare to get unmasked.