When it comes to taking a good photograph, the human eye is still the best sensor. What happens to the image after that is probably something a machine can do just as well or better, especially given the massive number of images piling up in our photo albums today. And that’s why the breakthroughs in the next era of imagery are going to be done by computers. Apps are already arriving that help us explore the possibilities, ranging from retouching tools to software that addresses the enormous availability of images in the digital age.
Here are five photo apps that are changing the way you think about pictures.
We are taking more photos than at any other time in history. Billions of images are being uploaded, many never to be looked at again. And they might just include something that you don’t want the rest of the world to see. But who has time to scour a photo archive looking for the stray bong or worse?
The idea that we need to focus on limiting–rather than creating–images is one way in which our ideas about photos are changing in 2014.
A service that neatly explores this concept is Social Sweepster. It’s a tool that scans your online social presence (currently Facebook and Twitter, but soon expanding to Tumblr and Instagram) and flags questionable images that you may not want in the public domain.
“Our primary user would be someone who has recently graduated from college and is looking to clean up their photos ready for job applications,” says founder Tom McGrath. Recruiters regularly look at Facebook and Instagram accounts as part of their employee screening process. One recent study even demonstrated that it’s possible to predict job performance based on the pictures on a person’s Facebook profile. That hilarious picture of you passed out at your end-of-year college party? Maybe not so funny now.
As with many next-gen smart photo tools, Social Sweepster doesn’t just look at the image itself to gather data. In addition to computer vision, the software also uses text recognition algorithms to sift through keywords associated with images. It’s even possible to examine the context of images, since metadata can regularly reveal where a photo was taken.
“We’re really trying to tackle one of the hardest computer vision problems out there, which is recognizing images in the wild,” McGrath continues. “Recognizing a single beer can in a photo that’s small, low-resolution, and badly lit is a real challenge. Being able to do that–and do it accurately–is very, very tough.”
Removing a stray plastic bag from a photo of people praying at the Ganges might get you disqualified from a National Geographic contest, but for everyone else it simply makes for a better picture.
Whether it’s removing a photo-bombing stranger from that lovely shot of you and your family, or taking out an ugly hotel from an otherwise stunning landscape scene, one popular request for photo apps is the ability to touch up existing photos. Unlike other methods of removing unwanted detritus from pictures, TouchRetouch intelligently carries this work out on your behalf, rather than requiring time-consuming manual work. Just select the image component you wish to remove, and leave it to the software to do the rest. Once an element has been selected, the app smartly analyzes what is going to be required to fill a certain area, and then sets about filling it using image components cloned from other parts of the photo.
The end result is impressive–and developer Kostyantyn Svarychevskyy credits it with the new processing power of smart devices, which can now carry out the kind of intensive graphical work that previously would have required a much larger graphics-oriented machine.
“Increase of computational power of smartphones provides the possibility of using new technology or advanced algorithms and user experience,” he says. “In newer versions we [also plan to] try to improve this technique on more complex backgrounds, such as buildings.”
The idea that the massive quantity of images we gather today opens up new possibilities for photographers is the idea behind Vhoto. “We talk about camera ubiquity a lot as a team,” says creator Noah Heller. “What does it mean when everyone carries devices with multiple cameras built into them? And what happens when those cameras are on all the time? You have to ask yourself what you’re going to do with this amazing amount of content.”
Vhoto uses computer vision technology to scan your videos to find and extract the best photographic moments. “The concept that you have to press a button to take a single picture is a really old idea that goes back to chemical cameras,” Heller continues. “That no longer has to be the case. If you want a record of a great moment in your life, why not just let the camera go and then let technology sift out and sort the best end images. Our mantra is that users should think of photography as fishing with a net, not with a hook.”
Some of the metrics Vhoto examines are fairly straightforward: sharpness, clarity, color, and the presence or absence of a face or smile. But the model also takes into account more abstract features like novelty, context, and composition. Rather than the photographer having to be consciously aware of all these elements, the app learns preferences based on the past behavior of individual users so it gets better at predicting what photographic elements you’re likely to be interested in. Whatever pictures you end up sharing, saving, or otherwise interacting with will be analyzed so that future similar images can be elevated within the model.
“It’s not our job to force people to like photos a professional critic might say is better composed, it’s our job to help people get the photos that they want,” Heller says. “If our users turn out to like photos with a certain color composition or facial expression, that’s what our machine learning model needs to deliver.”
The rise of Instagram has made filters increasingly popular, but some tools take the concept of post-processing pictures further than others. Color Thief is an example of a great color correction app that should be on every budding smartphone photographer’s device.
“Color Thief takes the colors from one of your photos and transfers them to another,” says creator Aaron Barsky. Blurring the line between functional image modification and something entirely new, Barsky likens the app to challenging a painter to repaint your photo, using only the color palette from another photo of your choosing.
“We count how often a color is used in both photos,” Barsky continues, explaining how the app functions. “We then transfer the most frequently used color from the source to the most frequently used color in the target, and similarly down to the least frequently used color.” The challenge, he says, is in grouping colors together. “A photo could have hundreds of subtle shades that a human would identify as light blue–but the computer sees as completely different colors. We use ‘mathemagic’ to make sure the color transfer happens with smooth gradients of color.”
Although Color Thief is a post-processing step for images, rather than a camera app in itself, it still benefits from the improved quality of smartphone cameras. “Color Thief works best on photos that have a sharp in-focus foreground with a blurred background,” Barsky says. “As our users can take better and better photos with the built-in camera, the more fun they’ll have remixing those photos with our tool.”
It started as a computer vision research project at the University of British Columbia, and now AutoStitch is a panoramic photo app that leaves it rivals in the dust. It has two major benefits over other similar apps, as well as the built-in panoramic functionality found in an increasing range of smartphones.
It’s a versatile tool that doesn’t require taking a single sweep shot. As long as the images overlap in some way, the photographer is free to experiment with images in any order or arrangement–including horizontal, vertical, or a mixture of both. It’s even possible to stitch together photos taken with different camera apps, as well as those imported from other devices.
The quality of the finished images is also vastly superior to other panorama apps. Inputs are composed of full resolution images, which allows for each photo to be composed individually. The overlapping regions of these high-def photos are then automatically blended to ensure seamless transitions between images. The end result is an impressively professional panoramic photograph.
“By using the other sensors on board, and with the sheer processing power available, the door is open to create tools that will take smartphone cameras beyond what is possible with traditional cameras in many ways,” says developer Geoff Clark, speaking about the future of smart camera apps in general. “Augmented reality shooting guides that analyze the images in real time, or light-field capture that allows for re-focus of images, are a couple of examples.”
So go ahead and snap all the photos you like. Just put them somewhere accessible to the algorithm that’s going to make them worth looking at again.