Does this video contain a cat?
Is the cat inside, or on the beach?
Is it night or day?
These are the kinds of questions SoHo-based startup Dextro can answer in real time about hundreds of public videos streaming through the Periscope live video app at any given moment. To rapidly analyze and sort all these videos, Dextro relies on computer-vision algorithms its founders have been developing since late 2012. The technology works in a similar way to the face-detection software Facebook uses to automatically tag photos, but Dextro's system can recognize thousands of moving objects and scenes.
Dextro's new tool, called Stream, launches today as a mobile-optimized site that sorts Periscope videos by their content: Cats, computers, swimming pools, and talking heads, to name a few popular categories. The system does not analyze stream text titles, which are often non-descriptive; instead, it groups videos based only on how its algorithms interpret the visual scene being filmed. Dextro already uses this technology to analyze pre-recorded videos for companies—for example, a brand may want to know which videos on Instagram contain its products—but this is the first time the two-year-old startup has applied its algorithms to live streams.
"A lot of interesting research work has been done, for example, on the images you might see if you did a Google Image search for pizza," Dextro CEO David Luan told Fast Company. "But we realized the value for the market is being able to analyze cluttered video scenes and still see the pizza."
The Stream tool helps to solve a major problem with the Periscope app, which Twitter launched in March. When using the app, it is easy to find videos from people you follow as well as videos currently being streamed by random strangers around the world. But it is not yet possible to search for videos by content, a limitation that sometimes adds to the fun of using the app—you never know what you're about to watch—but ultimately holds Periscope back from becoming a more robust social network.
With Stream, users can view a menu of videos to watch, but the experience isn't seamless. Stream operates through a mobile site, which means you must hop between your phone's web browser and the Periscope app. The design of the Stream site is a bit confusing at first: Topics appear as overlapping circles that are bigger or smaller depending on how many videos are available to watch in that category. Click on a circle to see a list of live videos that match that topic.
Since the team first began scanning every Periscope video last Tuesday, they have noticed a few persistent trends. "Selfie streams are the most common," says Luan. "There are a lot of nightclub streams from Turkey. And a lot more people stream dogs than cats."
From my experience using the tool, it accurately sorts the vast majority of videos, but it is not perfect. One video of a young woman talking to her phone's camera in a bedroom was labeled "beach side." Her ceiling was painted sky blue and her walls were bright white, so it was obvious why the algorithm tripped up.
But the technology is constantly improving. Dextro's team feeds pre-recorded videos to the system, giving it information about what it is seeing ("dog in a kitchen") and in this way the algorithm learns over time to recognize many types of dogs and kitchens. Dextro does not store the Periscope videos it scans, and it does not analyze audio information.
Luan, one of Dextro's founders, says he has not contacted Twitter about integrating with the Periscope app. That's likely because the Stream tool is merely a side project for Dextro, which was founded by Luan and Sanchit Arora in 2013. Luan and Arora created the system as a way for robots to "see" the world and analyze it in real time. For example, a robot tasked with opening a door will first need to identify the door, then the knob, and so on.
When Luan and Arora posted their project to Hacker News in January 2013, it got tons of attention from the tech community, and, soon after, from companies that saw other potential uses for the technology. The two friends were still students, and it was not until that summer that they could work on the startup full-time. Today, Dextro has seven employees and works with "large companies" (Luan will not give any more information about his clients) to help them organize massive video archives and analyze videos from social media.
"A lot of brands are thinking about how these live-stream apps fit in," says Luan. "Brands want to know when they are mentioned on these platforms." The algorithm can identify not only specific logos, but also distinctive product shapes, and it can tell whether people in the video are interacting with the product.
Of Dextro's long-term goals, Luan says, "we see ourselves being the visual intelligence layer for the Internet." The goal is to make videos as easy to search and analyze as text is today.