You’ve probably have seen 3D reconstructions of real-world games on TV. But these require plenty of human work to recreate. Now, the University of Washington, Facebook, and Google are developing technology that uses artificial intelligence to turn a regular soccer game TV transmission, and in theory footage of any sport, into a 3D game. The tech could change the way we experience games–and even affect the game itself.
The new system automatically transforms a monocular video–in other words, footage captured by a single camera–into a moving 3D reconstruction that you can watch in augmented reality.
The key to the system is its artificial intelligence engine, a convolutional neural network (or CNN) that’s capable of estimating a depth map of the field and each player. The CNN is trained using regular TV transmissions of games combined with real 3D tracking data of players, which means it can guess where every player is located on the field as well as the movements of those players based on its training. The novel system was presented this month at the IEEE Conference on Computer Vision and Pattern Recognition by the team of researchers behind it–Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, and Steve Seitz of the University of Washington’s Graphics and Imaging Laboratory.
The results are quite impressive, as you can see above, and open up a new way to experience soccer and any other game you train the software with. You can watch any given game from any position you want at your desk, getting closer or farther away. While the system doesn’t work in real time yet, I can imagine a near future in which we can immerse ourselves in games, “teleporting” ourselves into the stadium with help from high-fidelity virtual reality headsets.
The same tech could influence the way soccer itself is judged. A real-time version of this system would eliminate the need for VAR, the video assistant referee system that relies on a crew of people in front of several TV screens to decide dubious penalties or off-sides. It’s easy to imagine an artificial intelligence referee judging the whole game by “looking” at those same video feeds–which is sure to be just as controversial as real-life refs.