Hear the new experience designed to bring people who are blind into the excitement of a tennis match

The highs and lows of this year’s Australian Open have been delighting fans around the world, especially during global lockdowns. But not everyone is able to enjoy the thrill of live sports in the same way. For the tennis fans among the 600,000 blind and visually impaired people in Australia—and millions more around the world—experiencing sports broadcasts can be underwhelming. Kala Petronijevic, a young girl who has low vision, says her dad has to commentate every move to her. Maurice Gleeson, a blind tennis fan and the CEO of Blind Sports and Recreation Victoria, says: “Tennis, sadly, hasn’t been anywhere near as accessible as I would like.”

But, this weekend, the blind and low vision community (BLV) will have a brand-new way to experience the upcoming men’s and women’s finals.

It’s a confluence of two existing technologies, says Tim Devine, AKQA’s executive creative director. The spatial data is captured by Hawk-Eye, a computer vision system already used in various sports, which is made up of a series of high-speed cameras that track the trajectory of a ball in real time. It’s often referred to as electronic line calling (or goal-line technology in soccer). In fact, for the first time the Australian Open has this year done away with human line judges in favor of these “robo-referees.”

The new system converts that spatial data into 3D audio, or binaural sound. That’s the best way to digitally hear audio as we do in real life, where we instinctively know where, exactly, a sound is coming from. It’s a much more complex version of the older concept of surround sound. The technology has lately been developing fast due to its usefulness for virtual reality.

See what it’s like during a tennis match:

In tennis, that means people can hear different audio sounds according to exactly what the ball is doing, instead of just hearing an indistinguishable mono sound whenever the ball hits the ground. The team wanted to re-create the sensation of hearing a ball flying into the corner of the court at 180 kilometers (more than 110 miles) per hour, for example. “It’s incredible, and I think you miss out on that if you just hear a hit,” Devine says.

Now try it without any visuals:

With Action Audio, different elements of the game have different sound cues. (They used some of the concepts from blind tennis, Devine says, where the ball often contains beads to make it rattle or jingle.) You hear a blip each time the ball hits the ground, in your left or right ear, depending on where the ball is. The closer the ball is to the perimeter of the court, the more blips you hear. As the ball moves around in the air, you hear a jingling bell sound. You hear a high pitch for forehands and a low pitch for backhands.

“I can actually hear everything—and I can see the ball, for some reason,” says Petronijevic, after experiencing the new system. “I wouldn’t watch it any other way,” says Adam Fayad, who also has low vision. “You can just enjoy the game so much better.”

For the developers, tennis seemed like a good place to start for “sonifying” sport, since it involves only two players, with few other distractions. It’ll be more complex for sports with more elements, like soccer or cricket, since the technology is fundamentally about deconstructing the experience to all the individual actions that contribute to a live sport experience and finding a way to represent them sonically.

The key may be in a new app that the team has developed. Using similar audio recognition techniques as Shazam, the app works to detect any live sports broadcast and overlay the 3D audio technology onto that game. It would essentially create a secondary audio stream for the broadcast. “This isn’t a difficult thing for broadcasters to pick up,” Devine says, adding that it’s most important to keep working with the BLV community to hear what they need to create the most meaningful experiences. “Now, it’s time to continue to listen again,” he says, “and get people to help shape the next sport, the next iteration.”