advertisement
advertisement

This AI has synesthesia

Here’s what a machine “hears” when it looks at art.

This AI has synesthesia
[Images: Makalo86/iStock (pattern), Wiki Commons]

In 2016, MIT researchers released an intriguing paper. Using 2 million videos, they had taught a neural network to do something novel: recognize scenes or objects based on what they sound like. Based only on the noise in a video, it could detect a coral reef, for example, or a plane taking off. They called it SoundNet.

advertisement

This year, the DJ, artist, and Qosmo CEO Nao Tokui flipped the concept on its head. His project, Imaginary Soundscapes, is a convolutional neural network that hears sounds when it looks at images. Based on a given image, the software will choose from 15,000 sound files to find the “soundscape” that fits. First, he applied the software to Google StreetView to create an audio tour of the world, with AI-generated sounds to accompany any scene from StreetView, from echoing voices in Barcelona’s Sagrada Familia cathedral to chirping birds on rural backroads. Viewers could “immerse themselves into the artificial soundscape ‘imagined’ by our deep learning models,” Tokui explained on Medium. More recently, he launched the project as a tool anyone can try using any image. On the demo, he offers a few examples of what his AI “hears” when it looks at fine art, including works from artists like Monet at Cezanne. It’s like stepping into a living, breathing landscape painting.

Take Hubert Robert’s Ponte Salario. Painted in 1775, it’s a gauzy depiction of an ancient bridge in Rome, now crumbling and populated by 18th-century people. How does the AI interpret this scene, as an auditory hallucination? With a cascade of wheels on cobbles, bird caws, ambient and barely audible murmurs, and a ripple of bells. Listen here.

[Image: National Gallery of Art]

What does it hear when it sees a still life, like this Monet from 1862? The tinkle of flatware against a glass on a wood table–and, fascinatingly, a very anachronistic hum from an electric kettle. Listen here.

[Image: Wiki Commons]

And when the neural net encounters a more abstract painting, like Paul Klee’s 1917 Persian Nightingales, things get even more interesting. In Klee’s pastel-toned field of stars, birds, and banners, the AI “hears” an otherworldly layering of chiming bells, layered with the chirp of a bird and a dog barking in the distance. Listen here.

[Image: National Gallery of Art]

It’s a strange and evocative experiment–and one of several recent examples of the way many researchers are experimenting publicly and openly as the field evolves. Check it out for yourself right here.

advertisement
advertisement

About the author

Kelsey Campbell-Dollaghan is Co.Design's deputy editor.

More