What Happens When You Let A Neural Net “Curate” Art

Artificial intelligence built for the Tate digs into the gallery’s century-old collections to bring old works new life.


The question of whether life imitates art is a philosophical one, posed most notably by Oscar Wilde in a 1891 essay promoting Romanticism over Realism. But ask a computer, and it will inevitably take the rationalist approach: what does the data have to say?


That’s the premise of Recognition, a computer program created for the Tate’s IK Prize, a digital art competition held by the Tate and Microsoft which asked designers to use AI to explore the art in the gallery’s collection. It uses several types of machine learning to match art in the Tate’s extensive British art collection to photos of current events from Reuters. By analyzing both visual and thematic similarities between two images, Recognition can come up with a fine art doppelganger to any event or figure in popular culture.

Coralie Gourguechon, Monica Lanaro, Angelo Semeraro and Isaac Vallentin.[Photo: Marco Zanin]

Recognition, which won this year’s competition, was created by four designers at the Benetton-backed research center, FabricaCoralie Gourguechon, Angelo Semeraro, Isaac Vallentin, and project manager Monica Lanaro. The idea for the platform–and subsequent Tate exhibition–is both to resurface older works in the collection and experiment with how emergent forms of machine learning like deep neural networks can intersect with images and art.

“It’s a fascinating but unexplored territory with lots of potential,” says Semeraro. “How can we apply rational, objective thinking to a subject topic like art? [The Tate] collection inspired the idea that we can link our everyday to works of art.”

Recognition does this by analyzing an image in four different ways–isolating any identifying objects, or faces, as well as the image’s composition and its metadata–and searching the Tate’s collection to find an image that closely matches. For the program to “see” each image, the Fabrica team utilized several forms of artificial intelligence that sit behind the platform’s interface. They worked with web developers at JoliBrain on the AI, who developed some of their own algorithms and also built on existing technology.

Take the object recognition, for instance: using the open source deep learning servers DeepDetect and DenseCap, JoliBrain developed a deep neural network that finds an object from an image–an apple on a table, or a man wearing a suit–and labels it creating a short sentence. A similarity search engine then looks for a similar object in the Tate collection. The algorithms for matching image composition do something similar, but look for artistic elements like line and color instead of specific objects. For facial recognition, the team used Microsoft Cognitive Services‘s Computer Vision and Emotion APIs, which analyze any existing faces for age, gender and general emotion state.


Analyzing photos for context is more complicated: the team used a variety of deep neural networks to process both the images and their metadata to find semantic matching among words or sentences between the two images. Because of this mix of visual and textual factors, sometimes Recognition‘s matches are striking in their visual similarities, and other times the similarity is less obvious. Gourguechon points toward a photo of swimmers that the program pulled up during the Olympics as an example of this: the image was matched with a painting of people sitting around a table. The similarity wasn’t clear until looking at the data, which Recognition displays alongside each image in the match.

“The algorithm saw on the torso of one of the swimmers a tattoo of a woman’s face, and the face was at the same angle as the people around the table,” she explains. In this way, the tool’s limitations reveal connections and similarities that human beings might not make. “Right from the beginning we wanted to combine the subjective point of view and the algorithm’s rational point of view,” says Gourguechon.

Exploring how a computer “sees” art can be entertaining–analyzing a photo of Donald Trump, the computer interprets his hair as a “man’s hand on head”–but it also highlights how far machine learning has to come. In the meantime, projects like Recognition, and designers willing to present emerging AI technologies in compelling ways, are helping the field along.

You can play around with Recognition here, or visit the Tate, where Recognition is being displayed in its own exhibition.

[All Images: courtesy of Fabrica]

About the author

Meg Miller is an associate editor at Co.Design covering art, technology, and design.