Last month, the Google Arts & Culture app released an experimental new feature that uses machine learning to match your selfies to famous paintings. After percolating for a few weeks, the feature took the internet by storm over the weekend, with many thousands of people sharing their results online.
The tool is built on top of the vast archive of digitized art that Google Arts & Culture has created over the last seven years by partnering with art institutions and museums around the world. It uses facial recognition to create a biometric fingerprint of your facial features, then compares it to the more than 70,000 works in the app’s database. The tool is AI at work–designed to show off both the Arts & Culture project’s mission of making art more accessible and Google’s machine learning technology. The company includes a disclaimer that it won’t use the data from your selfie, nor will it store the photo beyond the time it takes to find a matching painting.
Some people were thrilled with the results of the app, which asks you to take a photo of yourself and the presents a series of comparable painted people, along with the percentage match.
Torn between which one I think is better likeness with the Google Arts and Culture app. pic.twitter.com/uSw8RmOip8
— Felicia Day (@feliciaday) January 13, 2018
But for others, the app presented some rather horrifying results, making it more amusing than accurate. What’s particularly entertaining when looking at other people’s results is seeing bits of the algorithm’s logic–even when the two images truly don’t look alike. In this photo, a woman with curly hair and arched eyebrows matches with a wig-wearing man with similarly arched eyebrows.
Honestly fuuuuuuuck google arts and culture pic.twitter.com/8W4gLO48PJ
— Linz (@LinzElah) January 14, 2018
I’ve taken like 20 selfies with the Google arts & culture app and gotten this horrifying guy as my top result EVERY SINGLE TIME, who do I sue pic.twitter.com/PjuvT0IXIU
— Caroline Wazer (@CarolineWazer) January 13, 2018
I too spent quite a bit of time retaking selfies to see if I could get the algorithm to match me with a painting that looks a little bit more like me. But try as I might, I’m a 53% match with Mrs. Robinson from a painting by Sir Joshua Reynolds and a 48% match with Marchesa Elena Grimaldi Cattaneo by Sir Anthony van Dyck. Not terrible, but not great either.
The app is both a testament to the progress Google has made with machine learning, and its failings. Some users have turned the tool’s outputs into memes, choosing their own “match” images to highlight how silly the tool can be.
Wow I mean wow google arts and culture … pic.twitter.com/i2ujsiA30N
— James Adomian (@JAdomian) January 13, 2018
The app’s deficiencies, particularly for people of color, also point out how white-washed the history of art is–and how neither Google nor its partner museums have chosen to include much art that represents all types of people.
The google arts and culture selfie feature is just a reminder of how little Asian representation there is in art history pic.twitter.com/pmfQsZ7xf2
— nydthakid (@nydiahartono) January 11, 2018
The tool is available via the Google Arts & Culture app, but so far it hasn’t shown up in every user’s app. Many have expressed frustration online that they downloaded the app hoping to try the feature but were disappointed when it wasn’t there. That’s because Google has limited the number of people who can try it by region. I’ve reached out to the company to inquire when the tool will be released more broadly and will update this story when I hear back.
You can try your luck in the roulette of machine learning selfie matching by downloading the app here.