I’m being hired as a babysitter, and I need to take the perfect, convincing photo for the job. I’m told this babysitter should look cool and wear sunglasses. But as the clock ticks down to take my photo, I realize, I have no sunglasses in reach.
Will I fail at this task? No way. I grab a toddler food bowl and my coffee cup, holding them over my eyes. The face-scanning algorithm hunting for sunglasses is fooled. The shutter clicks. And I pocket a virtual $5.67, the fee I get for looking like a babysitter.
Welcome to the gig economy of Facework. It’s an interactive game by artist Kyle McDonald, alongside Greg Borenstein, Evelyn Masso, and Fei Lui. The goal is to audition for jobs based entirely on how you look. But the larger point is to expose a world that only computer scientists and big corporations get to see.
Facial recognition algorithms are everywhere—in our iPhones, Instagram, Google Photos, and global network of surveillance cameras. But they’re black boxes, and faulty ones at that. Even our best image-scanning algorithms have massive, logical holes. Modern AIs are able to spot a person wearing sunglasses easily, but they might see a baby bowl and a coffee cup held up to your eyes as sunglasses too.
Learning to exploit that absurdity is what Facework is all about.
So imagine yourself as a Lyft or Uber worker, but instead of driving places in a car, you make certain faces into your computer for a fictional business. Your photos are then vetted by a facial recognition AI, and you earn money based on how “accurately” you look the part.
McDonald developed the project through his own experiments with an image set called LFWA+, out of the Chinese University of Hong Kong. This set is essentially a big, open-source pile of human faces—13,000 in all—which are labeled with qualities such as “mouth partially open” or “receding hairline” or, ugh, even “attractive woman.” Starting with this set, McDonald was able to train his own AI system to identify such qualities in his own software, with new faces. You may know that process as machine learning.
“One thing I do as an artist is try to look at research in academia and reproduce it so that I can understand it better and critique it better,” says McDonald. “As artists, one of our roles is to be cultural critics on the outside for the sciences . . . but we don’t always take the opportunity to try our hand at the technical side of things. I feel like I have the right background to do that and discover a lot of weird things quickly.”
As McDonald began to play with his own AI system, he was taken by how it analyzed his face, not in a single snapshot, but with real-time-estimated “accuracy.” That means while you or I can say with 100% accuracy that a person is wearing sunglasses, an AI is always taking a best guess, and the way you tilt your head or change your lighting can measurably impact it.
Platforms such as Instagram hide the constant assessments, which might appear as shifting sliders or numbers, from the public. But coders on the back end can see accuracy waver in real time in their own UI.
“When I was seeing these bars wobbling on my screen, sliders shaking back and forth, I felt like this is an experience I never had before,” says McDonald. “I realized this kind of analysis happens behind the scenes so often, out of our sight, and we never have a chance to get an intuitive feeling of what face analysis is doing. We’re told it’s bad. Facebook is doing it, or Google is developing new technology. But we never get to feel it.”
In these sliders, McDonald discovered his game mechanic: A meter that shows, in real time, how convinced an algorithm is that you are wearing a mask, have rosy cheeks, or are a CEO. (Yes, all of these are qualities that the AI can supposedly see.) In this way, Facework externalizes the internal work that algorithms do every day. “Because it’s so obscured from view, that’s part of the reason we don’t have well-informed conversations about it,” says McDonald. “For face recognition especially, those systems are kept outside the view of the average person.”
The game is enlightening, but also a lot of fun. When I have to be chubby, I puff out my cheeks, jumping from 30% certainty (thanks, quarantine pounds!) to 79% certainty. When I have to be a CEO, I’m passing with only 9% certainty. I’m surprised! I, like most CEOs, am a white man. Wouldn’t that be enough to juice my results? So not knowing what to do, I grimace. That bumps me to 17%. Not quite convincing but, somehow, more convincing than my resting face.
Facework is free to try on your phone or laptop. And all of your own face data stays locally on your machine, meaning it’s not collected by McDonald for future use. The entire game takes a quick 10 minutes and includes some fun narrative twists and turns that I won’t spoil here. And while it’s an entertaining diversion, chances are, Facework’s lessons in AI will stick with you into the future.