Karen Palmer is out to scare people. For the past three years, she’s been plunging people into a stressful and potentially deadly riot, firmly believing that they’ll emerge changed for the better. The riot is fiction–that is, it’s really just a movie, in which actors play rioters, cops, and protesters–but due to a boost from technology, as well as good old-fashioned tricks from the world of theater, it can feel unnervingly real.
The experience deploys scent, haze, lifelike sound, and set dressing, in the form of a wrecked garbage can, a traffic cone, and trash strewn around the floor. There’s also a real-life actor dressed as a riot cop who aggressively questions participants as they are escorted to the viewing area, ensuring that they begin the film with hearts already pounding. Finally, the film uses facial-recognition technology to analyze people’s emotions while they watch. At several key moments throughout the film, the technology reads the viewer’s face–are they exhibiting calm, anger, or fear?–and what happens next proceeds from there. Anger might prompt a cop to shoot you; calm will send you on to the next sequence. In other words, your emotions determine the outcome of the film.
The project was first inspired by unrest in Ferguson, Missouri, following the police killing of Michael Brown, an unarmed black teenager, in 2014. It is also part of a larger preoccupation Palmer has with fear, which began years earlier when the Brit started doing parkour, a form of movement derived from military obstacle course training. Palmer is fascinated by the emotional and psychological aspects of the discipline.
“In parkour, you learn that people can jump a certain distance on the ground, but if you’re up high jumping the same distance across open space, your brain will tell you that you can’t do it,” she notes. “So parkour is about navigating through fear.”
Interest in the project has been overwhelming: Palmer fields a constant stream of requests to demo it at festivals and events. It showed in the U.S. for the first time at the Future of Storytelling Festival in New York last year, and is currently on view at the Lucid Realities exhibition at the Phi Centre in Montreal; in 2016, the film won a “Digital Dozen: Breakthroughs in Storytelling” award from the Columbia University Digital Storytelling Lab. Palmer has spoken about it at the Google Cultural Institute Lab in Paris and the Copenhagen International Documentary Film Festival and in a just-recorded TED talk. At the moment, she’s in New York completing two fellowships at once—one with ThoughtWorks, and another with TED.
Palmer is now developing RIOT into a longer and more complex, 360-degree experience, able to accommodate multiple points of view, and perhaps deploying a greater range of technologies. She hopes to debut it in mid-2018. She spoke to Fast Company from ThoughtWorks’ bustling office in Manhattan about what to expect from the final RIOT, where film could go, and what can be made with our fears.
Fast Company: Why the interest in fear?
Karen Palmer: Because we’re constantly in a state of it. We are living in a society now where we are being bombarded with imagery and information aimed at elevating our level of fear. Like a shooting happens, it’s the worst shooting in American history, or you’re on the subway in New York, “Report anything suspicious on the subway . . . ” There’s so many things to keep you in a constant state of fear. But on a day-to-day basis, you’ve probably got nothing to really be fearful of. It’s more about what’s happening internally, in you, than what’s happening outside in the world.
And that’s what I want people to be more consumed with: their own emotional state and how they can navigate through that, because then that will reflect the world around them back to them. So I want people to go through this experience to almost replicate the sensation of going through fear. And if we can come out of that, then we will be able to have a much better comprehension of the world around us.
FC: In the current version of RIOT, the viewer watches the film on a screen, and there are four different points when the facial recognition software reads their face and the narrative branches accordingly. What will the final version be like? I know you mentioned adding in multiple points of view . . .
KP: The vision is for it to be a 360 film, a film that’s completely around you. What would happen is, you step into this space, and you hear the sound of a petrol bomb going over your head spacially–there’s 3D sound all around you–and the screen will light up and you’ll see the word “RIOT.” Then, a riot cop will run toward you and say, “Don’t move!” Then a looter will come toward you and say, “Look I’ve got something,” then a civilian will say, “Look, I need help, my shop’s on fire,” then another character like an anarchist will come toward you and say, “Come with us.” And then, depending on your eye tracking, that will determine which narrative you pursue.
I’m working with ThoughtWorks to develop the technology, which could range from eye tracking to more facial recognition to maybe gait [analysis], or infrared, with your body temperature. We are experimenting with all those now. And through TED, I’ve also been talking to anti-surveillance people, people in disruptive tech, people who’ve been in riots, developing the narrative from all aspects, from the riot cop to the anarchist and everything in between. Then I’ll turn that into a new film with my coproduction partner, which is the National Theatre’s Immersive Storytelling Studio [in London].
FC: Right now in particular, it feels like people are very much entrenched in their views. But you genuinely believe perspectives can be changed, is that right?
KP: I feel that shifting perspective is really viable, and that it can be done through my type of storytelling, where you’re engaging emotionally. As opposed to changing someone’s perspective through reading a newspaper article, say. Because that’s very information based. But we don’t respond to and interact with the world from a place of information, we respond from a place of emotion. When we go home and speak to our partners or flatmates, we go, “Oh wow, that person, they’re amazing,” or, “I hate that person”–it’s from an emotional perspective. So that’s why I put people in the middle of an emotional experience to then reflect back their reality.
I’m working with the Perception Institute as my R&D partner for the behavioral science aspect of the piece, and we’ve been talking about how you can measure or quantify changes in perception. They said when they study people watching dash-cam footage of police brutality, they can determine from the observer’s eye tracking where their empathy lies. If their eye tracking follows the trajectory of the cop, that means they’re empathetic with the cop. If their eye tracking follows the trajectory of the person being brutalized, they’re empathetic with the person being brutalized.
If you think about it, that makes sense. When I watched the video of Sandra Bland, the black woman who was arrested by police, then later died in police custody, I was like, “Oh my God,” because I was imagining being her in that situation. Because I was empathetic towards her. But if you’re empathetic towards the cop, you’ll probably feel, “Oh that cop is under threat.”
So I said to the Perception Institute, if I could change the tracking of the person, could I then change their perspective, and then change their empathy, and their perception of the situation? And they said, in theory yes, you could do that, but no one’s ever asked us that question. That’s why they’re interested in working with me.
FC: Shifting perspectives through filmmaking is also interesting to think about in terms of all the different types of footage we now see. There are the police body cams, people livestreaming from inside cars during traffic stops, cell phone videos from bystanders . . .
KP: Yes, that’s going to be part of my next RIOT iteration. You’re going to be able to float from maybe the riot cop’s body camera, to someone else filming a situation, to maybe a CCTV footage camera, to maybe a newscaster’s perspective of a situation–just seeing those five scenarios of the same situation, but they’ve got completely different perspectives. And together, maybe you’ll get the total story.
FC: You started out as a traditional filmmaker. How did you end up at the forefront of this kind of interactive, experiential storytelling?
KP: Maybe 12 years ago I realized that storytelling was no longer going to be a passive, linear experience. I felt that people would want to be participants within the storytelling process. I started to make projects that explored reflecting your emotional state back at you, through film. That evolved into the concept of moving through fear. So I started working with technologists, and with different forms of technology. Prior to this we were developing for my installation EEG experiences where you would put a sensor on your head, and the film would respond to the algorithms of your brain.
My most recent partnership over the last three years has been with Brunel University London and their computer science department. I worked with Dr. Hongying Meng and his students there to develop facial recognition and artificial intelligence systems.
FC: How does the technology work?
KP: So in RIOT, the narrative of the film responds to your emotions. The emotions are monitored by facial recognition software, which is driven by machine learning and artificial intelligence solutions. So what happens is, you watch the film, and the film watches you back with a webcam. The Brunel University PhD students created a library of expressions that determine whether the observer is calm, angry, or fearful. And what happens is as you’re watching the film, it’s comparing your expression to that of their library database of emotions. Once the dominant emotion at the end of a fixed period of time is determined, the narrative will respond. It will go on a different narrative branch in real time if it determines you were calm versus fearful.
FC: What kind of responses have people had to RIOT? I know you just showed it in October at the Future of Storytelling Festival (FoST) in New York.
KP: There were some amazing reactions at FoST. Two people actually started crying. One of them was a woman from Iran, and she got calm during it, but afterwards it was like she was holding in the whole weight of the world. I was like, “Are you okay?” and she said yes, but then got tears in her eyes.
And what I’ve realized is that I need to add another component to the experience called emotional debrief. Because right now there’s this kind of pre-experience, where you listen to instructions from the person in uniform, and see footage from the G20 riots in America, then you have the experience. But I realized there’s a post-experience too, where the person talks about their experience and has an emotional release. Because when you’re doing these new forms of storytelling with new technologies, whether it’s augmented reality, virtual reality, etc., it has the potential to affect people in ways that are different to other experiences.
Related: More Coverage Of The Future Of Policing
I’m not saying a linear film cannot be powerful, but MIT has done studies showing that your brain cannot decipher the difference between VR and reality, because of the way that your brain is neurologically reading the information. And I’ve created an experience where I’m using ambiophonic sound, which basically is as close to the human ear as you can possibly get, so your brain is fooled that this is actually what you’re hearing, and then I’m using scent, and I’m using haze, and I’m using film, so it’s a different type of emotive experience.
The other person who had a very powerful response was from Argentina, and she said to me that she’d been brought up in a dictatorship, and she had felt she was quite at peace with that, but what happened was that when she saw my film, three-quarters of the way through she was very calm, but when she got to the last part, where a young black girl is being handcuffed by the riot cop, she felt viscerally really, really angry. And the narrative on the film showed her anger. And she said, “Wow, I really thought I had come to terms with what I had been brought up in, but I really hadn’t.”
It’s interesting–in my mind, I’m crafting a particular type of experience that’s deliberately uplifting and positive and engaging and visceral. But when you put it out as an installation, then you see what works and what doesn’t work, and what aspects of it work more powerfully than you think. You don’t know what will happen, because everything is completely new when you’re working in these new forms of storytelling. A lot of the things that are happening, I’ve never experienced before, and a lot of other people may not have experienced before. It’s a new genre, and has the potential to and is often affecting people in a different way neurologically then they’ve been affected before.
And actually, two hours before the woman started crying, I bumped into Lance Weiler from Columbia University, who’s director of the storytelling lab there, and who I’ve been in a residency with. We were talking at length because he’s experienced my project, and he said, “What happens with people afterwards?” And I said, “They’re very emotional.” And he said, make sure you are sensitive to that emotionality, because that’s a big part that people aren’t doing with this new form of storytelling.
FC: Could you ever go back to traditional filmmaking? Or is this the future?
KP: Good question! I just saw the [trailer for the] new Black Panther movie and was like, “Oh my God, I should be a traditional filmmaker because, wow! It’s smashing it!” But, I’m immersing myself in my school of filmmaking.
When I came up with the idea for facial recognition and storytelling, I couldn’t even Google it. No one had done it before. Even 10 years ago when I became intrigued by it and started picking it up, my filmmaker friends were like, “What are you doing? It’s so quirky.” They couldn’t even understand what I explained to them.
And now my filmmaker friends are more like, “I’m trying to make an interactive film,” or a common thing is, “I’m trying to make a VR film, can you talk to me about that?” I get a lot of people on Facebook saying, “Hey I’m making a VR film, can you give me some advice?” And my first question is: “Why?” Why are you making a VR film? I know it’s really hip to the hop right now, but just because everybody’s doing VR doesn’t mean you should, too. Is VR the most appropriate format for your story? Part of storytelling is to be able to select the technology appropriate to the story.
FC: What other advice would you give to fellow storytellers?
KP: I would never tell anybody how to do their work. What I do is, I examine what I’m doing and why I do it. Like, what is the purpose of doing this stuff? I have a larger purpose in what I want to do, and that defines how I proceed.
As a storyteller, I see it as a responsibility almost to return to a kind of modern mythology, where storytelling was to empower the community, or teach or impart wisdom to the community, as opposed to its very base bottom dimension now, where storytelling is to sell a product or perpetuate a culture or lifestyle or brand, or show just one piece or perspective of a situation. I would like to use media to empower.
I feel that we are living in a time of unprecedented global social unrest. It’s like the ’60s, with the Civil Rights movement, but on a much larger scale, whether it’s Paris or Venezuela, or parts of America like Ferguson or Baton Rouge, or Bulgaria, Turkey, Spain–it’s just everywhere. And I feel a responsibility as a storyteller to create a dialogue, and do my part to create some kind of resolution. It might not be a resolution in my lifetime, but it needs to start. It needs to change.
Watch An interview With Karen Palmer:
Recognize your brand’s excellence by applying to this year’s Brands That Matter Awards before the early-rate deadline, May 3.
