At the end of the parking lot behind a glorified Chili’s off one of the main drags in Park City, Utah, is the New Frontier pavilion at the Sundance Film Festival. During a 10-day stretch in which it’s not uncommon to find yourself checking your makeup in the bathroom mirror alongside Kirsten Dunst or dancing to Samantha Ronson’s DJ set with Harvey Weinstein, it was refreshing to discover this pocket of festival activity that eschews glamor and instead glorifies the technologically innovative side of filmmaking.
During the festival, in an overheated back room located past “Evolution,” a large-scale 3-D collage of seemingly ever major moment in motion picture history, Newsweek journalist turned documentary filmmaker Nonny de la Peña explained her interactive installation, Hunger in L.A.
The innovative project combines filmmaking, augmented reality, and journalism to recreate a real incident that took place two years ago at a food bank line in L.A. De la Peña used game development software Unity 3D, motion tracking, and a head-mounted goggle display, combined with live audio she recorded during the incident itself to create an immersive, and affecting, experience.
As de la Peña describes how she created the project, her colleague John Brennan follows a man around as he walks in circles occasionally reaching his hand out or crouching down to interact with avatars only he can see. He’s watching a simulation of the events at the First Unitarian Church’s distribution line when a man waiting for food went into a diabetic coma. Six and a half minutes later, Brennan taps him on the shoulder and helps him out of the goggles. “Wow,” the man exclaims, visibly affected. “That was intense.”
After viewing the film myself, I sat down with de la Peña to find out what inspired this unique and provocative project.
You’ve been making documentary films for a long time. How did that evolve into the kind of interactive, immersive cinematic experience we see here?
In early 2007 my partner Peggy Weil and I decided to try to build a virtual Guantanamo Bay prison. We turned your computer screen black then put a picture of you being put on a C-17 transport plane and dumped into a cage. You’re free to move around and all the audio in there is from the real prison and pictures and stuff like that but we took away your agency. That was the first time we began to explore: A. What does it mean to do a spatial narrative? And B. What is it to do what we were calling an embodied edit? You walk into this replica of the interior of a C-17 transport plane, we ask you to put on a HUD, a heads-up display, the screen goes black and when it stops you are in another place. That’s an embodied edit, going from one place to another.
You built another project that simulates torture, tell me about that.
(The Guantanamo Bay piece) led to an invitation to work with one of the most brilliant virtual reality researchers in the world, Mel Slater. We decided to do a piece on what it’s like to be in stress positions for five or six hours. We took interrogation logs that had been released by the Bush administration of someone who had been tortured. We had the transcript read by actors and then we did something called the binaural recording. If you record audio using a dummy head where the microphones are in the ear canal then you are capturing sound the way we really hear it. We took that binaural recording and we played it through a wall and we asked people to sit straight up in a chair with their hands behind their backs clasping their hands and not touching any other part of their body except for their head. You’d look down into the display and you’d look like your knees were up and you were crouched over in a stress position. Afterwards, when they came out of it we’d say, well, what was your body like? Everyone reported being hunched over when in fact they were not hunched over at all. They all reported being in the stress position.
Okay, so how did that merge with your interest in journalism?
After [working with Mel] I was like, wow this is a really, really powerful place to build. If the mind can be tricked that easily we really need to start thinking about best practices in journalism. Very clearly gaming platforms are going to be utilized for journalism in the future and how do we start thinking about it in a way that would portray these stories accurately?
And that led to Hunger in L.A.?
Eventually I decided I wanted to tell one story with a linear narrative that might actually have more impact. Initially my idea was that we would record audio at a food bank when the food ran out and it would be about what it is it like for people when their last chance is gone. Can that help portray the difficult period we’re going to with food insecurity in this country? But we found that what happens is people kind of drop their head, because they’re so depressed. With that detainee piece I learned that the audio is absolutely crucial to trick the mind that you’re there. I had an intern working with me and she came home one day in a state because she’d witnessed the event we ended up using. I popped on the headphones and heard what she’d recorded. It was this particular day where the line is too big the food bank is overwhelmed and some guy goes into a diabetic coma while waiting for food. And then the chaos breaks out.
How long did it take and what sort of technical skills did you have to develop in order to make it happen?
It took me about a year and half–the idea came two years ago and it took about a year and a half to get it built. It was a slow-going process where I’m funding it myself–the whole thing was built for less than $700 of my own money–and I’m begging for favors. I decided to start learning Unity myself. I actually joined this Unity working that was going on last summer and I came regularly and built the first prototype. I got the humans–I paid for some of the humans and I got the others donated from a gaming company called AXYZ Design out of Argentina. You can go on a place called Turbo Squid where people put their humans–I mean, these are virtual assets that you buy and build with. It’s a really interesting concept and it’s a very fast growing market, virtual goods.
Tell me about the reactions people have been having to this film at Sundance?
It’s been crazy. I think this is the most successful thing I’ve ever built. I’ve had things in the top 10 most emailed list for the New York Times. I’ve had fantastic reviews from critics on my films. But the hundreds of people that came into that room were such a diverse lot. And there was almost nobody who didn’t take the head-mounted display off and just kind of go, ‘Wow, where am I?’ It’s shocking to me the number of people who were so upset that they couldn’t help this guy. I had one woman crying. I had one woman kneeling down and trying to talk to him telling him it would be okay. I had grown men trying to hold his hand. This is New Frontier and you didn’t have to have tickets to get into it so I had one kind of hard-working blue collar guy come in and nearly start to cry about his own condition. I had young kids 9- and 10-year-olds looking at the adults to do something. We put at least 500 people through those goggles and afterwards all of us looked at each other and just couldn’t believe it. It was like, what the fuck did we just build?
Why do you think this is so effective?
The number one reasons why it works is that the audio is so compelling. We spent hours recording audio and did good boots-on-the-ground journalism first. But I do think that people feel a sense of presence that they’re there. And that trick on the mind comes from the goggles covering your eyes with images that refresh according to when you move your head and therefore feels really natural. So the second reason it works is I’m building with this incredibly powerful technology.
What are the applications for this technology?
There’s a whole gamut of educational applications. You have museum installations. We’ve been working in our lab on an iPhone viewer, which is actually surprisingly effective. Then you start to think about distribution of news in a larger way. I argue, people don’t like to hear it, but I argue that content is so copy-able but experience remains unique. Starting to think about experience as part of your overall business plan is a good idea. The idea of commodifying experience is not new but this is like commodifying emotion, which is crazy.
Unlike when I show my films and people emote together or come up and talk to you after, for a large portion of these last 10 days when people come out of the head mount I was looking at their faces. And John Brennan, who is an incredibly insightful guy–he was holding the cable walking behind people so they don’t crash into walls–and he said, you know I felt like I was their spirit guide, I have to bring them back to earth. It’s extremely powerful, which is why I’ve been trying to make this argument about best practices.
What are the dangers of this technology as you see it?
Propaganda, right? There was a piece that just came out in the New York Times about New York City police officers watching this anti-Islam film as part of their training, and it’s a really hardcore film. There are obvious dangers with film and this gives it a whole other level. My purpose was to show that this technology can be used for journalism and there’s no doubt that it can.
A big criticism of the current state of journalism is that it’s been sensationalized. Issues like hunger, stories that aren’t sexy or salacious don’t get covered because it’s difficult to communicate them in an engaging way. This technology has the potential to change that, it seems.
That’s exactly right. That was my goal. Take a story about people who are hungry, who I felt were invisible, and make them visible in a way that was so compelling that people would really get what the fuck was going on out there. So that works. And it works in a way that has blown my mind.