Beyond The Cut: Exploring The Future of Movies With Neuroscience, Virtual Reality, And Architecture

Neuroscientist Sergei Gepshtein is researching alternatives to edit-based moviemaking with an eye to taking cinematic storytelling to a new, non-linear level.

Beyond The Cut: Exploring The Future of Movies With Neuroscience, Virtual Reality, And Architecture
[Image: Flickr user Salford University]

In 1903, a 10-minute silent film by director Edwin Porter revolutionized filmmaking. The Great Train Robbery was the first film to employ cross cutting to create narrative. In the movie, a train is shot moving left to right in one frame. In a second shot, the bandits are shown riding their horses from right to left. When these shots are placed one after the other, the bandits appear to be riding toward the train. “These cuts create narrative,” explains Sergei Gepshtein, a neuroscientist at the Salk Institute for Biological Studies, who has become interested in how audiences experience cinema. “When ordered in a specific way, you get a sense of continuity.”


Despite most filmmakers’ reliance on cuts, Gepshtein believes there are other ways to create a contiguous cinema-watching experience. He’s hoping that neuroscience can help directors develop new narrative techniques that are actually better suited to how the brain processes information.

Unlike the movies, our brains don’t order the world with cuts. When we walk down the street, we are selectively sampling information–sights, sounds, and smells. “And from what we sample, we build sensory experiences of the world,” says Gepshein. “Our experience is not linear. We can hold multiple thoughts in our minds at the same time.”

Image: Flickr user Bahman Farzad

One of his main investigations into alternate storytelling involves putting viewers inside of a movie, almost like virtual reality. To explore this option, he has turned to architecture. Architects have a similar challenge as directors. The latter want audiences to feel like they’re inside a narrative experience. The former use drawings and graphics to show clients what a physical space might feel like. Both cases fall short of “reality.” In the movies, there’s always a remove between the audience and what’s happening on the screen. As for architecture, “it’s difficult to get a good sense of a 3-D structure on a 2-D screen,” says Gepshtein. But what if architects could actually enter the space and experience the physical parameters of what they’re designing before they hire builders? Then they could make changes based on what it actually feels like to be inside the space.

To do this, Gepshtein is working with the Academy of Neuroscience for Architecture, the School of Cinematic Arts at USC, and the Department of Architecture at UCLA. In a series of experiments, subjects wear head-mounted displays or they sit in a room with large screens moved by robotic arms. In these situations, subjects’ perceptual experiences are recorded and catalogued using tools of visual psychophysics, a branch of neuroscience that deals with physical stimulus and sensation.

Oddly enough, Gepshtein and his partners have found sea-sickness and nausea to be a central problem. People in VR spaces are more likely to feel disoriented or confused, especially when moving from one “room” to the next. For some reason, our brains can process a bombardment of sensory experiences in the real world much better than in a VR world. “We’re trying to figure out how people can move from one [virtual reality] rendering into another in a way that is not disruptive or nauseating,” says Gepshtein. “This is where the scientific knowledge is necessary, because it tells us how to achieve this continuity [of experience].”

Image: Flickr user Jeff

“When you’re a director, you want to create different experiences that are unfolding in parallel,” Gepshtein explains. Virtual reality techniques could help accomplish this by letting individuals explore whatever aspect of a scene (a doorway, a character, a painting on the wall) their brains seized upon. “In fact, you may let the viewer’s mind decide which thread of continuity to follow, so one can enjoy the same movie differently on different occasions.” Like a kind of neurobiological Choose Your Own Adventure.

Image: Flickr user Sndrv

Gepshtein is also interested in more subtle ways to switch up cinematic storytelling–approaches that are less Matrix-like in scope. He’s interested in how directors might communicate story through mood. For example, what if scenes didn’t “cut” abruptly from one to the next but faded slowly, visually overlaying objects and colors? But how this would work for mainstream cinema (as opposed to rarefied art-house fare) isn’t entirely clear. And he understands that traditional cinema is both powerful and engaging. “We don’t want to remove cuts entirely,” Gepshtein says. “Instead, the cut should be viewed as just one of many methods for building perceptual continuity between shots.”

The end goal, of course, is pure escapism. A way to float free of ourselves for a few blissful hours. Even if it’s our minds that are helping us to fly.

About the author

Jennifer Miller is the author of The Year of the Gadfly (Harcourt, 2012) and Inheriting The Holy Land (Ballantine, 2005). She's a regular contributor to Co.Create.