“I love chaos; I like things to be really fucking out of control,” says designer and technologist Joshua Davis.
Take the 240-foot-wide, interactive projection mapping project he created for Pepsi and Superbowl 50–a pulsating crash of colors and glitchy patterns that careened across the facade of a building. Or the swirling prismatic sphere he created to serve as the avatar of IBM’s Watson computer for an episode of Jeopardy.
Davis, who is the media arts director at the New York creative strategy and design studio Sub Rosa, has lately been collaborating with Kiran Gandhi–an activist and drummer who’s toured with M.I.A. Together, they created a live performance for the festival OFFF Barcelona that involves programming motion graphics to respond in real time to Gandhi’s songs.
Together, the two artists are essentially painting with sound.
Though chaos is an obsession of Davis, Gandhi didn’t want that to be the result of their work together. “We want people to feel sonic joy,” she says of how the visuals–a hypnotic series of geometric patterns that looks like fractals passed through a kaleidoscope–complement her music.
Though Gandhi began playing acoustic drums, she often performs electronic sets with her band Madame Gandhi, and was interested in exploring ways to better connect with her audience.
Davis and Gandhi’s collaboration began in a seemingly fated way. Davis has created installations for OFFF every year for the past 15 years, and was racking his brain for what to do next. He was about to invite a musician friend to do an hour-long set that he’d design visuals for, but then he received a text out of the blue from Gandhi, who came across his work online.
“I was deep in an Instagram rabbit hole and thought ‘this person is a genius,'” Gandhi says. “I went on his website, which listed his phone number, and I said, ‘I’m just going to text him’ and wrote, ‘Josh, I’m Kiran. I play the drums for M.I.A. and I love what you’re working on. If I could collaborate with you or contribute my music to what you’re doing, it would be an honor.'”
“I knew her work and her vibe and this was the universe stepping in and saying, ‘Oh no, we have different plans,'” Davis says.
To build the graphics for the performance, Davis asked Gandhi to create a 40-minute set list. He then input the audio into a time coder, listened to track, and noted the precise moments–down to the millisecond–where there were sonic shifts, which could mean a change in mood or tempo or the types of instruments playing. Davis counted about 29 moments in all, which were based on his subjective responses to what Gandhi’s music. For example, a certain moment might evoke colors or motions. “I’m putting down in my notes ‘403,000 milliseconds–definitely purple,” Davis says.
Though Davis started building the visuals on his initial response to the music, he then added layers of symbolism to each graphic moment, tying them into the song’s content. Rather than just being a cacophonous, hypnotic pattern that looks cool, they add another layer to Gandhi’s message.
The lead scene is a prime example. Since free bleeding (a term that refers to not using any products to absorb menstrual blood) while running the London Marathon last year, Gandhi–a longtime feminist–has become a vocal supporter of menstrual equity and gender equality. As a result, she’s also become a polarizing figure in media. Gandhi decided to open her set with statements relating to gender issues. For example, Rosie O’Donnell called her “a radical feminist woman” and Gandhi layered that in her song. Donald Trump used menstruation to insult journalist Megyn Kelly saying, “You can see there was blood coming out of her eyes, blood coming out of her…wherever,” so Gandhi incorporated a clip into the opening of her set.
Riffing on the theme, Davis decided to base the pattern for this segment on triangles, a symbol for women, as well as use the (obvious) color red. With that as a baseline, a program he wrote using Java then modulates the motif based on the audio composition. Davis grouped the audible sound spectrum into 32 distinct numerical ranges. For example, the lowest is all bass tones, a snare drum is mostly 8 and 9, and cymbals are mostly 14 and 18. The computer algorithm then looks at the intensity of sound coming from each of those ranges and alters the graphics accordingly. Kiran’s drum heads are outfitted with midi controllers that also feed signals into the algorithm when she hits them in real time.
“All the graphics are listening to this audio data and are animating and dancing based on what’s happening sonically,” Davis says. “It’s software that’s making graphics based on what’s happening at that particular moment in time.”
Elsewhere in the composition, Davis created graphics that used traditional Indian colors and symbols, like mandalas, nodding to Gandhi’s heritage. One track repeats the lyrics “I can see” so he created a graphic that looks like a dilating eye. Davis also wanted to embody who she was as a person, visually, at a certain point in the sequence. “I spent so much time researching Kiran that I tried to think, ‘What image could I create that could encapsulate her aura?'” he says. The resulting abstraction features splashes of neon yellow and magenta in a 3-D amalgam of shapes.
Since the sequences are pattern-based, Davis was concerned that they might feel too repetitive, so he created two different algorithms for each of the 29 visual moments accompanying the sonic shifts. This gives him the opportunity to manually toggle between the two graphics–which are still both responding to the music via the software–to add more of a personal element to the visuals.
“One of the biggest issues with technology is when it removes the soul, the human element of it,” Gandhi says. “I want to make something good using technology but I want the person to feel the sorrow or joy that I felt when I made the song. I listen to a lot of electronic music now and it’s for the bars, it’s for the clubs–there’s no emotion in it. I feel it’s a very female way of thinking: I want you to feel my soul infused in every little Ableton note that I hit as opposed to ‘let’s get fucked up, let’s get stimulated.'”
Correction: An earlier version of the post incorrectly stated that Trump’s quote was about Gandhi. It was actually about journalist Megyn Kelly.