It feels like a cross between a video game and an acid trip. But as you walk through the neon, geometrically tripped-out cityscape, the music you hear is no hallucination. Every measure of ambient electronic tones and textures is composed in real time by your own brain waves.
It may sound surreal, but the above scenario was reality for several hundred people during last week's Moogfest music and technology festival in Asheville, North Carolina. It was there that emerging media studio Odd Division, working with production company Tool, launched Conductar, an experimental music and virtual reality app for iOS.
During the five-day event, participants were invited to don brain wave sensors connected to iPhones and walk around the actual city of Asheville while being electronically immersed in an alternate, far more surreal Asheville. These headsets measured each user's state of mind—How alert or meditative are you?—and played back a sample of minimalist, mood-appropriate music.
"What we set out to do is create a musical, generative song experience," says lead developer Jeff Crouse. "It's a 16-bar structure. Every four bars we play additional sounds. Each time a new sound starts, we ask the system what clip best suits this state of mind. It's never-ending."
While Conductar is best experienced with a brain wave-sensing headset, the cyborg-errific fashion accessory is by no means a requirement. As Moogfest attendees traversed Asheville with the devices, they left behind data points mapping their own mental state in specific locations. That way anybody else who downloaded the app could meander through the virtual city and hear collectively composed, location-aware music.
While the team behind Conductar has plans for expansion, right now the live, immersive walkthrough functionality only works in Asheville. But that doesn't mean it's not worth taking the app for a spin. Now that Moogfest is over, users of will be dropped in downtown Asheville with an on-screen joystick that lets them navigate and hear the citywide composition locked in time by the brainwaves of festival attendees.
The project, which took about three months to complete, is comprised of a few components: an unmodified MindWave headset, custom-built iOS app, and music clips composed by Odd Division's Gary Gunn. Those clips were mapped to states of mind and sequenced using a back-end system architected by Crouse.
"I don't think the technology we used was as defining as our approach," says Aramique, the founder and creative director of Odd Division. "From the beginning we wanted to create a system that could travel and be set up anywhere. We also wanted an audio-visual world that was platform-agnostic."
The app itself is built using the Unity3D gaming engine and various plugins thereof. "We picked Unity3D so that we could easily adapt the world to Oculus Rift, projection mapping, desktop or Android," says Aramique. Indeed, the team's installation during Moogfest included a version of Conductar running on an Oculus Rift.
Crucially, Unity3D has an audio package called FMOD that allowed Crouse and the team to build a music sequencer within the app. To map Asheville (and presumably other cities in the future), they used data from OpenStreetMaps.
Arguably the simplest component, says Aramique, is the brainwave sensor, which the team used in a previous project. The MindWave headset was used as-is out of the box to generate the state-of-mind data points that were then mapped to music.
"None of us are neuroscientists, so we're not totally experts on this, but they have an SDK and we really like working with it," says Crouse. "It has 11 different frequencies that it detects through these contacts on your forehead and on your earlobe. We use those as the input for a kind of music sequencer that's built into our app. This device just provides the raw data."
At first glance, Conductar seems like little more than a jaw-dropping art-tech project designed to wow festival-goers. And that it was. Around 800 people borrowed headsets during Moogfest, wearing them for an average of 30 minutes at a time.
But while the app would seem to have limited practical value post-Moogfest, it's wild-eyed experimentation like this that tends to drop unmissable hints about the future.
In this case, we can start to see where music consumption and creation might be headed in the age of wearable computers. Today, music apps like Moodagent and Beats Music try to auto-curate playlists for us based on our mood as we describe it. Meanwhile, engineers at Pandora are tinkering with ways to alter your radio experience based on your location and activity, pulling whatever clues they can from the device being used.
In the future, one can imagine a more seamless experience as devices get smarter and more aware of the context in which we're using them. Instead of tapping a button to declare my state of mind, I could focus my attention elsewhere as a music app simply reads my brainwaves.
And I won't need to wear a goofy headset to do it. Presumably, there will be a well-designed, socially acceptable version of Google Glass on the market at some point. Such a wearable heads-up display would eliminate the need for both the smartphone and a separate brainwave sensor. Meanwhile, Apple is already reportedly working on earbuds lined with biometric sensors.
As for Conductar itself, the team does plan on rolling it out beyond Asheville, once a few technical hurdles are overcome. Most notably, the building-level mapping data is not easy to come by for all cities. OpenStreetMaps doesn't provide it, so for the Moogfest project, Odd Division and Tool's designers had to trace the footprints of the buildings by hand. It might be possible for them to get that type of data from Google, but if not they'll have to find another source. In the meantime, the rollout would necessarily be limited to larger cities for which that kind of data is publicly available.
"We wanted to build a story world that could spread easily across the world," says Aramique. "For our next iteration, we would like to develop the world further into a more immersive non-linear narrative that takes advantage of OpenStreetMaps to make a site-specific story-world. We have also thought about turning it into a world-building tool where people could populate it with their own narratives, design, and game mechanics."