Two years ago IBM started a project with its Watson system to explore just how creative a machine can be. Since then the team has been applying computational creativity to various domains. They are starting to look at how it can be used to develop new scents for the fragrance industry, create personalized travel itineraries, and improve sports teams based on particular strengths and skills.
Earlier this week at SXSW, they also broadened an experiment they began last year by publicly demonstrating a cognitive cooking system—at a food truck parked in downtown Austin. While the Watson team has previously run small scale demos for investor meetings, this was the first time they took cognitive cooking to the streets, sharing what’s under the hood through an app (for research purposes only at this point) that allows the public to talk with the computer, program it to create recipes, sample those foods, share the recipes they created, and vote for favorite dishes on Twitter.
It is a fun experiment. But it's not just for kicks. Such breakthroughs lead to bigger questions: What are the limits of computational creativity? How creative can computers be? And how creative do we want them to be?
Recently, I was invited to preview Watson's recipe-generating system and speak with IBM research veteran and trained chef Florian Pinel, a conversation that led to the larger discussion of computer-human interaction.
“We started thinking about the area where creativity could apply to computers,” says Pinel. "We thought that scientific creativity, based on data, was more reachable for computers. So we gathered a knowledge base around food."
Pinel's team parsed public repositories like Wikipedia and wikia.com, as well as the proprietary, never-before-shared database of 20,000 recipes from New York's Institute of Culinary Education. "We added data about the food at a molecular level, information about the flavor compounds, all these ingredients, information about people’s likes and dislikes. And using all those data, we managed to first create a system that can generate trillions or quadrillions of combinations of ingredients and also predict if a) they will be novel and b) if they will be pleasant."
At a private dinner on the eve of SXSW, I was invited to eat a meal created by the system and prepared by (human) chefs, adding their skill around cooking times and techniques. The Czech Pork Belly Moussaka with pea, celery root, Swiss cheese, and dill (among other things) was a delicious starter. I went back for thirds of the Kenyan Brussels Sprouts; the brussels, almond, sweet potato really did work together well with unexpected spices like cardamom, ginger, and parsley. The consensus around the table was that the Ecuadorian Strawberry Dessert with cumin, yogurt, and avocado oil was one of the best desserts we'd ever had.
According to Pinel, IBM developed a number of hypotheses like "the more chemical compounds the ingredients share, the more likely they are to taste good together in Western Cuisine—whereas in Eastern cuisine it tends to be the contrary.” They used that data to develop a system that can be creative with food. Essentially, they are building imaginary cuisines, from scratch.
"We have categorized everything in our food database," says Pinel. "We have categories of dishes. We have categories of ingredients. We have categories of cuisines. We know the relationships between an ingredient and a cuisine, between an ingredient and a dish. That's really the basis of these machine learning algorithms. Those algorithms pair between 1 billion and 1 sixtillion possible ingredient combinations, trying out new combinations of ingredient types, spices, and ethnic-slants to create foods that are entirely new."
These foods include things like Austrian chocolate burritos, Kentucky bacon gumbo, or— a dish served at the SXSW food truck— Vietnamese apple kebab.
Pinel and the Watson team are quick to point out that it isn’t just about the creativity of the machine. Human partnership is required to program on the front end, evaluate options, and then interpret the final recipes created. “If it weren’t for the creativity of the humans involved, you would all be staring down at a plate of raw ingredients,” says Steve Abrams, a member of the Watson team, regarding the food truck experiment. The magic comes in the interface of man and machine.
Chris Chafe, director of Stanford University Center for Computer Research in Music and Acoustics (CCRMA, pronounced karma) agrees. He and his music faculty of computer scientists, electrical engineers, and composers are running hundreds of similar experiments—using algorithms and human-machine partnership to create new types of sounds and tools for creating them.
"Every composer who uses computers today participates with the computer in what we call algorithmic composition," Chafe says. "The general idea is that computer programs can be written that will do musical things: you write a musical line and then you make it repeat itself. The more interesting question is how do you get a musical line. Well, a composer can write it, as Bach did. Or a computer can do that."
Chafe says that the Watson experiment is highly analogous to what he is doing. And that the moments that could best be perceived as creative are common in his work—if difficult to predict. “When something is right in a kitchen or a meal or a smell, think of how that cuts through and takes over your whole consciousness and becomes memorable. Some new recipe could trigger that kind of moment," Chafe says. "I would say that I’ve had a great number of those moments working in computer music, and that’s really the tantalizing aspect. I want more of those moments. When I have a musical moment that comes from a coldly calculated ‘yeah it really worked’ kind of result and it gets to the point where it takes over your soul and becomes memorable—that’s the addiction for me."
CCRMA researchers and composers use computer-based technology both as an artistic medium and as a tool for exploring, among other things, creativity. They're playing in all sorts of interesting spaces—working with brainwaves and music, mobile devices to develop programs like Ocarina and the spiral piano on the iPad, and running hundred of projects around computer intelligence in music. And they're not the only ones. Projects like Infinite Jukebox, an app created as part of an MIT Hackathon, let anyone deconstruct and infinitely loop a song creating new possibilities in musical structure.
It is computer-generated creativity's ability to make things that have never before existed, Chafe says, that is one of the ultimate goals. "Why this is so fascinating for me? I do a lot of work where I go into data that is not musical data, and I use computers to translate that data into music. The translations that I do have to do with styles I’m interested in or styles I’d like to hear that have never existed. If I do something from synthetic biology molecules and I want to hear it in a musical fashion, I’m basically scripting these translations to give you both a musical resolve an intuitive understanding of the data. It is very different than looking using your eyes to look at graphs."
The challenging part, both Pinel and Chafe point out, is that creativity is based largely on subtlety. Chafe describes it this way:
"Sometimes the most effective, transcendent moments in music are the slightest thing. Like the way a violinist angles the bow during a certain note. Or a singer moves their jaw in a certain way to express something extraordinarily deep that is felt by a good number of people—people who hear whatever that was. Try and turn that into numbers and something you can reliably get at through a simulation; that’s tough. When you do simulations like that, you are basically responsible for recreating music from the ground up. Everything that goes into it, from mechanical systems of an acoustical instrument, to the halls that you listen in, to the sense of deep time that’s embedded in the culture of the music. There’s a whole long list of the strata of things that came from that little sound that came from twisting the bow in a certain way."
Capturing this nuance is not beyond the realm of possibility. In 1948 mathematician Claude Shannon said that “any sound that can be perceived can be reproduced.” His belief in the possibility of using zeros and ones to describe sound waves led to a sampling theorem that was a big step in computers and music—and laid the groundwork for current work being done by both CCRMA and Watson.
Technology has nearly caught up with Shannon's imagination.