There's a moment in every episode of Dora the Explorer and Blue's Clues when the character on screen asks the children watching at home to answer a question, followed by a silent pause. The child is supposed to respond, of course, but the adults know that the TV isn't really listening. But what if Dora was able to hear the child's answer, and respond?
That’s the idea behind The Winston Show, the first iPad app from a buzzy San Francisco startup called ToyTalk that is founded by technical goliaths from Pixar Animation Studios. ToyTalk’s breakthrough is that the app's wise-cracking host Winston, and his tangerine-shaped sidekick Ellington, ask questions of their pint-sized viewers, listen to the answer, respond intelligently, and get smarter as more children play along.
"We were possessed by the idea that creating entertainment through conversation is something possible," Oren Jacob, cofounder and CEO, tells Fast Company. The path to Winston started by watching his own kids Skype with their grandparents. "Martin (Reddy, ToyTalk's cofounder) and I decided that we could have a conversation between the audience and characters much like having a conversation with grandma and grandpa,” he says.
By 2012, Jacob—who spent 20 years at Pixar before cofounding ToyTalk—had gathered more than $16 million in funding, and assembled a crew of animators, writers, and tech experts.
The app, which is available for free in the App Store, transports its guests to a TV studio lot with five separate sound stages—six hours of activities in all. There’s the "Win With Winston" quiz show, which asks players about everything from nocturnal animals to world hunger. Another called "You vs." that pits the player against any number of guests including Marie Antoinette and a blue whale. And there's a clever spin on Masterpiece Theatre (and Franklin D. Roosevelt) called "Fireside Chat With Winston." The story begins when a child presses the pulsating microphone on screen and answers a question posed by the host.
The Fireside Chat strikes on dozens of topics, including ice cream, aliens, and teachers (“giants with chalk”). Winston might say, "Are you an astronaut?" As the child answers, voice recognition software uploads responses to the cloud and converts speech to text. (ToyTalk execs stress that the app never asks for a child's name or location, in keeping with the Children's Online Privacy Protection Act.) Next, "we pass it through various natural language processors, and then we use the artificial intelligence engine we built to figure out what to say back," says Martin Reddy, ToyTalk’s chief technology officer and the software lead on films including WALL-E and Finding Nemo.
Artificial intelligence in the company's proprietary PullString software supplies the brains behind the app. All of the speech, text, and animation for Winston are funnelled into PullString, which becomes a multimedia scriptwriting tool that the writers and animators use to create skits. For Fireside Chat, the cloud-based speech recognition API locates the ideal retort and then PullString advances it to the next logical branch of dialogue.
Winston: So tell me...are you a giant?
Winston: Oh, I knew you were. Are you an Earth-dwelling giant or do you live at the top of a beanstalk?
Winston: Oh good, I'm glad you live here. I'm your biggest fan. Heh, see? Cause you're a giant and I said big. Ahkay.
Conversation threads can go on for around five or six exchanges depending on what a kid says. If the player says he lives in a place the database doesn't recognize, Winston utters what the creators have dubbed a "fallback" response, such as "Beanstalks are quite a traditional home base for giants, is what I've been told."
Speech engineers interpret the data daily, and alert writers to fresh answers. In one Fireside Chat Winston asks, "What is your favorite ball?" The staff came up with all the ball types they could think of, but in testing the app several kids replied "gumball.” Since that was not in the lineup, the writers concocted a quip to respond to "gumball" and added it to the database.
So far, more than 3,000 lines have been recorded. The ToyTalk team expects to add fresh material to the app every week. And there’s also a full-time voice actor on staff to record the dialogue.
The process for creating The Winston Show is similar to the way Pixar movies are made, says chief creative officer Bobby Podesta (he should know, Podesta was the supervising animator for Toy Story 3). First, the story line is approved, followed by storyboard drawings. After that, every discipline gets unleashed simultaneously: Animation, writing, acting, compositing, user interface design, and sound. The staff concurrently labors on various show aspects and upcoming skits.
Winston is a neckless, noseless, banana-yellow Brit drawn only with a torso, arms, and a face with eyebrows, oval eyes, and a mouth. There’s also the trusty sidekick, Ellington. Like Charlie Brown's teacher known by her signature wah, wah, wah, Ellington doesn't speak in words but in high-pitched syllables like a kid squealing underwater. As 8-year-old beta tester Emily Jones puts it, "His job is to interrupt a lot."
The characters are simple on purpose. Rendering movements and streaming from the cloud has a huge impact on the app's speed and size. Winston is always behind a desk because it would take up too much animation time to draw him running. Live conversation and simple gestural animation takes up most of the power, explains Podesta. Anyone expecting Pixar movie-quality graphics will be disappointed.
Winston emotes in 10 different poses. Jon Collins, the app’s supervising animator and a former animation lead for Blizzard Entertainment, says "Creating animation in two-way conversation is very different." Winston does not use wild hand gestures as he listens to each reply since people do not normally respond that way. "It can just be a postural change, but we want to make sure the child understands what Winston is saying."
Creating multiple animations that keep up with the volume of written text was a challenge. PullString enables Collins and his team to tag sections of dialogue in order to create animations. Compared to a full-length feature, completing animations for this app is achieved in an entirely different timeframe. Says Collins, "We can do one line in one minute as opposed to six days."
And when a motion picture is finished, the staff can call it a wrap. Not so for The Winston Show. "There are new jokes to create and more characters to add, and Winston will grow with its audience as well," says Jacob. Along with recurring jokes and discussions of the badges that a player has unlocked, whole shows might change or even be replaced mid-season. The app is ever evolving—like a comedian who is doing stand-up comedy, notes Jacob. Or, as beta tester Emily says, "I think Winston should be a weather reporter next."
"This is all uncharted, groundbreaking, and new,” says Podesta. “We are creating what we feel is a great show for people."
Like any new talk show host, there are parts of Winston's act that could be improved. A weak Wi-Fi signal can sink a play session, for example. Winston smartly has a bounty of lines at the ready in case he can’t hear someone ("I'm sorry. I was thinking about llamas. What did you say again?"), but it still interrupts the conversation.
These issues will be addressed as the audience for Winston grows, and only then will the company figure out how to charge money for its app. ToyTalk may sell add-ons, such as backdrops or additional activities. Or even create a version for Android, though there are currently no plans to do so.
There is also talk of licensing the PullString technology so that an already famous character can start talking to its audience the way Winston does. With its software resident on Amazon's S3 platform, ToyTalk is gaining valuable expertise in scaling mobile memory and storage. And while the company is staying mum about which vendor is providing its speech recognition technology, the partnership is expected to yield valuable data about the quirks of childhood speech patterns.
In the meantime, the ToyTalk 20-person team has created a charming, witty, and unique genre of app. And it poses an important question: Can kids be as entertained by a good conversation as they are by shooting birds from a slingshot?
One early app tester (and an adviser to the company) who thinks ToyTalk got it right is Raph Koster, lead designer of Ultima Online and author of A Theory of Fun for Game Design. "The emphasis on the characters is right. The humor is right," says Koster. "Conversation is the magical piece of the equation, and it captures the imagination."
[Get The Winston Show in the App Store here.]