Last week at the MIT Tech Review Conference, Thad Starner, professor and director at Georgia Tech’s School of Interactive Computing, showed off some special gloves that could change the way blind people “see” the world.
A few years ago, Starner’s team built gloves that could teach people to play piano melodies in less than an hour, by sending motor vibrations through their hands. His new team just finished a study that proved they could do the same thing with Braille.
But this time around, something even more important happened. Researchers were surprised to discover that people could not only type Braille through passive haptic learning, but that they could actually read the Braille phrases afterward. (Note that these aren’t people who’d been exposed to Braille in the past.) And it wasn’t just their fingers that learned the phrases, some kind of muscle memory–it was also their brains.
“What we already found is that you can learn single hand melodies just using vibration while you’re doing other things like reading email or watching a video or doing a math test,” Starner explains. “You just go do your daily business and at the end of this, you can play the melody for ‘Amazing Grace.'”
However, this time around the problem is much more complex–and the implications much bigger. “Music is a particular low-level region of the brain,” Starner says. “But if you’re doing things like typing, that’s much higher levels of the brain. So this time we tried Braille. It’s six fingers, six dots. It’s chorded between two hands. It has all sorts of implications. It means we could access different parts of the brain. Braille was an ambitious goal for us.”
The study, conducted by Starner and PhD student Caitlyn Seim, worked like this: Each participant in the study wore a pair of gloves with tiny vibrating motors stitched into the knuckles. The motors vibrated in the same order as a typing pattern of a pre-determined phrase in Braille. Audio cues let the users know the Braille letters produced by typing that sequence. Afterward, everyone tried to type the phrase one time, without the cues or vibrations, on a keyboard.
The sequences were then repeated during a distraction task. Participants played a game for 30 minutes and were told to ignore the gloves. Half of the participants felt repeated vibrations and heard the cues; the others only heard the audio cues. When the game was over, participants tried to type the phrase without wearing the gloves.
According to Starner, participants who felt the vibrations during the game were a third more accurate. “Some were even perfect,” he says.
What’s cooler–participants also picked up additional unexpected skill. “Remarkably, we found that people could transfer knowledge learned from typing Braille to reading Braille,” says Seim.
“This finding is significant because it suggests that we can teach both these critically important skills at once, passively. Now that we’ve done it, what we can show is by having you play four hours of Candy Crush Saga, you hear the words spelled out and as you tap with your fingers, in about four hours you know the alphabet. The funky thing is since it’s Braille is simple, you can also read it visually and tactiley.”
How well did people learn the language? After the typing test, passive learners were able to read and recognize 93.3% of a phrase’s letters. Because the study also didn’t include screens or visual feedback, participants never got to see what they had they typed. And they had no indication of their accuracy throughout the study. It was all done through passive haptic learning.
“People always ask how long does this last,” says Starner. “The answer is we don’t know yet. The Braille study is brand-new.” However, Starner does tell the story of working with a passive haptic rehab patient. The man didn’t have feeling in his hand. Months before, he had used the glove to learn to play piano. And then he had forgotten how. On his next visit to the lab, after 10 minutes of wearing the glove, he could play the whole song again. Muscle memory is powerful stuff.
Passive haptic learning opens up all sorts of new possibilities for blind people–especially since Braille is hard to learn for diabetics, wounded veterans, or older people with cognitive impairments. According to the most recent World Health Organization statistics, there are 39 million blind people in the world today, most in developing countries.
It also points to all sorts of new opportunities in haptic learning. There are currently very few other companies or researchers working in the space. Jun Rekimoto’s work at the University of Tokyo on the Possessed Hand is an interesting example. So is the powered exoskeleton at Atlanta’s Shepard Center for spinal column injury, a wearable robot that can not only help paraplegics stand up and walk but also send electrical stimulation pulses to muscles (assuming that there is some muscle control). It essentially retrains them to walk on their own.
“There’s some astonishing stuff in rehab,” Starner says. “But it’s not passive learning. It’s active. The only other passive project is by [M.M.] Dimitrijevic. It is Mesh Glove Electrical Simulation. This is for patients who have 50 degrees of flexibility in their elbows. You wear this up-to-the-elbow metal glove. It stimulates the muscles. You can’t feel it. The brain releases more and more neurons in response to the stimulation and flexibility increases to 100 degrees, which is really quite astonishing.”
While that’s a great example of passive haptic rehab, the technology could be applied in all sorts of ways to help healthy muscles learn new skills.
“Can you do it for dancing? Can you record the sequence of muscles required for throwing a baseball? Can you do something for sign language, which has very complex hand motions? You could certainly imagine it for any type of typing,” says Starner. “I think there are a lot of things where you might not be able to teach the whole system, but you can definitely speed up learning.”
“We spend a lot of time working on this crazy technology,” says Starner. “And sometimes it just pays off.”
While the Braille studies have not yet been broadly released, they will be presented in Seattle this September at the academic-focused International Symposium on Wearable Computers (ISWC).