Roozbeh Jafari’s contraption straps onto the arm of anyone who uses American Sign Language and translates the arcs and flicks of their hand and wrist into written English. The words are then sent to a smartphone for any non-ASL user to read.
The device uses Bluetooth to send data from two kinds of sensors to a notebook computer, where the signs are decoded. But it’s the sensors that make this smarter than other options, especially the unwieldy video-camera-based systems in common use, which are hampered by low light and involve pointing cameras at people.
Motion sensors, like those found in your smartphone, are combined with electromyographic sensors that measure electrical activity in the muscles. This gives a large view of the movements of both the hand and the arm, combined with a detailed view of what the fingers and wrist are doing. This combination data allows the system to interpret the small differences between similar gestures.
Jafari, an engineering professor at Texas A&M University, wants to make the device small enough to wear, so that the decoded words can be sent straight to the user’s own smartphone. “Wearables provide a very interesting opportunity in the sense of their tight coupling with the human body,” Jafari told the Texas A&M University newsroom. “Because they are attached to our body, they know quite a bit about us throughout the day, and they can provide us with valuable feedback at the right times. With this in mind, we wanted to develop a technology in the form factor of a watch.”
A watch seems the perfect place for this device. The Apple Watch, for example, already contains much of the hardware necessary for Jafari’s device–accelerometers and gyroscopes for motion sensing, plus a Bluetooth link to a powerful computer (the iPhone) for interpreting the data into words. It only lacks the electromyographic sensors.
Right now the prototype recognizes 40 American Sign Language words and is 96% accurate, so the bulk of the work would seem to be in adding more words. Wikipedia says that there are around 300 different sign languages in use today, so it might take a little while before it’s ready for everyday international use.