Can't Touch This

He knew some professors at NYU and, despite his aborted stay at Cornell, landed a research position at the Courant Institute, where he has been for the past four years. The scope of the projects he's involved in is a testament to the sheer wattage of his brain. Two are funded by DARPA, the Defense Advanced Research Projects Agency under the Department of Defense, including one involving visual odometry: Modeling his work on the brain of a honeybee, Han has been looking for ways to make a computer know where it has been and where it is going--part of an attempt to build a flying camera that would be able to find its way over long distances. Han has also made it to the second round of a DARPA project to create an autonomous robot vehicle that can traverse terrain by learning from its own experiences. The goal: to perfect an unmanned ground combat vehicle that could operate over rough trails, in jungles or desert sand, or weave through heavy traffic as if it had a skilled driver behind the wheel. One non-DARPA project involves reflectometry. Han came up with a way to scan materials so they are faithfully reproduced digitally. The process typically requires shining a light on a piece of fabric, a flag, say, from dozens of different angles, and scanning each one into a computer--a time-consuming proposition. But Han developed an elegant shortcut: He built a kaleidoscope with three mirrors that reflect one another. Once a swatch of cloth is inserted, the scope yields 22 reflections mimicking different angles of light. When data from each reflection are scanned, the result is a flag that can be formed into any shape--one that looks like it's waving in the breeze, with each ripple and each slight shift in light rendered to a photographic exactitude. The whole process takes a fraction of the time Hollywood's best computer animators would need.

Han brought a similarly pragmatic do-it-yourself attitude to his study of touch-screen technology. When he began looking into the idea, he discovered that a few researchers were working on interactive walls and tabletops, and there were a number of art pieces. But that was about it. The concept hadn't advanced much from where it was in the 1980s, when Bill Buxton, now a Microsoft researcher, was experimenting with touch-screen synthesizers. "Most of it was designed with toys in mind," Han says, "something you project on-screen like Whack-a-Mole with hand gestures. But they weren't asking themselves what purpose it served. I wanted to create something useful."

Inspiration came in the form of an ordinary glass of water. Han noticed when he looked down on the water that light reflected differently in areas where his hand contacted the glass. He remembered that in fiber optics, light bounces on the inside of the cable until it emerges from the other end miles away. If the surface was made of glass, and the light was interrupted by, say, a finger, the light wouldn't bounce anymore, it would diffuse--some of it would bleed into the finger, some would shoot straight down, which was happening with his water glass. Physicists call the principle "frustrated total internal reflection" (it sounds like something your therapist might say).

Han decided to put these errant light beams to work. It took him just a few hours to come up with a prototype. "You have to have skills to build," he says. "You can't be strictly theoretical. I felt fortunate. I walked into a lab with crude materials and walked out with a usable model."

He did it by retrofitting a piece of clear acrylic and attaching LEDs to the side, which provided the light source. To the back, he mounted an infrared camera. When Han placed his fingers on the makeshift screen, some light ricocheted straight down, just as he thought it would, and the camera captured the light image pixel for pixel. The harder he pressed, the more information the camera captured. Han theorized he could design software that would measure the shape and size of each contact and assign a series of coordinates that defined it. In essence, each point of contact became a distinct region on a graph. "It's like a thumbprint scanner, blown up in scale and encapsulating all 10 or more fingers. It converts touch to light." It could also scale images appropriately, so if he pulled a photo apart with two fingers, the image would grow.

"People want this technology, and they want it bad," says Douglas Edric Stanley, inventor of his own touch-screen "hypertable" and a professor of digital arts at the Aix-en-Provence School of Art in France. "One thing that excited me about Jeff Han's system is that because of the infrared light passing horizontally through the image surface itself, it can track not only the position of your hand but also the contact pressure and potentially even the approach of your hand to the screen. These are amazing little details, and pretty much give you everything you would need to move touchable imagery away from a purely point-and-click logic."