Get Some Therapy From An App That Reads Your Feelings Through Your Voice

By monitoring changes in your conversations, your phone will soon know if you’re happy, stressed, or sad, and can nudge you toward feeling better.

Get Some Therapy From An App That Reads Your Feelings Through Your Voice

At moments of high stress, the less self-controlled among us might come close to throwing their cell phone against the wall. But soon, your phone might help you relieve some of that stress (or at least better mange it). An app on your smartphone could soon tell you when you’re stressed, or worried, or suffering from a more serious mental health problem.


Devices like Nike+ or Jawbone’s UP bracelet keep a daily, constant record of physical fitness, but the system that Tanzeem Choudhury, a computational wizard at Cornell University, designed reaches out a step beyond physical fitness to the comparatively untracked terrain of mental health.

Choudhury’s sensing system uses machinery in a smartphone–mainly the accelerometer and the microphone–to monitor people’s movements and speech, respectively. Collaborating with Andrew Campbell at Dartmouth, she’s created algorithms that can extract reliable behavioral information from the phone sensor data based on vocal cues.

“We never look at words,” Choudhury is quick to clarify, “Because people are not comfortable [with us] recording them 24/7.” Studies on social behavior and the mind have suggested that our mental state is reflected in the way we speak to our friends: how quickly we talk, how much, and how we take turns. The variability in these stylistic factors like intonation can be correlated with mental health, Choudhury explains.

To make sure they’re looking for the right signals, Choudhury’s targets are guided by a medical professionals. Her collaborators–a team lead by Ethan Burke at Dartmouth Medical School–evaluate the data offered up by the system. They also advise the tech team on what specific signals will be valuable as mental health indicators.

The app has been tested in two very different environments: in a retirement community and among a group of medical interns and residents. The pilots have been a success, Choudhury says, and compare well with the gold standards used to assess mental goodness and depression.

“We’re latching onto the right behavioral traits,” Choudhury says, “But we need to diversify to different scenarios.” Her next steps will be to develop the system so that it can be used in any environment, and used by people with different personalities.


One of the challenges to such an app will be in designing how users are informed by the results that the app finds. In the current interface, an early pilot, an aquarium is set as the phone’s screensaver. Many fish indicates an active, relaxed life. A darkened aquarium means the fish (and user) might need a good night’s sleep.

The system extends beyond people idly curious about their daily mental health. It could help alert parents to early signs of depression in schoolkids, without placing them in the artificial confines of a doctor’s office. That’s a demographic that usually isn’t tracked for mental health, but now it could be, fairly painlessly. For people already diagnosed with mental conditions, their daily mental state would be monitored continuously, without the need for periodic surveys and questionnaires, which are less reliable because they are not real-time, and even less so if the mental state of the patient makes them less fit to give reliable answers.

On an even grander scale, a monitoring system like this one could provide valuable data for population studies of communities–informing researchers about factors that affected the mental health of whole groups of people. Of course, privacy and anonymity are key to such a study, and groups are already working on ways to anonymize such a pool of data, Choudhury says.

For people simply interested in their mental health on a daily basis, it could become the latest in a series of wellness tracking apps and devices, but that’s still far away–Choudhury and her team still need to tweak the system so that it can take on extroverts and introverts, and respond appropriately to different social and environmental situations. “But we have all of the building blocks,” she says.

The group hopes to roll out a demonstration of the application for specific demographics, like the elderly, or schoolkids, soon. A more universal monitor could take three to five years to show up. In the meantime, try not to take your anger out on your phone. Pretty soon, that phone might be your therapist.