MIT Scientist Captures 90,000 Hours of Video of His Son's First Words, Graphs It

Cognitive scientist Deb Roy blew the curve for Flip cam-packing proud pops. Since he and his wife brought their son home from the hospital, Roy has captured his every movement and word with a series of fisheye-lens cameras installed in every room. The purpose was to understand how we learn language.

In a talk soon to grab several million views on TED.com, cognitive scientist Deb Roy Wednesday shared a remarkable experiment that hearkens back to an earlier era of science using brand-new technology. From the day he and his wife brought their son home five years ago, the family's every movement and word was captured and tracked with a series of fisheye lenses in every room in their house. The purpose was to understand how we learn language, in context, through the words we hear. 

A combination of new software and human transcription called Blitzscribe allowed them to parse 200 terabytes of data to capture the emergence and refinement of specific words in Roy’s son’s vocabulary. (Luckily, the boy was an early talker.) In one 40-second clip, you can hear how “gaga” turned into “water” over the course of six months. In a video clip, below, you can hear and watch the evolution of "ball." 

Unreal 3-D visualizations allowed his team to zoom through the house like a dollhouse and map the utterance of each word in its context.

In a landscape-like image with peaks and valleys, you can see that the word “water” was uttered most often in the kitchen, while “bye” took place at the door.

The video was processed to show "time worms," below, charting the family's movement from room to room.

Most moving of all was the precise mapping of tight feedback loops between the child and his caregivers—father, mother, nanny. For example, Roy was able to track the length of every sentence spoken to the child in which a particular word--like “water”--was included. Right around the time the child started to say the word, what Roy calls the “word birth,” something remarkable happened.

“Caregiver speech dipped to a minimum and slowly ascended back out in complexity.” In other words, when mom and dad and nanny first hear a child speaking a word, they unconsciously stress it by repeating it back to him all by itself or in very short sentences. Then as he gets the word, the sentences lengthen again. The infant shapes the caregivers’ behavior, the better to learn.

Roy is now taking the amazing research capability and team he’s developed and applying it to commerce. He’s on leave from MIT and has founded a VC-backed company called Bluefin Labs that applies these same high-powered analytics to relate, not the speech of a child to that of a father, but events broadcast on TV to conversations taking place in social media, the better to chart “engagement” with the State of the Union Address or Jersey Shore or a car commercial.

"After 15+ years of academia, I want to take some of my ideas out of the lab and into the world," Roy told Fast Company. "I also feel that the changes in the world of mass and social media provide a perfect environment for these ideas to have real impact (not just commercial, but also social), a opportunity that I feel compelled to seize."

The methods he's developed are still being applied to babies; some of his senior graduate students at MIT  continue to analyze the data, and he's designed PlayLamp, a less intrusive recording device currently being used in pilot studies of children at-risk of autism.

see also: BBC News, MIT Media Lab

Read more TED coverage

Add New Comment

21 Comments

  • Testking VCP-410

    Roy is now attractive the awful search potentiality and unit he's matured and applying it to commercialism. 

  • Testking VCP-410

    In other words, when mom and dad and nurse premier pore a somebody talking a phrase, they unconsciously stress it by continuation it hind to him all by itself or in very small sentences.

  • jerry biuse

    rofessional style unfit to a peak and slowly ascended rearmost out in complexness." In another words, when mom and dad and nursemaid front hear a male tongued a statement, they unconsciously evince it by continuance it corroborate to him all by itself or in really parcel sentences funny christmas greetings sayings . Then as he gets the statement, the sentences lengthen again. The infant shapes the caregivers' activity, the gambler to hear
     

  • J. Crawford

    Interesting article.  I wonder how old this child is (seems to be younger then 10 years old), because there's only 8766 hours in year?  How does he get 90K hrs?  Are there multiple cameras running simultaneously?

  • Steven

    Lol at Comments below regarding the milano group. This was all part of Anthony's ploy. he directed you all to his website to "Draw Traffic" Even people who instantly hate you and click on your site draw traffic. So for the people below who are "so intelligent" you fell right into the buffoon's trap. Hopefully you dont consider yourselves intelligent. As far as the article, incredible.

    http://www.seocentralflorida.c...

  • Playground Rubber Surfacing

    Here a guy who was stuck in academia for 15 years and wants to go out in the real world.

    Good Luck Chuck..

    Stick with what you know and leave the rest for us.

    Be Seen, Be Heard, Be Connected

  • Steven

    Lol at Comments below regarding the milano group. This was all part of Anthony's ploy. he directed you all to his website to "Draw Traffic" Even people who instantly hate you and click on your site draw traffic. So for the people below who are "so intelligent" you fell right into the buffoon's trap. Hopefully you dont consider yourselves intelligent. As far as the article, incredible.

    http://www.seocentralflorida.c...

  • Donald Langdsorf

    "Stick to what you know and leave the rest for us."

    How ignorant! I read this article and thought, "Wow - this stuff is genius." Mr. Roy has an incredible amount of dedication, vision and intelligence. We should be welcoming him into the "real world" - it is minds like his that make a difference and shape our technologies. Academia is not a vacuum, void of any interaction with the world outside. It is the place where ideas are spawned and cultivated into reality. Without it we would still be in the dark ages.

    In fact, you may end up using this man's technology in your own SEO business Mr. Milano, and it seems as though you can use it. I'm no SEO expert, but if I were touting myself as such, as you are, I'd be disappointed and embarrassed to see how low my page rankings are on Google when direct search terms are used, including your name, SEO, and location as search terms.

    My point is not to attack you, but hopefully to help you open your mind a bit and realize that gentlemen like Mr. Roy should be welcomed into society at large, not shunned because they chose to initially pursue a noble career in academia.

  • andrew keith

    Thats a pretty mean thing to say even though you have never met Roy before. Roy has done a remarkable job and it will surely bring him success with any endeavor.

  • Karthik Narayanaswami

    Clearly, a clichéd SEO services company is way more challenging and "real" than a novel and innovative creation of a tenured MIT professor.

  • Chris Grant

    Good to know that you didn't use and advanced education to get where you're at.

    Stick with what you know and leave the rest for us. Well said. I'd recommend you take your own advice.