This Robot Points Out Your Mistakes, Like The World’s Most Annoying Roommate

You left the fridge open, and your robot pal is going to keep reminding you until you close it.

This Robot Points Out Your Mistakes, Like The World’s Most Annoying Roommate
Photos: Flickr user Cha già José

Imagine a robot that follows you around, watching as you leave the milk out on the counter, or forget to turn off the TV, or leave the gas stove burning, then reminding you to fix it. If it sounds like the world’s most controlling roommate, then yes it is. But if you are absent-minded, and you live alone (apart from hassle-bot of course), then it could save your life.


The robot is called the Watch-Bot, and it consists of a Microsoft Kinect on a tripod, motors so it can move its head, a laptop for a brain, and a laser pointer. The Kinect is a gaming accessory that used 3-D cameras to watch you and track you, so you can control a game with your body movements. In the case of the Watch-Bot, the Kinect is its eyes into your filthy, chaotic world, and the laser pointer is used to point out your mistakes, like a terse, passive-aggressive parent.

Watch-Bot was presented at last week’s ICRA 2016 conference on robotics and automation, in Stockholm, and is detailed in a new paper from the teams at Cornell and Stanford.

The average human adult, say the researchers, forgets three important facts, chores, or events every day, and the Watch-Bot is there to make sure that stops. Remarkably, it can teach itself what you have forgotten, with no training input from anyone. The Watch-Bot can learn by watching videos, or by watching you, and in future it may even be able to head over to YouTube and train itself there, although that sounds like it could be robot suicide.

If our robot sees a person fetch a milk from the fridge, pour the milk, and leave without putting the milk back to the fridge, it would first detect the forgotten action and the related object (the milk), given the input RGB-D frames and human skeletons from the Kinect; then map the object from the Kinect’s view to the camera’s view; finally pan/tilt the camera until its mounted laser pointer pointing to the milk.

The Watch-Bot’s brain takes the 3-D scenes from the Kinect and uses them to work out what’s going on in the scene, and to infer the human’s intentions. For example, the task “make tea” requires the actions “fill kettle” and “boil kettle.”

To help it to infer intent, Watch-Bot is trained to recognize human gestures, and to isolate the objects the human interacts with in a scene. And because it can teach itself, the robot actually becomes more effective over time. Using this, the robot can learn, unsupervised, what is intended and what has been forgotten, just by watching for a while. It’s almost exactly the way you or I might do it, watching to see how things are done, before switching over to nagging people when they get it wrong.

While it would be tempting to deploy the Watch-Bot in the office kitchen to make sure that your coworkers don’t leave their unwashed coffee cups in the sink, it also has potential uses in the world of health care, especially helping, say, Alzheimer’s sufferers from forgetting dangerous tasks, such as leaving outside doors open or a pot on the stove.

About the author

Previously found writing at, Cult of Mac and Straight No filter.