The Hardest Problem In Baseball (And The Robots Solving It)

Baseball’s new player tracking system uses binocular vision to accurately record every movement players make.

The Hardest Problem In Baseball (And The Robots Solving It)
[Image: Flickr user ArtBrom]

Baseball is a game of numbers and inches, but to date, analysts of the game have had to rely on paper, pencil, scorecards, and calculators. But not anymore.


At the 2014 SLOAN Conference, Major League Baseball’s Advanced Media department (MLBAM) announced in-stadium technology that can track every detail of the game entirely with cameras. That means every data point in the game’s geography can now be recorded–crucially, without the encumbrance of sensors on the players, equipment, or uniforms.

But a sensor-free data tracking system would require MLBAM to solve a problem called “seclusion”–essentially teaching computerized eyes to see players the same way humans do. Here’s how they did it, and how it’s going to completely upend the game experience for fans, coaches, and players.

New, Clear Escalation

Without exception, teams and players will utilize whatever information they can to gain a competitive advantage. But the numbers available haven’t always told the whole story.

“When I was a video coordinator and we were doing stats analysis, whatever we had, we were going to use,” says MLBAM CTO Joe Inzerillo, who likens baseball’s statistical obsession to an arms race. “I absolutely think that we’re going to [accelerate] analysis and say ‘this is the best way to run bases’ and ‘based upon the way that people are running bases this is the way that we should actually counter that. This is where the positioning can go or this is the way that we should do this thing.’”


To track players, the system employs a small set of cameras, organized in pairs across the field from each other, to create a “binocular effect” that allows computers a certain depth of field. For the ball, they use basic radar.

The Pickle: “Seclusion”

The system differentiates between players crossing over one another in the camera’s field of view by way of a process called “seclusion.” Employing parallel vision to provide depth, computers actually see two images of the same player. By way of this “binocular” breakdown, individual players can be isolated from one another in-frame so that their data can be collected, rather than confused.

“This is by far the hardest problem to solve,” explains Inzerillo. “It’s a lot different if we were on say a basketball court where you have larger players, larger quality with a camera directly overhead, it helps with the seclusion side.” But MLBAM has risen to the challenge.

“We finally feel like we got the technology to a point where we can solve it in a period of time. Our seclusion detection resolution started out at 67% of the seclusions. We’re now getting to 90% of seclusions automatically resolved by the computer accurately,” says Inzerillo.


The league plans are laid out in three stages. First and foremost, the technology is geared toward baseball operations, with nominal fan use in 2014. This year, three stadiums have been outfitted as a sort of soft opening for the technology: Citi Field, Miller Park, and Target Field. The reason for that is accuracy.

The typical camera array depends entirely upon the geometry of individual ballparks–some have three, some have six–all depending on dimensions. Each camera, however, has a twin on the other side of the park. On the backend, the two angles are stitched together in order to provide the necessary depth and perspective for the computers to understand x/y/z axis, field of play, and plane.

The system itself generates an astounding amount of data, up to 20,000 data points per second for a ball in-flight. For players, data points are recorded at a slower rate–something closer to 30 dp/per sec. But that’s not all.

“Right now it’s even more data because we’re keeping the original video to do enhance trading and things like that. When you see a play that doesn’t resolve with the artificial intelligence either quickly or accurately, we can adjust the algorithms and then leave them on the play without having to have another game played and get back to ground truth,” says Inzerillo.

MLBAM envisions a future where fans can view games live, with these new data visualizations, in real time. And though the statistics will not be available to fans in-stadium at launch, the league has plans to get there.


“We want you to watch on your phone. We’re putting in the connectivity on DAS (digital attached storage) and Wi-Fi so that you can actually get a [data signal] in the ballpark. It’s all part of the coordinated plan to provide rich data sets, but also [give] the fans the connectivity to do so.”

Why Not Wearables?

So why not put the technology on the players themselves? Some might suggest that placing a chip in players’ jerseys would provide better, player-specific data.

“I think that’s cheating,” says Inzerillo. “When you fundamentally change the playing surface or change the uniform or change what the players are doing I think that it alters the game in some way, especially in a game with a rich tradition like baseball. I also feel like we’ve gotten to the point where the technology is good enough, we don’t have to do that to achieve results that you’re seeing both in real time and both highly accurate.”

“As cameras are able to have more and more megapixels focused on the field, we get better and better detection, then it’s easier and easier to dig up those details. We will be able to eventually resolve body position down to where the guy’s fingertips are. We want to make sure the data is right. We want to make sure it’s perfect.”

About the author

Matt Hartigan writes about sports technology for Co.Labs. He graduated from the University of Southern California in 2006 where he studied English, Psychology, Fine Arts and spectatorship