SimCityEDU: A Video Game That Tests Kids While Killing The Bubble Test

A $10.3 million collaboration between Electronic Arts, Pearson, and a nonprofit yields a simulation game that tests thinking and emotion, not just knowledge.

Dammit, more brownouts!

I am trying to cut pollution while maintaining my city's energy supply. I've bulldozed the coal plant too soon, without realizing that the brand-new solar plant has a variable output. Industry, and therefore revenue, is being squeezed by the power cuts—meaning I don't have the money to upgrade or add an additional wind plant. As mayor of this 3-D cityscape, I'm feeling about as effective as Toronto's Rob Ford.

SimCityEDU: Pollution Challenge!, the game I'm playing, debuted last week. For those who played SimCity in the 1990s or 2000s, this PC-based game feels familiar; it's built on the same bits but radically simplified into chunks that take no more than 10 minutes to play, with specific tasks for the player to complete. But what makes SimCityEDU different from other video games, even other video games that have been modded for educational use, is that while middle school players are figuring out how to play this game, the game will be figuring them out right back. As they are zoning neighborhoods or planning school bus routes, the software is gathering detailed evidence about their thinking processes and skills, and whether they're engaged or bored.

The creators, a multidisciplinary team known as Glasslab, have a wild ambition. They want to use game-based assessments like these to wean our education system off fill-in-the-bubble tests, which are optimized for gauging memorized content knowledge, and instead start measuring what really matters in the 21st century: how well people can think.

"We have all these high-stakes assessments focusing the majority of their testing on rote learning and not application of skill," says Seth Corrigan, the Director of Education & Evaluation for Glasslab. "We’re never going to transform education and prepare kids for success if we don’t transform assessment to look at higher-order skills. Everything pointed to games as the way to do that."

Avoiding Chocolate-Covered Broccoli

With its realistic simulations of energy use, pollution, and zoning, SimCityEDU conforms to Next Generation Science Standards recently created by the National Research Council, and includes reading tasks that match the Common Core. Both are voluntary, state-led attempts to create nationwide benchmarks for learning in K-12 schools. But SimCityEDU is not just about teaching content. It's designed to gather evidence about students' "systems thinking."

Systems thinking, the ability to weigh the influence of different variables on a desired outcome for a system as a whole, is a higher-level cognitive ability crucial to success in both business and life. Yet it's difficult to measure such a complex skill set in a multiple choice test, says Corrigan. "A student who sits down in front of a paper-pencil test gets each question right or wrong. Zeros and ones. That's all the information you've got to work with after the student leaves the room." By contrast, while I'm deciding where to cite my new power plant, SimCityEDU is tracking my mouse movements each fraction of a second. They're mining the logic behind every major gameplay event—how many times I take a certain action, when in relation to other events, and how much time I spend accessing information relevant to the game. Do I look at the air quality map to check the effects of my decisions? Do I shut down the coal plant as an experiment before charging ahead to bulldoze it? (No, no I do not. Stupid!)

Aside from providing feedback about systems thinking, the kind of information collected during game play will also be able to build a case for how the player is feeling moment to moment—whether she's frustrated and in need of additional support, or bored and wants a new challenge, or engaging in "WTF" behavior like bulldozing the entire city just for fun. The emotional tracker, produced in collaboration with Columbia's Ryan Baker, will be available in game updates within a few months. Not only will it help designers improve future versions of the game; eventually, it could be used to model the mastery of non-cognitive skills, like perseverance and grit, which researchers are calling far more predictive of peoples' future success than traditional scores or grades.

The challenge for the game designers, which included a team from Electronic Arts, was to gather all this data and perform the analytics while avoiding the "chocolate-covered broccoli" syndrome common to many educational media products—a thin coating of cool over a large serving of good-for-you. Michael John (MJ), senior creative director at Electronic Arts who oversees GlassLab's creative product development, says a lot of that is about finding the optimal level of challenge. "I’ve made lots of games for middle school kids, and the worst thing they can tell you is 'it was too easy.' You have to strike a really careful balance to make the games feel fun and the accomplishments feel satisfying."

Based on a few hours of play, SimCityEDU: Pollution Challenge! isn't quite there yet, but it's close. A set of interstitial tasks that ask you to fill out a diagram showing the cause and effect between, say, sulfur dioxide, coal plants and air pollution were added on the request of the assessment team to provide a more explicit basis of evidence for systems thinking. But they are marred by a clunky interface and feel more like filling out a worksheet than playing a game. The simulation still has the ability to suck a player in and make them feel like they're immersed in another world. I just found myself wishing I could play it for longer and do more exploring.

So do the 1,700 students who've beta tested it, according to them and their teachers. Denise Cruz, a beta tester at Robinswood Middle School in Orlando decided to use the game with two groups of English Language Arts students—a remedial class and a gifted group. The game motivated her struggling readers to pay close attention to its short text passages. The gifted kids, meanwhile, enjoyed using the diagramming tool that I thought was dull. "We modified things based on where we were trying to take the kids," says Cruz. "We would have liked to see teacher-created challenges in there as well."

As each 10-minute challenge is completed within the game, students get on-screen feedback on what went right, what went wrong, and what to try next ("Can you use fewer bus stops to get all the kids to school?") Teachers see a dashboard with updated charts that display each student's level of competency in systems thinking, as they move from a "black box" concept of how inputs affect outputs toward the ability to weigh the effects of multiple variables. The lesson plans that come with the game will include conversation prompts that help students move along to the next step in that trajectory.

Cruz saw students applying their new awareness of cause and effect to the real world. "Kids are relating their experience in the simulation to the general area where they live," she said. "For instance, close to where they live there is a factory. They see an arrow showing the wind direction [in the game] and they're going, oh my gosh, when the wind blows that factory is sending pollution to our neighborhood! How could they even zone that area like that? One full day we took talking about that issue."

MJ says this is part of the elemental appeal of games: "Once something is realized as a world it becomes a lot more resonant than somebody else’s story that I’m reading," he says. "All games do that if they’re done well."

Starfleet Academy?

An independent research firm is observing students at two schools in the Bay Area as they play SimCityEDU and having them "think aloud" about their gameplay decisions to record the effects on problem-solving and learning. The biggest question to be answered as these games hit the classroom is whether they'll catch on enough to build the case that this kind of assessment tool can really work. Getting the GlassLab project to scale is crucial to the team's long-term goal of upsetting the status quo of multiple- choice, static standardized testing, and replace it with a model of continuous formative feedback for continuous improvement.

"Traditionally folks have looked at assessments as filtering mechanisms or ranking systems that put you at the top or bottom of the heap," Corrigan says."The damn thing is, that’s not actionable." Instead, he says, why not use all the tools of big data collection and analysis that current technology makes available to locate someone on a trajectory of understanding and graph the shape of a student's mastery as it changes dynamically over time. To know where you are and where you want to go: this is the kind of feedback that motivates and informs people so they can keep playing, keep learning, and become the best they can be.

The next Glasslab game, with the working title Heroes, will be produced in collaboration with NASA. Set on a Mars colony, with pieces recognizable from Pokemon and Magic, the game tests students' mastery of the elements of argumentation, including claims, evidence and reasoning. Glasslab is in final talks with a few other big game development companies to adapt the assessment infrastructure to more big-name games as well.

When your city goes up in flames during a game, it's not a reason to quit—it's an (awesome) reason to try again and maybe do something different next time. What if failure could feel the same way to students in school—not a downer, but an opportunity?

[Photos courtesy of Institute of Play]

Add New Comment

3 Comments

  • XFuncCarter

    The thing about "Toronto's Rob Ford" is this: Toronto is booming now! You might think that... I don't know.. the mayor just might have something to do with that... maybe.

    (off topic, I know...)

  • Laura H. Chapman

    Interesting, but not related to current federal and state policies that will be testing kindergarten through grade 12 students on the Common Core State Standards--1,620 in English LAnguage Arts and Mathematics ( not counting high school courses), all parsed by grade-level mastery and imagined to be essential for "college and career readiness."
    Seems to me this effort is worthwhile but watch out for unintended consequences. I did a dissertation based on game theory as metaphor and tool for education in 1966, but that was then. Now the analytics part bothers me because the "big data" possibilities align perfectly with the surveillance systems that the US Department of Education wants and that Bill Gates, Jeb Bush (among others) support, not for the learning value, but to exploit for other purposes. See for example, U.S. Department of Education. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Retrieved from http://www.ed.gov/edblogs/tech... Also the $400 million Gates-funded "Teacher Student Data Link Project." (2011). Use and purpose of teacher-student data link. Retrieved from http://www.tsdl.org/UseandPurp...