Skip to main content

The Video Game Engine in Your Head

Scientists use video games to explain how the brain interprets the world around us. 


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


For years now, physicists and engineers have been building computer simulations of physics in order to understand the behavior of objects in the world. Want to see if a bridge would be stable during an earthquake? Enter it into the simulation, apply earthquake dynamics, and see what happens.

Recently, the prestigious Proceedings of the National Academy of Sciences published work by MIT psychologists (and my labmates) Peter Battaglia, Jessica Hamrick, and Joshua Tenenbaum, arguing that all humans do roughly the same thing when trying to understand or make predictions about the physical world. The primary difference is that we run our simulations in our brains rather than in digital computers, but the basic algorithms are roughly equivalent. The analogy runs deep: To model human reasoning about the physical world, the researchers actually used an open-source computer game physics engine — the software that applies the laws of physics to objects in video games in order to make them interact realistically (think Angry Birds). 

Battaglia and colleagues found that their video game-based computer model matches human physical reasoning far better than any previous theory. The authors asked people to make a number of predictions about the physical world: will tower of blocks stand or fall over, what direction would it fall over, and where would the block that landed the farthest away land; which object would most likely fall off of a table if the table was bumped; and so on. In each case, human judgments closely matched the prediction of the computer simulation ... but not necessarily the actual world, which is where it gets interesting.

Whether or not a tower of blocks (or a bridge, etc.) will fall over depends on its exact dimensions and the exact position of each block, potentially down to the millimeter. But a human can't tell the dimensions and position of every block down to the millimeter just by looking. That is why we invented rulers. Instead, we know approximately how big each block is and where it is. If the simulation was given the exact coordinates of the blocks, it predicted human judgments reasonably well but far from perfectly. If the simulation was given approximate coordinates, taking into account human uncertainty, it matched the actual world less well but human judgments very well.

In retrospect, it may seem intuitive that when we made predictions about the physical world — what will happen to towers during earthquakes or books on shelves — we query an internal, virtual simulation of the real world, but it represents a sharp departure from previous scientific thinking. Many scientists thought that we use rules-of-thumb to predict the world around us.
 
As a theory, the rules-of-thumb account seems to be a failure: Battaglia and colleagues tested a number of plausible rules-of-thumb. For instance, maybe we base our guesses as to whether a block tower will fall based on the tower’s height or center of mass. None of the rules-of-thumb fared as well as the simulation account. Even worse, rules-of-thumb have to be tailored to the question: The heuristics for the tower of blocks scenario are useless for the table-bump scenario. In contrast, simulations are one-size-fits-all.

But that leaves the question of whether the simulation account is plausible. After all, if we already have a physics simulator in our heads, why did scientists have to discover the laws of physics, and why do we have to learn physics in school (or not at all)? Part of the answer may lie in the distinction between implicit and explicit knowledge. A bird doesn't have to be able to teach a university course on aerodynamics in order to fly, and we don't have to understand biophysics in order to walk or neuroscience in order to think. 

Are you a scientist who specializes in neuroscience, cognitive science, or psychology? And have you read a recent peer-reviewed paper that you would like to write about? Please send suggestions to Mind Matters editor Gareth Cook, a Pulitzer prize-winning journalist and regular contributor to NewYorker.com. Gareth is also the series editor of Best American Infographics, and can be reached at garethideas AT gmail.com or Twitter @garethideas.