Through coverage of the market, research and up-to-date analysis, Games and Learning reports on the opportunities and challenges facing those seeking to unlock the educational power of games. more »
Developing The Radix Endeavor, our multi-player online game for STEM learning, at the MIT Education Arcade meant working with lots of different groups. My colleague Susannah Gordon-Messer recently reflected on our experience collaborating with teachers, but the input of their students was equally, if not more, informative.
Over ten months, we worked with over 100 students in our target grade range, in a variety of educational settings. We all know prototype testing is important, but getting actionable feedback that addresses issues at the heart of your game is no simple task.
Beyond the basics you most likely already know, the following are six strategies that represent our top tips to ensure you get the most out of user testing:
1. Create a partnership
While some of our testing was conducted ad hoc, we formed ongoing relationships with a number of schools as well as formal and informal afterschool programs. This meant that a few of the student groups we worked with participated in playtesting several times. For some groups this meant interacting with the breadth of content in the game. Others saw one particular quest evolve over time from paper to digital prototype to final gameplay.
This allowed the students to feel like they were a valued part of the design process. More importantly, over time they got used to the types of feedback we were looking for. They became more skilled at reflecting deeply on their experiences and verbalizing their opinions.
2. Test early, design incrementally
Rather than waiting until we had a fully formed concept of the game to conduct playtesting, we worked with students very early on to help us think through how certain mechanics could potentially work for a specific purpose.
We conducted a few rounds of play testing using paper prototypes and then a few more using crude digital prototypes. This gave us a foundation upon which to build the tools and quests because we already had an idea of what students would expect to be able to do in the game, or what approach they were likely to take.
CALLOUT: Strategy 2 in action
Paper-based versions of the algebra marketplace enabled players to tweak the rules and challenges of the quests as they played, in a way digital prototypes don’t allow for. Letting vendors add their own deals and letting traders haggle over prices showed us that the amount of different goods available for trade wasn’t as important as the sense of agency in bartering. Based on players’ expectations we designed for more open-ended negotiations and added the opportunity for players to choose their own trading goals in certain quests.
Developers started testing with paper prototypes long before the Radix Marketplace feature took its final form.
3. Observe teachers observing their students
Directly observing students’ facial expressions, body language, clicks, and remarks can tell you a lot about the game’s playability. However, they don’t tell the whole story about what students are getting out of an educational game. By having teachers participate, we got to see how teachers interacted with their students to support their learning, how they helped them problem solve, and connections they made to prior knowledge.
We also got to hear teachers’ “behind the scenes” notes on which students were more engaged or persistent than in other classroom activities. This gave us a bigger picture of how Radix would fit into a classroom and what was needed to support the core game mechanics.
4. Bring the dev team
In service of efficiency, we often delineate roles in a hard and fast way. This puts us in a position of having the outreach or design team translating player feedback to the development team, and important things can get lost in translation. When developers of a particular feature witness playtests firsthand, they can more clearly envision why a particular mechanic was frustrating for users and are more likely to come up with a good solution. Developers carry this experience with them throughout the development process, greatly improving both their understanding of the team’s goals and their contributions to the project.
5. Ask thought-provoking questions
Asking for general feedback, or even questions such as “What did you like about the game?” doesn’t always elicit a response that fully depicts the experience. Students often tend to mention one or two things then stop, or simply talk about what they think you want to hear about. To go deeper, it helps to craft more targeted questions that direct a students’ thinking or frame it in a different way.
CALLOUT : Strategy 5 in action
To get at whether a genetics quest was too difficult, rather than asking, “What was hard about this quest?” we might ask something like, “What strategies did you use to breed the right bugs?” This way we are more likely to get specifics about what the player figured out, and whether they knew it right away or had to try lots of things before solving the problem. From there, as a designer we can gauge whether the level of challenge was appropriate and adjust the task as needed.
Critical to good playtesting, the Radix team found, was asking specific questions about gameplay and challenges.
6. Don’t listen to your playtesters
Of course, you should actually listen to your playtesters. Students are the experts at how other students will use and react to your game, and designers playtest in order to improve all aspects of their product. But, sometimes you make design decisions in order to move the needle in a certain area and if that is outside users’ comfort zones, they are not likely to give rave reviews right away. In these cases, if you’ve carefully analyzed their feedback and your intentions, you may decide that a negative reaction is in fact a positive sign.
CALLOUT Strategy 6 in Action
One of the goals of Radix is to provide opportunities for exploration and inquiry. This means we want students to have to figure out how to solve a problem. In playtesting evolution quests, there were many students who said they had a hard time because they didn’t know what trait they were looking for and suggested adding more instructions. Some teachers also gave similar feedback.
However, we felt that these expectations were a side effect of traditional school-based activities. Keeping the activity less directed was key to creating the space for students to make their own discoveries, so we chose not to do everything that would make it smoother for the player.
Six Strategies to Make the Most of Student Playtesting
By Louisa Rosenheck - Feb 16, 2015
Developing The Radix Endeavor, our multi-player online game for STEM learning, at the MIT Education Arcade meant working with lots of different groups. My colleague Susannah Gordon-Messer recently reflected on our experience collaborating with teachers, but the input of their students was equally, if not more, informative.
Over ten months, we worked with over 100 students in our target grade range, in a variety of educational settings. We all know prototype testing is important, but getting actionable feedback that addresses issues at the heart of your game is no simple task.
Beyond the basics you most likely already know, the following are six strategies that represent our top tips to ensure you get the most out of user testing:
1. Create a partnership
While some of our testing was conducted ad hoc, we formed ongoing relationships with a number of schools as well as formal and informal afterschool programs. This meant that a few of the student groups we worked with participated in playtesting several times. For some groups this meant interacting with the breadth of content in the game. Others saw one particular quest evolve over time from paper to digital prototype to final gameplay.
This allowed the students to feel like they were a valued part of the design process. More importantly, over time they got used to the types of feedback we were looking for. They became more skilled at reflecting deeply on their experiences and verbalizing their opinions.
2. Test early, design incrementally
Rather than waiting until we had a fully formed concept of the game to conduct playtesting, we worked with students very early on to help us think through how certain mechanics could potentially work for a specific purpose.
We conducted a few rounds of play testing using paper prototypes and then a few more using crude digital prototypes. This gave us a foundation upon which to build the tools and quests because we already had an idea of what students would expect to be able to do in the game, or what approach they were likely to take.
CALLOUT: Strategy 2 in action
Paper-based versions of the algebra marketplace enabled players to tweak the rules and challenges of the quests as they played, in a way digital prototypes don’t allow for. Letting vendors add their own deals and letting traders haggle over prices showed us that the amount of different goods available for trade wasn’t as important as the sense of agency in bartering. Based on players’ expectations we designed for more open-ended negotiations and added the opportunity for players to choose their own trading goals in certain quests.
Developers started testing with paper prototypes long before the Radix Marketplace feature took its final form.
3. Observe teachers observing their students
Directly observing students’ facial expressions, body language, clicks, and remarks can tell you a lot about the game’s playability. However, they don’t tell the whole story about what students are getting out of an educational game. By having teachers participate, we got to see how teachers interacted with their students to support their learning, how they helped them problem solve, and connections they made to prior knowledge.
We also got to hear teachers’ “behind the scenes” notes on which students were more engaged or persistent than in other classroom activities. This gave us a bigger picture of how Radix would fit into a classroom and what was needed to support the core game mechanics.
4. Bring the dev team
In service of efficiency, we often delineate roles in a hard and fast way. This puts us in a position of having the outreach or design team translating player feedback to the development team, and important things can get lost in translation. When developers of a particular feature witness playtests firsthand, they can more clearly envision why a particular mechanic was frustrating for users and are more likely to come up with a good solution. Developers carry this experience with them throughout the development process, greatly improving both their understanding of the team’s goals and their contributions to the project.
5. Ask thought-provoking questions
Asking for general feedback, or even questions such as “What did you like about the game?” doesn’t always elicit a response that fully depicts the experience. Students often tend to mention one or two things then stop, or simply talk about what they think you want to hear about. To go deeper, it helps to craft more targeted questions that direct a students’ thinking or frame it in a different way.
CALLOUT : Strategy 5 in action
To get at whether a genetics quest was too difficult, rather than asking, “What was hard about this quest?” we might ask something like, “What strategies did you use to breed the right bugs?” This way we are more likely to get specifics about what the player figured out, and whether they knew it right away or had to try lots of things before solving the problem. From there, as a designer we can gauge whether the level of challenge was appropriate and adjust the task as needed.
Critical to good playtesting, the Radix team found, was asking specific questions about gameplay and challenges.
6. Don’t listen to your playtesters
Of course, you should actually listen to your playtesters. Students are the experts at how other students will use and react to your game, and designers playtest in order to improve all aspects of their product. But, sometimes you make design decisions in order to move the needle in a certain area and if that is outside users’ comfort zones, they are not likely to give rave reviews right away. In these cases, if you’ve carefully analyzed their feedback and your intentions, you may decide that a negative reaction is in fact a positive sign.
CALLOUT Strategy 6 in Action
One of the goals of Radix is to provide opportunities for exploration and inquiry. This means we want students to have to figure out how to solve a problem. In playtesting evolution quests, there were many students who said they had a hard time because they didn’t know what trait they were looking for and suggested adding more instructions. Some teachers also gave similar feedback.
However, we felt that these expectations were a side effect of traditional school-based activities. Keeping the activity less directed was key to creating the space for students to make their own discoveries, so we chose not to do everything that would make it smoother for the player.
News Feed
Monkey brains and video games: Pittsburgh researchers learn how to learn
Learning a new skill can be tricky, and neuroscientists aren't entirely sure how humans do it. Researchers at the University of Pittsburgh and Carnegie Mellon University are using video games, brain implants and Rhesus monkeys in an effort to figure it out
Via WITF. June 12, 2019
It’s Game Over for the Institute of Play. But Its Legacy Lives On.
“When we heard the news, it was definitely sad... The idea of how education could be transformed through play and games was inspired by the research the institute was doing on games and learning, and which inspired us as social entrepreneurs and practitioners.”
Via Edsurge. June 10, 2019
STEM School Center Combines Air Force Training, Gaming
According to the Air Force Research Lab, the goal of the Learning Laboratory is to "serve as a national authority on the integration and application of game-based technology to address USAF education and training needs. In addition to leveraging off-the-shelf technology to benefit Warfighter training, our goal is to inspire student interest in science, technology, engineering, and math (STEM), specifically modeling and simulation, and to equip the next generation defense workforce."
Via Military.com. June 10, 2019
Tweets
Follow @games_learning