12 Steps Toward Immersive Learning

Learn by being there. It is this sense of presence that powers the promise of immersive learning. That promise-to enhance the acquisition of new knowledge and improve its far transfer-is supported by theory [1, 2]. Research into situated learning reveals the value of coherent frameworks that faclitate exploration and experimental problem solving [3]. Immersive learning embeds the learner in a such a playful framework.

Immersive learning employs technology to intercept the learner's senses with synthetic signals. The consistency of these signals persuade the perceptual system that it is embedded in the virtual environment. The learner is promoted from an observer of the syntheric world to one of its inhabitants.Immersion promises learners new access to experience. Learners can visit worlds that were once forbidden: worlds too remote, too costly, too dangerous, or too consequential. They can step right into a hypothetical. Learners arrive as lucid dreamers, awake to the illusion, and accepting it. Open-minded, they are ready to learn.

This promise isn't new. Renaissance painters achieved immersion by inventing vanishing point perspective. Audiophiles pursue immersion by piling up speakers. Although grainy, jumpy, and silent, early movies were stunningly immersive to their new audiences-people ran from the theater to escape the onscreen image of an oncoming train. Today, we seek to create immersion in the emergent medium of virtual reality (VR).

Different technologies offer different levels of VR. We can consider a spectrum that ranges from the classic stereoscopic viewer to a stadium-scale multiplayer VR experience. At each level, immersion is deeper, as more senses are more deeply engaged and the learner becomes more of a participant. Each level demands its own immersive learning strategy. We can best study this spectrum by deconstructing the individual features that each contribute to the sense of immersion.

Isolation

One of the most immersive techniques employs the least technology. Whether it is the dark and hushed house at a Broadway play or the built-in blinders of a VR headset, we achieve a lot of immersion simply by masking out the competing signals of real reality [4]. Researchers have measured 36 distinct external classroom distractions [5]. Most of these can be eliminated by the VR headset, regardless of its content.

Stereo Vision

Deep insight always results from the aggregation of differing points of view. That abstract truth, applied literally, explains the principle of binocular vision. Each eye sees a slightly different view of the world, and the brain integrates these differences, yielding depth perception This phenomenon of binocular vision has been exploited since the Victorian era. In the early 1900s, audiences poured pennies into peepshow arcades. Tonight they will line up for tickets to 3-D movies [6].

VR headsets easily deliver excellent stereoscopy. Learning prorofessionals have explored the potential of stereoscopy for the past 100 years. In many domains, it adds only the power of novelty, When that is exhausted, it offers a modest increase in realism that rarely justifies the awkward apparatus required for its display.

However in certain learning domains, depth perception is critical to undertanding. In these domains stereoscopic viewing has been found effective, specifically in the fields of structural chemistry [7] and anatomy [8], especially when applied to the training of surgeons.

360 Vision

When an image contains everything that can be seen from a given point in space, the image is called 360 (also pano, photosphere, surround photo). The learner is at the center of the scene. As in real life, the viewer sees only one sector of the scene at any time, but can turn and tilt the view to see other sectors of the environment. Everyone who has used Google Street View is familiar with 360 vision [9].

Figure 1. A 360 photograph of the author in a VR headset. © 2018 GamesThatWork

360 images are particularly useful in the study of architecture, urban and interior space planning, as well as world culture and history [10]. In any study of the built environment, the traditional frame of a flat image introduces a gross distortion. Through evolution and design, these spaces offer 360 expeiences to their inhabitants. For example, congregants within a cathedral will not only gaze straight down its central axis toward the altar, they will also experience the peripheral rhythmic symmetries of columns and arcades flanking the nave in which they stood.

Headtracking

There are different ways to view a 360 image. When it is displayed on a screen, large or small, the viewer changes the angle of view with the motion of a mouse or fingertip. VR systems, however, introduce the new feature of head tracking. To select the angle of view, learners simply turn to face what they want to see.

This direct, natural control of the field of view enhances immersion. It is associated with headsets, but it can be achieved with a plain cellphone. As you turn your head, you hold the cellphone screen in front of your face-offering a moving window into the virtual world.

Head-tracking has been used to enhance learning in many fields. It is employed extensively by those learning to overcome phobias using techniques such as exposure therapy [11].

360 Video

The 360 image can be video. In this case, the surrounding world is one of moving images. The location of the point of view (the camera location) may also move [12]. While viewers generally cannot control the location of this camera, they can control its angle and tilt.

Recent work has shown the learning effectiveness of 360 video in journalism [13]. Much of the attraction of the new medium is its novelty, but certain subjects get genuine value from 360 video. These include complex and decentralized dynamic interiors like factories and markets, unfocused outdoor scenes such as crowds and cityscapes.

360 Audio

Immersion becomes more profound as more senses are involved. The auditory sense contributes powerfully to the sense of presence, particularly when sounds are located in space and therefore they change as the field of view turns [14]. Audio production is far less expensive than producing visual imagery, yet it can be even more compellingly immersive.

Sometimes known as holophonics or ambisonics, 360 audio can be used to cummunicate acutely first person experience. Current examples are sparse, but these indicate the range of utility by illustrating experiences as diverse as trench warfare and a haircut [15].

Synthetic Imagery

The virtual world becomes far more fluid when it is live, rather than pre-recorded. The viewer sees fresh computer-generated content, rather than static photographs. The view is rendered in realtime, as in a videogame. Now, viewers not only change the angle of view but can also shift their head location to see around objects in the foreground. This effect, parallax, is a key component of depth perception.

Real-time synthesized imagery offers a live world, which can respond to user action. It includes the hybrid technology of photogrammetry, in which the synthetic world is not an artist's model, but the result of scanning a real scene and its contents.

Figure 2. This Sketchfab model shows a highly stylized character, "Claire Blare." Cartoon worlds are generally more immersive than realistic worlds with similar visual fidelity. Visitors in cartoon VR see a self-consistent world, while in more realistic VR, the eye seeks out all the defects. © 2018 GamesThatWork

Reality

When the live virtual world includes a live view of the real world, this is mixed reality. This category includes augmented reality, where the system embeds virtual images into the live environment, either by intercepting the video feed of a camera or by reflecting synthesized images into the view seen directly through a visor.

It also includes augmented virtuality, in which the system embeds bits of the live world into its computer generated scene [16].

Figure 3. Paul Milgram, with this "virtuality continuum," offered an early map to the spectrum of virtuality [17]. © 2018 GamesThatWork
[click to enlarge]

In augmented reality, viewers control not only the angle of the camera, but also its location. A geolocated application, using GPS for example, is responsive to location all over the world. Pokémon GO provides a well-known example.

Figure 4. Augmented reality helps Air Force trainees learn by interacting with virtual anatomy superimposed on live standard patients. © 2018 GamesThatWork
[click to enlarge]

In some learning venues, particularly traditional classrooms, the full immersion of virtual reality is suspect. Administrators are loathe to allow learners to escape the classroom, even if they escape to a virtual learning environment. To these administrators augmented reality offers a welcome reduction of immersion. Students can enjoy stereoscopic visualization of interactive 3-D models without the isolation of a virtual reality headset.

Locomotion

High-precision location becomes possible when GPS satellites are replaced by local landmarks, such as the miniature scanning electronic beacons called lighthouses. Alternative "inside out" systems use machine vision to extract ad hoc landmarks from the environment. Virtual reality systems with one of these technologies allow learners to walk around, stand up or crouch to see new things. Free movement adds extraordinary verisimilitude to the virtual scene [18]. Walking around and exploring, rather than sitting and looking, makes the user far more present in the virtual space.

Different systems offer different volumes: HTC's mass-market Vive operates at "room scale" with outside-in lighthouse technology, while emerging inside-out systems work at warehouse scale or larger.

Walking in a tightly registered virtual space makes it feel rock solid. But systems that lack the requisite positioning technology still often attempt some sort of virtual locomotion. Users can move in the virtual world, but their real bodies go nowhere. The dissonance between visual cues of movement and the vestibular sense of stasis frequently results in nausea.

"Simulator sickness" has been a major obstacle to the broad adoption of virtual reality. It can result from many techncal sources-frame rate, latency, tracking fidelity, and horizon drift-but it is most often the result of a misguided design decision to compel a visual sense of locomotion that is unsupported by vestibular sensation.

Instruments

A system that can precisely track the learner's head can (with more hardware) also track moving parts. In particular, it can track the hands. The virtual world becomes more immersive as the learner reaches out to touch it. Handheld controllers also deliver haptic signals to the hands, engaging more of the sensorium in the virtual world [19].

There is a rich history of scientists who advocate hands-on learning. These include seminal thinkers such as Seymour Papert and Maria Montessori, who promoted the practice of learning with her manipulatives. Montessori wrote "…first in play and then through work, the hands are the instruments of man's intelligence…" [20].

Interactivity

When the player touches the world, the world responds. Virtual worlds can respond to real actions-throwing an object, wielding a weapon, or opening a door. The player is not just present, but participating. The players acts and enjoys the consequences of action. A world that responds offers profound immersion and throws open the doors to exploratory learning.

The arguments for learning based on interactivity and simulation have been made for decades, and center on the phrase "learning by doing" [22]. Entire schools, conferences and careers are devoted to the study of this effect.

Figure 5. In "Kitchen Designer," players perform situated problem-solving in a virtual space. © 2018 GamesThatWork

Challenge

It is not enough to be present. It is not enough to be interactive. A deeper immersion comes when the learner is motivated to achieve a particular goal in the virtual world. The learner is fully immersed when the world presents challenges that must be confronted to achieve an end. Immersive learning is acheived when these challenges lead players to explore the learning objectives.

The introduction of challenge into the leaning process forms the core of game-based learning [23]. Learning games can be produced many media from simple cards on a table top to a massive, military wargame exercise. Digital platforms for game based learning range from standalone hand-held devices to global networks of specialized hardware. But there is a unique synergy in the combination of learning games and virtual reality. Its adherents (among them the present author) would argue this combination represents the peak of each technology.

Figure 6. This Korean fan site shows "Brush Up VR." It is a vigorous, humorous action game in room scale virtual reality. Children learn about conscientious dental hygiene and tooth surfaces, while having fun. © 2018 GamesThatWork

Learning

Visitors to VR worlds can return with new skills and insights that will improve their experience of live reality. The challenge and privilege of this generation of learning designers is to discover how to employ these new componenets of technology and their associated psychological effects to create a new discipleine of immersive learning.

References

[1] Dede, C. Immersive interfaces for engagement and learning. Science 323, 5910 (2009), 66-9.

[23] Jacobson, D. Constructive failure. Serious Play Conference. 2015.

All illustrations in this article represent GamesThatWork projects.

About the Author

Dov Jacobson is a pioneer in game-based and immersive learning. He leads the Atlanta-based studio, GamesThatWork, to develop interactive mixed realities of virtual, mobile, social and web. They apply game science to help players acquire new knowledge, exercise new skills and form new habits. The player's achievements in-game confer real world benefits . These benefits include oral hygiene behaviors for children, disciplined thinking for U.S. intelligence analysts, design tactics for Boeing's aerospace engineers, market segmentation strategy for entrepreneurs, and more. This work has recently won top awards from authorities such as the National Academy of Medicine, the U.S. Education Department and the Entertainment Software Association. Dov is committed to articulate the principles that his team explores and share the progress they achieve.