Hello VR Friends, in this blog post I describe the predominate design philosophy that has driven our VR work to date. It’s a philosophy that has resulted in producing primarily instructional based VR experiences for the purposes of teaching, learning and research. I will describe what I see as flaws in this philosophy and I will propose a new approach that might influence our work going forward. This new design philosophy will favor producing assessment focused experiences over instructional based experiences.
I recently finished reading a book called Experience on Demand, by Jeremy Bailenson, who is a professor of communication at Stanford University and a founding director of the Virtual Human Interaction Lab. This book chronicles some of Professor Bailenson’s extensive research with virtual reality. Bailenson has been working with VR technology since the early 2000’s but he also is somewhat of a historian and references work that was done as far back as the 80’s. Point being VR research is not new however, what is new is for the first time virtually reality technology is now small enough and affordable enough for mass adoption to take place. While previously VR equipment could only be found in labs like Bailenson’s for a cost of upwards of $30k now anyone can own a HMD (head mounted display) for under $300.
My experience with VR is small compared to Bailenson’s. While I have been developing interactive media for over 22 years, including educational game design, I have only been working with VR since 2014. I was excited to read his book to see if any of his theories, findings and philosophies, around VR development, were similar to my own. It turns out they are! The book is largely stories about some of the VR experiences his lab has developed, with some brief reporting about research findings around those experiences. While the book does not offer any concrete advice on what makes for successful VR, it does provide some guidelines. It’s these guidelines where I found the most commonality between what Bailenson’s lab has done at Stanford and the work I try to do, and encourage, at Penn State.
Here are my biggest take-aways from the book:
- While VR might be more engaging than traditional instruction, getting people interested and caring about the knowledge, actual knowledge transfer shows only minor gains over traditional, far less expensive, methods.
- When you directly experience something, the way you can in VR, you see it in a different light.
- “VR is about exploration, storytelling is about control.” Using a technique called “Discovery Narrative” is more effective than traditional linear storytelling.
- What makes good VR is when the entire environment surrounding the viewer is important to the experience. The experience should require the user to explore by either turning their head and or moving around.
- “Ask yourself, does this need to be in VR?” By considering all other options first you can design better VR experiences.
- At all costs don’t make people sick. Framerate over graphics, ALWAYS!
- VR experiences should be supplemental to other learning methods, not a replacement. The experiences should be short in duration, even 20 minutes is a long time in VR.
Working for Teaching and Learning with Technology, at Penn State, it’s my job is to discover ways of integrating VR in the classroom. Most of the work we have done so far focuses on the instruction side of teaching and learning. We primarily have been trying to find ways of using VR as a knowledge transfer tool. After reading Bailenson’s book, and learning about some of the experiments and studies his lab has done, I think a shift in our design philosophy might be beneficial.
The shift I am proposing is to change our current ‘design an instructional experience’ mindset to a ‘design an assessment experience’ mindset. This is not a new approach. It is a method I’ve used over and over when designing educational games. With VR experiences, we have drifted away from assessment, and measurable metrics, as the foremost design considerations. (Even though for research this is ultimately what we are looking for and in some ways have been tacking on at the end.) I believe there has been a rush to cram instructional content into a headset and then simply examine if people respond well to it. After being involved with the creation of several under whelming instruction based VR experiences, and after reading Bailenson’s book where he concludes from multiple studies that knowledge transfer from VR experiences has only minor gains over traditional methods of teaching, it’s time for our design philosophy to move on and create richer, more meaningful user experiences.
While thinking about this shift in design philosophy a few (possibly research worthy) questions came to mind:
What separates instructional based design and assessment based design?
Instructional based design is filled with temptation to create content not suitable for VR. If we always carefully consider takeaway #5 from Bailenson’s book, “Does this need to be in VR?” We should realize most instructional content can be more effectively, and more efficiently, delivered by other means. User’s can quickly identify aspects of a VR experiences that would not require them to have a headset on. Wearing a headset is uncomfortable. To ask users to listen to a lecture, or read large amounts of text, or participate in any other poorly conceived VR content can results not only in distraction but in rejection of the purpose of the experience.
There is also the temptation to cram too much content into instructional based VR experiences. Remember takeaway #7, VR experiences need to be short, definitely not exceeding 20 minutes. The visceral reaction to a VR experience feels more like a rollercoaster than a long slow train ride to most users. It’s a difficult proposition to expect to a user to absorb a great deal of information in a short amount of time, especially if the content does not clearly justify itself as appropriate for VR.
The goal of designing assessment based VR experiences over instructional based designed experiences is to avoid using a screwdriver to hammer a nail. We want the user to care about the experience, discover on their own why they are participating in it, and realize it wouldn’t be the same if it the experience happened any other way. VR is a chance to experience situations not be fed information. Properly designed assessment based VR will allow instruction to happen more organically. It should allow the user to explore while being observed and measured in real-time, as opposed to being told information and tested later. Bailenson points out that VR experiences should be supplemental to other learning methods. Assessment based VR could fit nicely as bookends to other learning. Provided as pre and post tests, VR experience could provide users with measurable tasks and expose where challenges lie. These challenges, which may be unique to everyone, can be addressed with feedback and instruction, outside of VR, before letting them try the experience again. Not only would this validate the VR experience but also the instruction that accompanies it. A well designed VR assessment experience is an instructional experience that disguises the lesson in activity suitable to the technology.
Can VR be the most effective assessment tool ever developed?
Maybe. I don’t like to prognosticate too much into the future with these technologies. I’d rather focus on what we have today and what we can do with it. But it’s not hard to imagine that 5, 10 or 50 years down the road, VR might be capable of creating experiences undisguisable from reality. If that is true, then combining VR experience design with game design could result in assessment methods that would give participates an unrivaled measurement of realistic experience and feedback. Imagine police training that could assess when officers are ready to handle hostile situations because they had performed well in virtual ones. From Bailenson’s experiments we know VR can make user’s care more, even feel more empathy, and that direct experience leads to seeing situations in a different light. With hyper realistic VR it would be possible to give people experience in any situation, allow them to fail safely, but make them feel as if though the consequences are real. The experience then could be evaluated, provide feedback, and repeated, as many times as needed, until an acceptable level of performance is achieved.
Can VR be effective enough at convincing the user they are present in a moment that they react as though they are?
If VR experiences can make users feel present in a moment then they might also make them feel the consequences of their actions in that moment are real. Obviously, there are real consequences to traditional written exams that come in the form of a letter grade, but in real life scenarios, examinations of knowledge don’t always come in written form and consequences are rarely revealed as a letter grade. VR experiences could give users a heighten understanding of the impact, and the importance, of the knowledge they are being asked to display beyond an evaluation of themselves.
Displaying knowledge, or a skill, is one thing but, can you do it in a moment that matters? Understanding, feeling, caring about, the consequences of your actions add pressure to any examination. I’m a golfer. I can tell you there is a big difference between playing golf with my friends and playing in a tournament. The phycological difference between casual performance and pressure filled performance is real. It can effect one’s ability to both mentally and physical react. Any professional athlete will tell you that “game time” experience is difficult to simulate. VR experiences might be uniquely equipped to provide pressure filled performance assessment because of their ability to make user’s feel present in a moment.
Does the assessment based design plan include good VR mechanics tied to quantifiable metrics that define success, failure, progress?
When designing educational games, we always identify what the learning objectives are first. We then search for metrics that can be used to indicate those objectives are met. Finally we design actions and events that allow the players, and the responding system, to effect those metrics. These actions and events are referred to as the mechanics, or game play aspects, of the experience. It’s vital that the mechanics are designed to utilize the interactivity affordances of the delivery platform. If all goes well, this design methodology results in an effective assessment tool in “game” form.
For example, here’s an assessment focused design plan we developed for a game targeted at managers of a walk in health care clinic. The game is called Time and Patients. The learning objective of the game was to successfully manage the clinic for an entire week. The metrics we came up with as measurements of success were 1) revenue, how much money is the clinic making, 2) throughput time, how long does each patient spend in the clinic from arrival to check out, 3) satisfaction, how satisfied are the patients with their experience in the clinic and, 4) walk outs, how many people leave the clinic because the wait was too long. Next we designed actions like reviewing resumes, hiring staff, scheduling staff. We also gave the player the ability to add amenities to their clinic, like a fish tank or WIFI for example. Every action we made available to the player was designed to somehow effect the metrics of the game.
We were not yet using VR when I developed Time and Patients. If we had been, we might have considered building this as a VR experience however, it would not have been a very good one. Why not? We had a perfectly good design plan but, we still had to consider #5 on the list of takeaways from Bailenson’s book, “Does this need to be in VR?” Our design plan’s main actions included reading resumes, and filling out a schedule, neither of which require the user to explore or interact with anything spatially. Accessorizing the clinic could have been done in a full scale virtual clinic, seemingly satisfying #4 on the list of takeaways from the book, however, the spatial arrangement of the amenities was not particularly important, only the fact that they were there was a factor in effecting the metrics. The mechanics in our design plan, while effective in assessing knowledge of the learning objectives, were not good VR interactions. Time and Patients ended up being a successful browser based HTML5 game.
Time and Patients is a good example why planning is important. A design process that uncovers learning objectives, and methods to get to them, is critical regardless of the delivery platform. It is the process that will identify how a technology might be useful or why it might be ruled out. Like games, creating VR experiences is expensive and takes a long time to produce. However, designing assessment based experience using metrics, and the actions and events that effect them, can be prototyped fairly quickly using even simple text. This is what is referred to in the game industry as paper prototyping. I actually developed a web based tool for doing this years ago called WIND. This cycle of planning and prototyping allows for quick iteration before any time consuming production of graphics and programming takes place.
Does VR technology have any unique ability to capture assessment data?
Absolutely. There is already built in technology in some HMDs that allow for eye tracking and heart beat monitoring. This is the start of capturing data that can provide answers to what users pay attention to and how they are react to it. Decision making, reaction time, movement through space, identification of priorities are just some examples of data that could be collected during an assessment based VR experience. This ability to generate assessment data, not usually found in traditional grading transcripts, could have profound impact on evaluating the level of one’s knowledge and the effectiveness of education.
Is VR assessment a chance to rewrite the gradebook?
Maybe. There is opportunity in VR experiences to evaluate, and provide feedback, from a wider spectrum than traditional letter grades. There are grey areas between success and failure in many aspects of life. The separation between an A and B letter grade might be feel significantly different if conveyed as consequences in a VR experience. VR experiences could provide insights to a user’s knowledge beyond right and wrong answers.
This shift in design philosophy, away from purely instructional, to assessment based design is not as radical as it may seem. I mentioned above that essentially assessment based design is instructional design with the instruction hidden in the mechanics of the experience. In many ways the term ‘assessment based design’ is way of rephrasing the term ‘game design’ to make it seem less frivolous. It still seems, to many educators, the word “game” is a four letter word. We often disguise that word but saying ‘simulation’, or we tack on other adjectives like, ‘educational’ game or ‘serious’ game. The truth is instructional design, game design and even software development cycles, have a lot of similarities. (I will talk about that in a future blog post.) We’ve drifted away from game design in VR for a number of reasons. We need to circle back to some established design theories rather than trying to learn everything about VR from scratch. I believe we’ve become too enamored by the new(ish) technology of VR. There has been a lot of experimentation with taking content and throwing it in a headset with out considering the user experience in the design. Ultimately that is what VR is, a technology vehicle for user experiences. It is not simply a content delivery mechanism, we already have plenty of those. The fact is not all content is a good fit for VR but, with a proper, and proven, design philosophy in place, we can better position ourselves to find out what is.
Leave a Reply
You must be logged in to post a comment.