My Experience

From September to December 2016, I worked as "Usability Producer" for Tinkertainment, an 18-person capstone project team at The Guildhall. My primary collaborators were:

I was responsible for:

Scenario

As SMU Guildhall's first VR game, Mouse Playhouse charted new territory for all of the developers involved. We developed the game using Unreal Engine 4 for the HTC Vive. Mouse Playhouse is a light-hearted puzzle game where players guide a pet mice to cheeseby placing blocks to redirect its movement. Players can also play around in a toy area where they can throw objects, play basketball, throw darts, and explore. 

In our playtests, we wanted to capture observations about:

We used a combination of the following methodologies:

These methodologies are very standard in the game industry, and the feedback generated was key to making our game fun and comfortable within our short development timeframe. Nonetheless, I learned that user research for VR games involves unique challenges for researchers.

VR Challenges for User Research

Limitations on group size

4 considerations limited the number of people that we could playtest at one time:

Small expert user population

Across development, I experienced this tension between having access to novice VR users and needing expert VR users. It was often challenging to gather expert VR users from our convenient testing pool of students, friends, and family. Whereas, we had an over abundance of novice users, and sometimes the novices were below the threshold of our demographic. Novice users are great for testing comfort and tutorials. However, expert users were needed to assess the full level progression of easy, medium, and hard puzzles.

VR Sensitivity

Simply put, every user is different, so every user has a different tolerance to VR. Some people are more prone to motion sickness, dizziness, or claustrophobia than others. This is a handy distribution when finding frame rate drops and comfort issues; however, it can disrupt a full length session and remove a much needed data point if the player has to quit the experience early. This is sometimes an issue for intense PC games, but it is more prevalent in VR.

Novelty Skews Gameplay Reaction

VR is a mindblowing, immersive media that many people have strong reactions to on their first time. It is easier to say "that was awesome" even if a game had inconsistent visual language, bad graphics, or boring gameplay. However, experienced users are quicker to critique games and find problems because they are used to the seratonin-boosting effects of VR. As I mentioned above, I had an abundance of novice users and shortage of expert users. It was hard to know when players truly loved our game content or if players were just reacting to their first time in the Vive. 

Unique Mechanics & Lack of Standards

New players must learn an entire input system to playtest your game, and this takes time. Expert Vive users tend to expect some kind of teleporting or navigation control, along with some kind of grabbing control. However, many Vive games introduce new interactions that players must discover on their own. In Mouse Playhouse levels, we had a radio with buttons for different languages to localize text for Spanish and French language users. However, a new player may base their expectations off of mental models of more pervasive technologies -- smart phones and PCs. As a researcher, I had to add buffer time to my experiments to walk players through Valve's provided Vive tutorial to give each tester a clear baseline of controls. VR games must teach users the "WASD" or "A, B, X, Y" of an entirely new game system.

Lack of consistency breaks immersion

I had one user who assumed that no toys were interactive in the toy room because a train didn't do anything when he played with it. That was one toy in a room of 20+ interactive objects. Visual language, rules, physics, and control mappings -- any schema that players build up -- can deteriorate in an instant if something breaks that consistency. Researchers may need to nudge players if they make a faulty assumption based on a bug or incomplete feature. 

User Fatigue

Although time does pass more quickly in VR, after a certain time threshold, users tend to reach a limit. It is fatiguing to stand up for multiple hours straight while waving your arms around, with a high resolution screen fastened to your face. Typically VR fatigue is faster than PC fatigue. 

Observation & Data Collection

As I hinted earlier, I only see a user's computer monitor, not their actual Vive visuals. Furthermore, I cannot easily monitor users' hands or controller usage while watching the screen. This is somewhat different than with PC game testing, because the player is right next to the viewing terminal. Everything is spread out in roomscale VR testing. This problem is amplified if you have only 1 researcher.

Strategies