Testing User interaction for AR: Moving Objects

Two common ways to interact with Augmented Reality are using the touch screen, and by moving the phone itself. The results from a user study highlight some of their strengths and weaknesses.

Moving 3D objects has recently become a basic interaction task in Mobile Augmented Reality applications (AR). Applications like IKEA place and AR Measure lets users interact with AR. Selecting, moving, rotating and scaling are quickly becoming common features. To do these well, intuitive controls are needed so interaction tasks are smooth and easy perform.

ARcore demo by Google

Touch screen and Device Movements

Interaction tasks in AR are usually done by either using the touch screen or by moving the device to aim with a cursor (see image below). Existing guidelines and best practices describe these interaction techniques in some detail. Google recommends moving the phone itself to position an object, next to using dragging gestures on the touch screen (Google, 2018). Design guidelines by Apple recommend using the touch screen to select and drag objects to a new position, and to make objects move across the surface they are resting on (Apple, 2018). However, these guidelines tell little about which interaction technique is the most efficient, or if novice users will find them easy to learn.

                                  From left to right: Playground and AR Measure. Images courtesy of Google           

Testing Controls: which work best for your AR app?

To help give an answer I did a user study to compare common methods for user interaction in AR. Several questions I had were:

I also wanted to know if either method was suitable for beginners, and why.

An earlier study by Cammarata (2017) about selection methods for AR pointed out that the movement controls took people some time to learn. AR is new for many people so applications should be accessible and easy to use, not just for people used to interacting with 3D controls.

User Testing

Firstly, I replicated the two interaction techniques in a test application on a Samsung Galaxy S7 phone (See images below). The direct touch screen method was called surface. The other method used device movements to move a selection cursor in the center of the screen; I called it pointing.  
 

                                                        Surface ( Left) and pointing (Right)

Next, a user study was held to compare the efficiency of these methods at a simple interaction task. The task involved moving a red cube into a blue marked area as precisely as they could ( see image below). This simple interaction task was meant to represent simply moving objects to a new place, which is common in AR.

                                                                   Start screen of the positioning task    

Users were also instructed to move the cubes both as precise and as fast as they could. An on-screen timer and score counter reminded them to keep moving the cubes! When they placed a cube into the blue area, a new cube would appear nearby. The task would be over when a hundred cubes were placed this way. This gave plenty of measure points to see how well people did at the task with the different methods.

Results

In terms of efficiency, both techniques were usable for the placement task. With the surface technique, users placed objects the fastest; The entire experiment task was done 111 seconds faster on average when using surface. However, pointing caused objects to be placed 1 centimeter closer to target than surface. This was likely because pointing caused the touch screen remained mostly unobstructed, so you could see what you were doing better.  

                Task completion time, Two way Anova: F( 1, 36) = 24,5 , p= .000 , partial n^2 = 0.40

You can see in the graph that experienced users were faster at the task than beginners. But this was only the case when they used the pointing method, and not when they used surface. This was surprising; because using device movements is a more recent interaction technique than the touchscreen, I assumed people would do better with the surface.

The results show that device movements are a good choice for challenging game applications where controls need to be mastered. With surface, the task results of novice and experienced users were not significantly different.

User Feedback

People commented during the user study that pointing was “… more of an experience”, and “Feels like lifting an object”. Overall, there was a slight preference for the surface technique. Comments like “I don’t have to move the screen as much”  and “(surface) allowed me to stay in place” reveal that device movements were not always preferred.

Also good to know was that both interaction methods were usable for ten minutes or longer. You can imagine that holding the phone with outstretched arms becomes tiresome, but this did not happen during the study, even with the pointing method. This gives a hint of how much time users are able and willing to spend in an AR app.

Thinking about User Interaction

Both interaction techniques were usable for the task, and intuitive to use. Results of the study show that novice and experienced AR achieved different results at the placement task, depending on the interaction method they used. The interaction technique based on device movements gave better results for experienced users, making it a good choice for gaming applications in AR which rewards mastery over the controls and practice.

AR is an exciting medium and developing fast, making its user interface likely to keep on changing in the near future. I hope that this article inspires developers to observe and try more options for user interaction in AR as well. If you have questions or feedback feel free to leave a comment!

Bibliography

AR Ruler App – Tape Measure & Cam To Plan - Apps on Google Play. (n.d.). Retrieved October 17, 2018, from https://play.google.com/store/apps/details?id=com.grymala.aruler&hl=en

ARCore Overview | ARCore. (n.d.). Retrieved October 4, 2018, from Google Developers website: https://developers.google.com/ar/discover/

ARKit - Apple Developer. (n.d.). Retrieved October 31, 2018, from https://developer.apple.com/arkit/

Cammarata, A. (2017, October 24). Rethinking interface assumptions in AR: selecting objects. Retrieved September 17, 2018, from Medium website: https://medium.com/googleplaydev/rethinking-interface-assumptions-in-ar-selecting-objects-a6675c7c1d1c

Content Manipulation - Content - Augmented Reality Design Guidelines. (n.d.). Retrieved May 27, 2019, from https://designguidelines.withgoogle.com/ar-design/content/content-manipulation.html#content-manipulation-selection

Goh, E. S., Sunar, M. S., & Ismail, A. W. (2019). 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review. IEEE Access, 7, 40581–40601. https://doi.org/10.1109/ACCESS.2019.2906394

IKEA Place - Apps op Google Play. (n.d.). Retrieved October 16, 2018, from https://play.google.com/store/apps/details?id=com.inter_ikea.place

Playground (voorheen AR-stickers) - Apps op Google Play. (n.d.). Retrieved October 17, 2018, from https://play.google.com/store/apps/details?id=com.google.vr.apps.ornament&hl=nl

Polvi et al. - 2016 - SlidAR A 3D positioning method for SLAM-based han.pdf. (n.d.).

Translation - Interacting with virtual assets - Augmented Reality Design Guidelines. (n.d.). Retrieved November 2, 2018, from https://designguidelines.withgoogle.com/ar-design/interacting-with-virtual-assets/translation.html#