Making Portal Painter in ARCore

For the last few weeks, I’ve been working on some fun projects with folks at Google Creative Lab, and playing with ARCore, a new Augmented Reality SDK for Android. I do a lot of independent game development, and work primarily in Unity, and it’s been cool to see how fast I can go from zero to finished AR project.

Portal Painter is one of my favorite prototypes that I’ve made so far. It lets you take surfaces around you, and use them to doodle windows into alternate worlds.

Below, I’ll talk a little about the process of developing Portal Painter with Unity and ARCore, and some tips and tricks I learned along the way. I won’t go into getting the project set up (which you can read more about here), but there will be some code ahead, so it will probably be most useful if you have Unity experience. That said, ARCore handles most of the heavy lifting for you, so most of the Unity concepts are pretty basic. Let’s dive in!

Ideation

Before starting with ARCore, I took some time to dive into other VR/AR experiments. I was drawn to ideas that were playful and abstract (probably not too surprising, given the whole game developer thing) — things that focused less on precision and realism, and more on the creative potential of blending the digital and the physical.

A lot of the AR demos I saw incorporated portals into other worlds. I loved how surreal they were, but noticed that most of them were focused on the act of stepping into the other world, with the portals themselves generally static objects. Was there potential for creativity and delight in making portals dynamic? In letting people draw them themselves?

I was inspired by other folks’ experiments like Sprayscape and Speak To Go, and Google resources like Blocks, and I wondered: what if we could use AR to actually draw new locations in our own? What might you draw on your bedroom ceiling or office wall? What kind of delight could we create by letting people doodle new worlds into their own, scratching away the surface of their world to reveal something delightful underneath?

And with that, Portal Painter was born!

Development

General code

Before diving into ARCore, I developed the overall structure for how the portals would work. They have four core parts:

I won’t get into all that code here (but if you’d like to see it, the open source link is at the end!). The portals generally function like this:

A high-level overview of how the portals work.

Once they worked in the virtual world, it was time to actually place these portals in the real world with ARCore!

Positioning

One of our biggest priorities was to make it feel like the portals really lived in your space and on the objects around you. We could have just dropped them in space some standard distance away from the camera every time, but we all agreed that the delight came from being able to paint a window directly on your wall, table, or floor — like you were etching away at its surface to reveal something fantastic underneath.

First, we had to figure out where the location of a portal should be. That meant figuring out whether a user touch intersected with a real-world surface, and if so, where that point was. Typically, in game development, I would place objects in virtual space by raycasting from the camera, and seeing if that ray intersected with any object of interest in the 3D world. For AR, I used the TrackableHit and Session classes, which let you raycast against real-world objects:

(See on the repo)

Our TrackableHitFlag specifies that we want raycast data about any relevant TrackedPlanes and any relevant points in the current frame’s point cloud (this distinction will be important in a moment). The TrackableHit that it returns gives us, among other things, its location (in Unity world coordinates). We use this to instantiate our portal at the correct location on our surface.

Orientation

So now we have our location — but what about our rotation? We wanted to mimic the feeling of painting on floors, walls, and other surfaces as much as possible, and that won’t work if we keep painting upright portals on floors, or flat portals that extend through walls.

In order to orient portals perfectly on surfaces, we would want to put the portal in the right place, and then rotate it so it’s flush with the surface. One way to do this is to get the plane’s normal (the vector perpendicular to its surface), and adjust our portal’s orientation to match. But complicating this is that ARCore — like many other AR SDKs — only supports flat plane detection, ie floors and tabletops. Vertical features, such as walls, are treated as part of the point cloud, not detected as planes. As such, they do not have real normal values. So how can we work around this?

While watching folks play around with Portal Painter, I noticed that when they drew on walls, they typically drew on them head-on — not from an angle. This meant that most of the time, we could assume that the correct orientation of a vertical portal was simply facing the camera. We could get a surprisingly good hit rate by combining plane normals for the horizontal portals, and “facing the camera” for the vertical ones.

Once the initial placement was done, users could draw on it from any angle they liked. ARCore would keep everything in place, allowing users to walk around and peer into them as usual, and the abstractness of the drawing and the other world would help blur instances where orientations were slightly off.

I wrote a quick ‘n’ dirty function to cover both situations:

(See on the repo)

This ended up bridging the gap between horizontal planes and vertical points surprisingly well, allowing folks to draw both without encountering jarring differences.

Now you try!

I had a great time prototyping with ARCore, and was really excited to see how quickly I could integrate it into my projects. You can see Portal Painter and other playful prototypes on the AR experiments website. You can also learn more about ARCore here, or dive right in with ARCore + Unity here.

Want to try out Portal Painter yourself? The code is open source — you can check it out here. Have fun!