Real time in game facial animation changes everything

I have had a subscription to the Mixamo All Access Pass for the last year. As a one man indie developer, Mixamo has brought otherwise unattainable characters and animations to me at a fairly affordable price.  Last week at Unite, they made their products entirely affordable with a new pricing model. They also changed the face of 3D art forever with a host of new announcements and the completion of their art pipeline.

In the last few weeks, Mixamo has brought their character creation, animation, and facial animation pipeline together in a way that allows non-artists to create, rig, and animate a character, then bring it into Unity, in about ten minutes. Developers could then create facial animations within the Unity Editor using a webcam.

This is a phenomenal pipeline. I can literally create a new character and have it running around in my game in less time than it takes to cook a pizza. I can sit down and make facial animations that are more complex than AAA games of a few years ago, without animating anything by hand.

This is astounding on its own, but Mixamo put the final piece in place last week when they quietly announced that Faceplus is now in closed beta for real time, in game integration. Using only a webcam, your players facial movements will now be animated on their avatars in real time.

Mixamo Face Plus Demo. This is the version that works in the Unity Editor. Now imagine it in your game, in real time, working for your players.

This will soon be a required feature for a slew of different applications. MMOs, virtual worlds, instant messaging and any game that needs real time facial animation should be looking seriously at Mixamo right now. This year it will put your game above the competition, next year it will be mandatory for not looking dated.

These are all obvious uses, but I believe a whole bunch of new opportunities just opened up for those who are more experimentally inclined. In a crowded marketplace, new opportunities are always nice. I’ve spent some time thinking about this whole pipeline, and what it means for the future of our craft. I believe that enterprising developers and content creators will use it to create new experiences that have not existed up to this point.

Here are seven ideas to get you thinking about ways to use this amazing tech. Please feel free to add your own in the comments. 

Paid actors within interactive experiences

The first time I was introduced to this concept was in The Diamond Age by Neal Stephenson. There are many ways to do this, and I think that when it takes off we will see some really different implementations. Is it virtual theater with the players taking part? Is it the videogame version of a tabletop RPG, mentioned below, but with a paid Game Master? Are unpaid actors practicing their Macbeth? Does an actor drop into your game for two minutes to make you haggle with a real person? I don’t know what all will come of this, but the possibilities are huge, and they weren’t here last week.

A performance capture system from Avatar, 2009. Will professional actors drop into your story based game to perform a real time interaction with your players? They'd have to get paid. Is there a new marketplace waiting to be created? 
The image opens to an interesting article about acting in the age of animation.

Realty T.V. Video Games

Imagine a space battle game in the Descent Freespace or Tie Fighter vein. Player pick from a huge number of available avatars (thanks to them taking minutes for the developer to create.) They then play with a webcam pointing at their face. The drama is high because viewers are seeing the players’ real emotions. When they shoot down an enemy craft, and whoop with joy, their avatar does the same. When they get blown away and thus eliminated from the show, their avatar is yelling in anguish as it dies, mimicking the player’s frustration.

Maybe we don’t even need the drama of combat. In the spirit of Survivor, how about a Robinson Crusoe game where the players are all trying to survive together. With appropriate shortages, player bickering is soon to follow. Being able to see players’ facial movements, even though they are laid on top of an avatar, gives viewers a window into their feelings. Some will be reviled, some loved. Betrayal will surely follow, drama will overflow. It’s going to be wonderful!

Video Game Tabletop RPGs

Tabletop RPG style games within a virtual world are suddenly a possibility. Imagine a game where the adventuring party spends most of its time bickering, trying to figure out what it is doing, and twisting the Game Master’s carefully laid plans into a broken heap. We love it! It is the social aspect that has been missing, and up until this point there was no way to recreate it satisfactorily. Now the players can see each other’s face move while they speak, transforming avatars from plastic masks into believable elvish versions of their friends.

LARP, now we can all look this good. At home. On our computer. Maybe we'll need a new acronym. PCVRP. MCRPG.

With voice modulation, the GM can play the part of many different NPC’s, all with pleasing facial movements to go along with the spontaneous and probably ridiculous dialog. Imagine the time savings of throwing that whole thing on the GM’s lap! I suggest premade scenarios with nice scripts for the GM, rolling by like a teleprompter. That is, when the players do her the service of sticking to the plot. The rest of the time she will have to roll with the players in a way that we cannot approach in a premade story.

YouTube and Twitch Shows

It is now possible to broadcast your version of the real world nightly news as an alien or giant troll. Co-anchors can do so remotely. You could have an elf in Seattle and a troll from D.C., interviewing the castaway in Berlin who was died of starvation during this weeks game.  

High Quality Story Creation on the Cheap

This isn’t groundbreaking for AAA studios, except for the low price point. It is completely groundbreaking for those of us who have never had access to high end 3D content, though.  

Voice actors become voice and face animation actors. Imagine the speed of story asset creation when your voice actors can generation animation files as they read their lines. By the time they are finished they will be submitting not only the dialog, but also the facial animation to go with it! If you have a voice actor who can change her voice, as some can, she can create all of the dialog assets for multiple characters in one sitting. She can also do it remotely, and send the files to you.

Remote Therapy

Why would someone using this do so instead of just using a webcam? I don’t know that it will happen, but I can see some potential reasons. One is anonymity. Perhaps using friendly avatars for people with trust issues may be a good use. Another possibility is situational based therapy. It may be useful for treating things such as war related PTSD. Being able to see the patient’s facial reactions while going through a battlefield simulation may be helpful. Mostly, I included this to try to break us out of the box and into some totally unexpected directions.

An Army program for combating PTSD using VR.

Machinima

Movies (and cutscenes) just got a major production value upgrade. With access to facial animations, will we see a surge of new Machinima? Will it change to encompass more slow paced drama, heavy with dialog? Could we see a blending of Machinima and theater, allowing different actors to render their interpretation? With the movie already created, a single actor could host a live performance of their portion.

In theory new artists should be able to go in and replace previous performances with their own while leaving the rest of the movie intact. Viewers could vote on the best, allowing a definitive version to evolve over time.

I think we are looking at a fast approaching renaissance in 3D interactive and non-interactive media, single user and multi user. I do not doubt that you have some great ideas that were outside of the realm of possibility last week. This week, some of them are possible.

Take a look at Mixamo.com and see which of your unrealistic dreams are suddenly achievable. If you don’t have time to do them yourself, write about them in the comments so someone else can give it a go.