AI and Games is a crowdfunded YouTube series that explores research and applications of artificial intelligence in video games.  You can support this work by visiting my Patreon page.

Facade is one of the most unique and novel games of its kind: a real-time interactive drama where players communicate directly with non-player characters in a short dramatic scenario. While over 15 years old, the game is one of the rare cases of an academic research project that achieved global status and popularity. With exposure in mainstream press, award recognition and a cult status on the internet as popular YouTubers run lets plays through its quirky and occassionally awkward drama.

In this AI and Games case study, we’re going to take a look at how Facade works. The years of research and development required to make it happen and how every layer of the games AI systems had to be invented, in order to make it work.

About Facade

In Facade players are invited to the apartment of Grace and Trip: a married couple who met courtesy of the player character back in college. However, things aren’t going so well in their relationship and your coming round for drinks that evening is the make-or-break moment for their marriage. You are unwittingly dragged into passive-aggressive psychological head games as the couple lash out at one another about their hobbies, their career choices, their families, their sex life and much more.

Players interact with the couple by wandering through their apartment and interacting with objects and furniture around the house, or by chatting with them directly. What makes Facade stand out from so many other games is that you are talk to Grace and Trip by typing into the keyboard and the characters interpret your inputs and respond accordingly. They react with elation, surprise, laughter, disgust or even anger as you challenge their points of view, compliment them on their decisions, ask them about issues they don’t want to discuss or just outright insult them.

Ultimately depending on your actions, the game will result in one of several pre-defined endings. If you help the couple navigate their issues successfully, their love for one another is rekindled and they politely ask you to leave. However, like in real life, this is a difficult process to navigate and can easily slip into one of the many bad endings. Players can be ejected from the house for their treatment of the couple or other antisocial behaviour, with Trip forcing you out the door. In fact you can even have Trip slam the door in your face at the very beginning if you successfully insult him. You can even ignore the drama that awaits and simply turn around and get back in the elevator. However, the remaining bad endings arise from your continued discussions with the couple, where Grace can decide to leave Trip or vice versa, or potentially being told to leave after nothing is resolved, only for the couple to continue to bicker and argue after you’ve left the apartment.

Facade is well-known in the YouTube ‘let’s play’ community.

Facade holds a special place in game AI history: a research project that achieved cult status! A quirky and eclectic game that stands out as one of the most novel applications of its time. Creating this exciting and dramatically rich environment where players can interact with and shape the story as it happens. Resulting in the game being downloaded over 5 million times since its release in 2005 – and I suspect that number is dire need of an update. But despite its success, the game is most certainly more famous for when it goes wrong thanks to the agency given to the player. In the past 10 years Facade has become infamous within YouTube and the let’s plays community, with high profile creators garnering millions upon millions of views for their playthroughs of the game. It’s ability to react – or fail to adjust – to player behaviour ranging from the perfectly normal to the completely irrational makes for great comedic fodder. Its cult status means that many people are not aware of its AI research origins and know it purely for the meme fodder it generates. But the game is but an early chapter in the larger story of its developers, seeking to create a fully interactive drama and the one big step forward taken in making that a reality.

Andrew Stern

Facade was developed by two people: Andrew Stern and Michael Matteas. Before the game began production, Stern was a game developer and designer at PF.Magic in San Francisco and had worked on several entries of the highly popular Petz series. Meanwhile Matteas was a PhD candidate at Carnegie Mellon University and his early research at that time had been around building physical and digital avatars that expressed some form of intelligence that could build emotional connections with human users. Both were interested in the idea of autonomous interactive characters and systems each in their own way. This led to Stern attending AI research conferences, and ultimately the pair crossed paths. Their mutual interest in intelligent avatars led them to pursue the next step, intelligent actors and the formation of the interactive drama.

Their passion project was a labour of love with no budget to speak of and pair worked on it around other projects – with Matteas a grad student and Stern working self employed. The 5 years of development included 3 years spent authoring the drama itself, plus over 5 hours of recorded dialogue for Grace and Trip provided by actors. Despite this herculean effort, the team achieved only around 30% of what they had originally envisioned. But that was more than enough to make an impression.

Released for free on Windows XP in 2005, with a MacOS version in 2006, Facade became one of the most famous and successful projects of its kind. Having previously been a finalist at the 2004 Independent Games Festival, the game subsqeuently won the Grand Jury prize at the Slamdance Independent Games Festival in 2006. With both Stern and Matteas garnering significant exposure with articles written about the game in the Atlantic, the New York Times and the Guardian. Meanwhile numerous academic publications in journals, conferences and books have been published discussing the design, development and processes behind the game itself. Not to mention all the YouTube walkthroughs, guides and jokes that have accumulated millions of views over the years.

But how does it all work? Well in order to really understand what Facade is doing under the hood, we need to take a look at a number of connected but distinct elements of the game.

How Facade Is Designed

At the time Facade started development, the idea of interactive story-driven was still very much in its infancy. While story branching and interaction systems already existed in the likes of point and click adventure games and some RPGs, Facade sought to achieve a high level of agency. Where the player could have a substantial impact on how the story plays out. Even now in 2020, games where players have a huge impact on the story are still relatively unique. Even series lauded for their story telling and character interactions such as The Walking Dead or Mass Effect don’t provide the player with much influence over the story, but rather different paths within it they can take based upon criteria being met.

Facade was designed with the intention of crafting a character-rich plot that based upon your interactions and conversations with the other characters, it would either immediately or gradually influence how the game pans out. To achieve this, the player is secretly playing along in several of what are known as ‘social games’. These social games are specific phases of Facade where based upon your interactions, they can influence Grace and Trip’s feelings on a particular subject matter, their self-awareness about their underlying problems and their affinity towards the player. Every provocation, criticism or praise found within the natural language typed in from the keyboard will nudge the characters feelings on each subject. All the while the game is monitoring the tension between Grace, Tripp and the player. All of this can influence the story beats that play out, how a character responds at a particular point in time and ultimately, which ending is reached in the drama. It’s important to recognise that Grace and Trip are essentially actors, where instead of learning specific stage directions or cues for a rehearsed play, they’re effectively improvising in real-time as an AI stage director is putting together the play in real time. However, while it’s being improvised on the fly, their performances are all pre-built and are designed to work in specific circumstances. And the underlying challenge the two actors face is in working together in real-time to ensure those pieces are put together correctly and ensure each story beat plays out successfully.

To help structure this and balance out how specific topics are explored or provoked, each playthrough of Facade is broken up into two main social games: the Affinity Game and the Therapy Game. The affinity game is where the system tries to determine whose side the player is on. This is actually also running another social game within called referred to by the developers as the ‘Hot Button Game‘. Hence while discussing the issues Grace and Trip are facing in their marriage, it can spill over into other more incendiary topics such as their sex life or the prospect of divorce. While this helps the player gain more backstory on each character, if you continue to push too far into a given topic, it can actually undo the affinity score you’ve achieved for a given character, and they will act annoyed as a result.

But the second phase is where things reach their apex. The ‘Therapy Game’ is where the player has influence over whether each character reaches self-realisation about their underlying issues. If the players actions increase the self-realisation meter for that character, then specific elements will unfold. However, throughout this segment the game itself is monitoring the supposed tension between all the actors in the scene. The state of each character and the tension in the room then impacts how the story plays and how each character responds in the closing minutes.

In order to pace and manage each of the social games within the intended drama of the story, Facade coordinates this within what’s known as the Drama Manager. It figures out where the plot goes based on whats happened and how the emotions each character is ‘feeling’ (or rather, has recorded) at that time. In amongst this, is the Action Behaviour Language ABL, pronounced “able”, which encodes not just the reactions and dialogue chosen of each character in specific situations, but also how they position themselves around the room, interact with world objects and behave in a way that fits what the Drama Manager wants to occur. And lastly, there is your input as the player. Facade runs an interpreter in real-time that attempts to pick out the underlying sentiment of what is typed by the player, and then uses that to influence how the drama manager coordinates story beats – potentially adding new story moments, re-writing the plot as it goes – which ultimately influences how the actors respond to you.

The Drama Manager

The Drama Manager – otherwise known as a beat sequencer – decides which story beats are played out in realtime during execution. It looks at the current state of the story and the characters and decides which unused beats would help move the story through the intended tension arc. Only one beat is active at any given time and based on what’s happening in the game – or rather what you’re doing as you play – it can decide to modify or change the current beat accordingly. The game has 27 main story beats, with a collection of them used in the first act to set up the tension, a handful for the collection of available endings in act 3 and the remainder being selected for the core social games in the middle of the story. The game never uses all of the story beats in a given playthrough. Hence while there are 27 main beats, the average playthrough only uses around 15 in a complete run.

Each beat – much like their inspiration from dramatic theory – is designed to be the smallest narrative sequence the game can run. Each beat is encoded in the game as a collection of what are known as Joint Dialogue Behaviours or JDBs. These are game actions that will happen in a specific sequence in order to complete the beat. Each beat is written in the ABL language and we’ll get into what they look like in a minute. But for now just know that each JDB is comprised of lines of dialogue between Grace and Trip as well as actions for moving around the world and interacting with objects and the player. These are usually built around one topic and only last maybe a few seconds each. But one beat can be comprised of anywhere from 10 to 100 JDBs. This allows for full plot points to be played out, such as Grace’s obssession with redecorating, or Trips attempts to force Grace to enjoy their Italian holiday. In any case, each JDB once executed with change one or more values of the story state, as well as the affinity each character has to the player, the tension of the story and how close both characters are to reaching their self-realisation.

A typical story beat is written as a collection of specific JDBs known as Beat Goals. These are the narrative goals that we would expect the game to play out for this beat based on specific configurations of the characters state. Essentially, each beat has written within it how the story should play out without the players involvement. This means that a character can bring up a new topic or an existing topic thats already been covered in the current or previous story beat (TRANSITION-IN) and then have the two speak their lines. In many cases this presents situations intended to force the player to respond in-line with the social games I mentioned earlier (BODY-GOALS). Plus it the ability to wait for the player to respond (WAIT-GOAL) and how to transition out of the story beat based on the input (TRANSITION-OUT). Either because the player didn’t respond to it, or specific behavioural triggers that happen because of something the player said.

Now it’s important that the structure of these narrative beat goals do not change. Otherwise it runs risk of breaking the flow of the story. But the drama manager needs to be able to interject so that player input can be responded to. To do this, the system can use a special JDB known as a Beat Mix-In. Beat Mix-Ins enable the drama manager to change what future beat goals are going to be executed. It can add a new beat goal to address a specific point made by the player, remove an upcoming beat so that the story can change direction or re-order future beats so that topics that were lined up for discussion later are brought forward to now.

Normal beat mix-ins only change what’s happening in that main story beat but there is also the Global Mix-In, which is designed to bring new topics into the conversation. None of these are directly related to the main story, and are considered ‘satellite topics’, which are in orbit around the main plot and independent of the flow of the narrative. These can include discussing the objects in the house, such as the furniture, Grace’s ornaments, the couple’s wedding photo or the view from the apartment, to more delicate topics such as Grace and Trips sex life or whether they’re receiving therapy. Each of these mix-ins add in whole new discussions that have three tiers of discourse, which are only revelaed depending on how much you dig into them in a way that the characters responds.

This allows characters to acknowledge your behaviour – including things like praising or even flirting with them – react in some form or another and then get back to the main story. The drama manager is always retaining a memory of what the core storyline is. So when interrupted with a mix-in, it can get back on point. In fact you can even interrupt a mix-in with another mix-in and the game will eventually get you back to the main narrative. When a main story beat is interrupted by a mix-in, the drama manager can decide whether to restart the last beat goal it attempted, or return to where it was during the execution. This is actually handled within the JDB, where specific ‘gist points’ are assigned that declare that once the player reaches a specific point in the story, they’ve clearly got the gist and it can resume the beat goal if interrupted because you know what the point of the conversation is. If the gist point wasn’t reached, or there isn’t one for that beat, then it will restart.

As mentioned, there are 27 different main story beats within Facade. In order to flesh those out and bring them to life, there are approximately 2500 joint-dialogue behaviours written into the game. Handling all the different directions that the story can take. Of these 2500 only around 66% of them are beat goals and mix-ins, with the around 33% being global mix-in progressions and the final 1% the long-term autonomous mix-ins.

A Behavior Language (ABL)

So now we know how Facade builds and maintains its story through the Drama Manager, how do Grace and Trip tell the story? As we just discussed, the drama manager’s joint dialogue behaviours are written in the ABL language. A Behaviour Language is based on an existing agent beahvioural language called Hap developed in the mid 1990s, but the key part is the ABL handles multi-agent coordination and therein lies the great secret of Facade. Grace and Trip are working together to put on a performance for the player and communicate between one another the whole time you’re playing.

The beats within the drama manager are all written in ABL as collections of actions to execute. Some of these actions are designed to respond to events in the world, while others solve specific goals given to a character to complete. Consider the opening of the game, where you knock on the door and wait to be invited into the apartment. This is a sequential behaviour for Trip encoded in ABL, where first he needs to wait for a knock on the door to occur, followed by that he’ll let out a little sigh, then two subgoals that are then satisfied: opening the door and then greeting the guest.

Image from “A Behaviour Language for Story-Based Believable Agents” by Matteas and Stern (2004)

ABL actions have conditions that need to be met before one variation of the chosen action can be executed, and effects are computed as a result of successful completion. If we consider the OpenDoor behaviour, the character needs to find the door that was knocked upon and compute the distance. If they’re close enough, they’ll answer the door, if not there is a fallback behaviour to tell the player to open it and come in the house. Subgoals often have more than one way to complete them, hence you might see Grace and Trip execute the same behaviour in slightly different ways between each playthrough depending on where they are and what’s happening at that time. Meanwhile once Trip has successfully opened the door and welcomed the guest, it deletes the fact that there was a knock on the door from the WME, the Working Memory Element of the character, which stores all the knowledge Trip has of the world at that time. This is of course to stop him trying to answer the door again if you tried knocking on it.

While Trip inviting you into the house is a single sequential behaviour, a situation such as Trip making drinks at the bar while Grace argues with him is a joint behaviour (and the crux from which the JDBs inside the drama manager are built from). Grace and Trip secretely synchronise their entry into the behaviour and take turns playing their part. Trip offers you a drink and will walk to the bar, continuing the discussion and making the drinks if he makes it there. Meanwhile Grace will roll her eyes and interject, making comments about his cocktail shakers, forcing Trip to argue with her. Grace and Trip secretely signal to one another what actions they are going to commit and agree to participate together or reject the action if they’re unable to complete the request. This is all carefully synchronised and actions can’t be executed until they agree on whether they will work together or not, hence you’ll notice they never talk over each other. Sometimes actions will go smoothly and hit the end of the beat goal as planned, while others require numerous fallbacks in the subgoals to achieve the desired effect. It actually has a lot in common with the likes of professional wrestling: where the performers know the end goal they want to reach, but how it is executed is based on how the world is configured at that time and they adapt to suit.

As I mentioned before, the Drama Manager has the mix-in behaviours ready for how to address the players inputs in the world or from the text systems. But how does that get fed into the ABL system? Well each behaviour can have handlers in place that allow for them to execute slightly differently depending on how things are panning out. Like say if you walk into the apartment before Trip invites you in, then he won’t invite you in anymore. Meanwhile if you ask where Grace is before he yells for her, this will prompt him to move that action forward to be executed now.

In amongst all this are special Cross-Beat Behaviours. While the story beats are assigning specific joint behaviours as goals to execute, the characters can still make decisions for themselves that span across a particular story beat or into the next one. This is mostly of use when the characters executing staging requests: meaning they’re moving around the apartment for dramatic effect. If you once again consider the opening minute when you’re invited in, the two characters move into positions around the apartment either together or individually based on whats happening and those movement actions can happen while other dialogue behaviours are being executed.

The Text Systems

The last big element of Facade to talk about is the system for reading in the players keyboard inputs. This is what’s known in Artificial Intelligence as a natural language processing and understanding problem. And while the technology behind this has made significant strides in more recent years, with commercial grade voice and text recognition technology becoming more commonplace, this was a major problem for Facade to try and overcome 20 years ago and the resulting system cuts several corners. Which is perfectly understandable given otherwise it would’ve taken another five years to come together.

Facade‘s language processing is broken down into two phases: first it maps text to one of around 30 Discourse Acts, recognising the that player is say criticising or flirting with a specific character. Once the discourse act is established, it then maps this is a JDB in the drama manager and is injected into the game.

A collection of the available Discourse Acts lifted from “Natural Language Understanding in Facade: Surface-Text Processing (2004)”

Figuring out the discourse act is done through a series of rules, which recognise typical utterances and statements of users. There are over 800 rules built into Facade, allowing it recognise behaviour ranging from greetings, expressions of particular emotions, confusion, apologies, criticism, praise and much more. All of these rules are hand written and are designed to transform a specific sentence in to a corresponding JDB for the drama manager. The rules are often fairly complex, given they need to differentiate positional facts (“that looks great”) from compliments (“you look great”), multiple facts in the same sentence, double-negatives such as saying “I’m not bad”, when you actually mean “I’m good”. Plus the system can recognise around 1000 idioms typically used in the English language, ranging from calling something “a piece of cake” to telling someone to “hold their tongue”.

The system is designed to compile down into Jess: a rule engine written in Java. Translating the roughly 800 rules into approximately 6800 Jess rules for runtime execution. And this is but one of many content bottlenecks Facade suffers from, but it’s the one that players often spot first. Given despite over 800 rules in place to recognise statements from players, there are so many nuances within the English language that are still not successfully captured and would require hundreds more rules to successfully grasp. What is interesting that – in what appears to be a stylistic choice by the authors – the allows for a lot of false positives to be accepted, whereby it will incorrectly interpret what the player says to mean something else. As mentioned in a research paper by Matteas and Stern published in 2004, it was more important for the system to make some sort of response, rather than ignore the users input and do nothing at all. In addition, the system fails to grasp any nuance in the text typed and instead focusses on the more literal elements provided.

The developers believed that – when played as intended – the system should still generate a response, even if that response reads the situation wrong or just results in the characters looking confused. It’s argued that one of the benefits of the scenario is that given Grace and Trip are written as upper-middle class, it’s in character that they don’t really understand when people use slang terms or shout obscenties in a positive way. But ultimately, the authors themselves acknowledge that the system is not robust enough to survive players trying to break it. Which has essentially lead to the rather fun and bizzarre YouTube lets plays over the years!

Closing

It’s been almost 15 years since the release of Facade and Stern and Matteas have gone on to numerous other projects. With Stern co-founding Playabl Studios, developing a variety of VR and AR applications for the likes of the NBA, the NFL and Hasbro as well as conversational bots for the likes of Amazon Alexa. Meanwhile Matteas continued his career in academia, becoming a professor first at Georgia Institute of Technology followed by his current position as a professor of Computational Media at the University of California, Santa Cruz since 2006. As a result, there was never a follow-up or sequel to Facade. While Stern and Matteas originally sought to produce a follow-up project called The Party back in 2006, it was confimed back in 2013 that the pair were no longer working on it.

The immensity of the task faced is something that cannot be understated, be it the number of natural language interpreter rules, story beats, JDBs and much more. Not only did the overarching systems have to be developed in order for it to work in the first place, but all of this content was developed by just two people in order to create what is ultimately a 20 minute experience. It’s an interesting hot pot of tools, systems, and languages that has never really been reproduced in the same way. Largely I suspect because of the literal years taken to build it.

But all that said, Facade is an important research and cultural milestone. A game that made left an indelible mark on video games and AI research, creating an experience that is still unique to this day. While it has been cited in academic texts for well over 10 years now, arguably its biggest impact is that people know what it is outside of the academic conference circuit. Achieving Mateas and Stern’s vision of creating a game that sits outside of the norms of the games industry and becoming a piece of art in its own right.

Bibliography