Allow me to start by saying that I mean no disrespect to Dialogue Trees. I don't think dialogue trees can or should be replaced. They are an art form of their own -- a perfect compromise among the competing forces game developers require from such a system: Easy to predict, to implement, to hook into other systems. Enough choice to give the player a sense of free will, without having to account for every contingency. The first two of my radical proposals below are essentially innovations on the dialogue tree.
There have been variations on the dialogue tree in the past, of course. Two pre-existing innovations that come to mind are Time Limits and Interruptions. In a way, you can think of both as interruptions, where either you're interrupting the game, or the game is interrupting you. These days the freshest examples of dialogue trees with time limits are found in the Telltale games. A recent game where you can interrupt the game is Westerado (in this case you interrupt with bullets).
With that out of the way, let's dive into these radical ideas for dialogue systems.
1. Symmetric Dialogue
As we know from Richard Garfield's book Characteristics of Games, a symmetric game is one where all players (or computer players) have all the same information. For most dialogue trees, the process of choosing what to say is invisible to the NPC you're speaking to. You can think about the process of choosing an option as similar to the normal cognitive process of deciding what to say, so it stands to reason that your interlocutor wouldn't have much information about the options NOT chosen.
But imagine if the NPC could see all the possibilities you might say, as well as the one you actually chose. Imagine if NPCs had visible options, too, which you could WATCH them choose from among. What if you could see the look on their face as they read and considered each choice? What if they could be offended at the choice you made, given the other options?
"Why didn't you pick the one where you tell me I'm cool? Do you not think I'm cool?"
Sure, this seems very artificial, but games always make tradeoffs between artificiality and realism. There's certainly room for games that are metatextual and self-aware enough to support this kind of dialogue system.
2. Trees Within Trees
The idea of dialogue trees is that with imperfect information, the player uses choices to Build a conversation. It's like doing a binary search (or tertiary, quaternary, etc. depending on how many choices there are at each step) through a conversation to find the story they want.
Well, the idea of Trees within trees is that the player uses choices to Build a single utterance. Here, the player will search through phrases to find the thing they want to say. Before they have 'said' anything, the player is already engaged with the process of composing a remark. With variable length phrases, it allows the writers to fall back on simple dialogue trees when necessary. Also, if you want to be extra kind to the player, allow them to backtrack and explore all the avenues before they submit their utterance.
I admit that this is hard to write for, since what this amounts to is many more options for the player and thus many more possibilities that will have to be taken into account, but surely clever designers such as yourselves will find ways to mitigate the fluffiness of the possibility space by making choices which are essentially equivalent, making them later converge, or loop back on themselves, etc.
3. Magnetic Poetry
This is almost a halfway step between a dialogue tree and a free text parser. The designer controls the player's vocabulary -- and potentially the length of the overall comment -- but the player chooses the permutation they will put the given words in. This causes combinatoric explosion problems, in that the player can place the words in any order and the number of possibilities can be enormous for relatively few words.
Luckily, English grammar is pretty restrictive (not as true of other languages like Russian), so the number of possibilities which actually make sense will be relatively small. With a little skill, you can make some of the possibilities more-or-less semantically equivalent, and suddenly the combinatoric explosion looks more like a combinatoric poof.
4. Free Text Parsing of an Imprecise Language
OK, hear me out.... Emoji. By 'Imprecise Language' I mean emoji, or something similar. The problem with free text input parsers of English or another Natural Language is that there will inevitably be some unanticipated inputs which are clearly comprehensible to any literate human, and yet completely baffle the system. This breaks the expectation that the parser is working somewhat like a person reading chat messages.
The beauty of giving the player complete freedom within an imprecise language like Emoji is that it's perfectly rational that the other characters in this interactive story might not know what they're saying. Heck, another human player might not know what you mean by a string of emoji.
This reminds me of the principle I've heard many times regarding enemy AI in games -- it helps to set expectations low by making the enemies something the player intuitively knows are stupid, like Orcs, Zombies, or primitive-looking Robots. Well, this is that principle in reverse. Set the NPCs' expectations of the player low -- players must respond to clear language in opaque emoji -- and the NPC's failure to interpret the player's statements will be understood as the player's own failure to express themselves clearly.
5. Timing-based Choices
This is basically a cross between the dialogue tree and Guitar Hero. Imagine that as someone is speaking to you, various options for what you might say come floating down the screen. You don't need to select every possibility you're offered, but the NPCs will become exasperated with too many interruptions and frustrated by too many awkward silences, so it's best to try to find something near the natural end of the NPC's utterance.
If you select a declaration with the correct button press at the correct time, then that's what your avatar will say. If you press at not quite the right time, perhaps one of the longer words will be mispronounced. Too long a pause after the NPC stops speaking and they may walk away entirely.
I like the way this one conveys not just the content of a conversation, but the rhythm of talking to another person we all understand from real life -- an aspect most other conversation systems ignore. It also allows us to throw in some devil-on-the-shoulder dialogue choices which might get chosen more often than in a normal dialogue tree, just due to the player's desperation to hit the right timing.
There are lots of new things that can be done with dialogue systems, and I hope I've inspired you to think a bit further outside the box when it comes to how dialogue could be represented in your game. I haven't had the opportunity to use any of the ideas above, yet, and I honestly hope that someone beats me to it. The field is wide open, and we've only scratched the surface.
Of course, if you want advice or help implementing an unconventional dialogue system please feel free to reach out.
Rob Lockhart is Creative Director of Important Little Games and a Senior Designer at Phosphor Game Studios. You can follow him on twitter.