Fail states in video games are something that no player wants to see happen, and yet they are required in order to create tension and allow the player to feel like they've won. Figuring out how much to punish the player for failing is tough, and can mean the difference between giving them the push to rise up or pushing them away permanently.
Catching Flies with Vinegar:
Good game design is all about rewarding the player and motivating them to keep playing. However, just as the player can win at a video game or level, they can also lose. In the old days, the fail state for a video game was harsh with the player losing all progress. This was partly due to the metrics-driven game design of the arcade industry.
As games evolved and became longer experiences, wiping the player's progress was no longer viable. Over the years, different game philosophies have adopted different thoughts on failure.
Rogue-like Design is all about the threat of losing all progress, while most games adopt a checkpoint system for preserving it. We also have games that focus on a story and downplay or outright remove all fail states. There is a discussion to be had about whether or not games without fail states can still be considered video games, but that will have to be for another time.
For the rest of this post, we're going to assume that the game you're designing will have a fail state to it. Balancing failure is not the same as balancing success, and requires different considerations.
Degrees of Failure:
When we talk about fail states in video games, the main point has to do with the severity. As we've talked about before, the player should always feel like they're making forward momentum in some way when playing. Getting stuck or losing progress is an easy way to push them away from your game.
With that said, you still need to have a fail state in order to challenge them. The big question about the fail state is: What's lost and can it be recovered? In rogue-likes, again, you lose everything when you die. For games aimed at more casual players, failing will usually mean repeating the section or level again.
Short-term progress is often the most popular fail state, because it's severe enough to impact the game, but not so damning that it ruins the play. Even the Souls series despite their difficulty have very lax fail states compared to other games. Dying in a Souls game will reset the area and you'll drop your souls (exp/currency) where you died. As long as you get back to your grave, you'll regain everything back.
For boss fights, the fail state doesn't matter all that much, as you can keep retrying the fight with very little progress lost. One of the downsides of Bloodborne was the fact that you lost any bullets and blood vials (healing potions) and had to buy them again.
The danger of a strong fail state comes in when it creates a feedback loop of failure. One of the ways that MMOs tried to create a fail state was by punishing the player with stat debuffs until they recovered their corpse. The problem was it created a self fulfilling prophecy of you dying, being weakened, and then being killed easier on subsequent attempts.
Avoiding the feedback loop is very important; especially in games with fixed progression. In XCOM Enemy Unknown, having a squad wipe in the first two months of play was a death sentence for your run on the higher difficulties. The reason was that you needed the promoted units for their skills, the resources to build, and for unlocking larger squad sizes to survive. By the time you would be able to field another group, the enemies would have outpaced you.
In XCOM 2, while that is still important, there are other means of progressing and ways to get around a bad mission. The latest expansion even introduced hero classes that you'll start with one unlocked to give you a bump at the start.
Unless you are going for a full-wipe as in rogue-like design, you want to avoid long-term fail states for most video games. The player should feel the sting of failing, but not have that impact the next several hours of playing. This becomes all the more important if your game is built around consuming resources for progress.
If the player must spend X amount of time to grind in order to even attempt something, forcing them to repeat the process is akin to wiping the time spent from the game.
Fail states in video games are the other side of the coin when it comes to designing your title. It's important to motivate the player to get better, and not just push them into the ground.
For our last point it's important to point out the obvious: The more demanding or involved your game is, the less punishing your fail state should be. Having the threat of a full wipe on a rogue-like designed around one hour playthroughs is fine; having it on a 50 hour RPG is a different story. The only exception would be having an "ironman" mode, but that is something the player has to opt in to experience.
And as a question to you: Can you think of examples of games with good fail states that kept you coming back, or games with bad ones that stopped you from playing?