This is an excerpt from the book Level Up: A JRPG Creator's Handbook, available in the current Storybundle for a limited time, about using non-transitive character and monster concepts to design interesting encounters. Level Up is a guide for new and inexperienced JRPG-style game creators, with an emphasis on users of the RPG Maker series of "construction set" tools, and includes chapters on planning out the attack power of foes, scaling your encounters, building your experience system, an examination of the monsters generated in one area in Dragon Quest II, descriptions of scenarios and notable innovations for many classic JRPGs, long lists of crazy ideas for your own game, and quite a bit more.
A time honored strategy for designing games is to take a pre-existing game, learn how it goes about its business of being fun and/or balanced, and seek to bring that into another game. One of the most popular of these prior games is the one known variously as Janken, Roshambo, or Rock-Scissors-Paper.
The part of this game that’s of most use in design is its principle of circularity, that no move is better than the others. This is contrary to the basic idea of using numerical stats, since they are linear, one is always measurably, obviously better than the others, so usually some greater design principle must be brought to bear on it.
In RPGs, one obvious way this design idea can be brought in is in the idea of vulnerabilities. An enemy with a lot of physical attacking power needs a vulnerability, some character class that can defeat it relatively easily even at lower level. So you may decide, when designing your monsters, that enemies that have high attack may have little or no magic resistance.
For more substantial games, you needn’t stick to a three-node arrangement. You can go beyond simple Rock-Scissors-Paper and gain the same benefits.
One game that makes explicit use of this is Pokemon, which expands on this idea with the concept of asymmetric weaknesses. Each of the game’s 17 Pokemon types is strong against, and weak against, multiple other types, in a system where few monsters are universally capable. This idea, that every Pokemon has a moment where it may shine, is ingrained in the culture of the game, to the extent that in the rare cases where the system breaks down it feels unfair. The original games had, at the very end of the game, the fearsome Mewtwo, which was not only of Legendary strength but was also of Psychic type, which was explicitly created to be unbalanced in that version of the game. The result was that Mewtwo was nigh unbeatable in player vs. player battles unless you specifically built a Mewtwo killer, which was usually either another Mewtwo or a Parasect, a Pokemon of the sole type (Bug) against which Psychic types were weak. But even then, Parasect was a normal Pokemon, not a Legendary, so while it had a chance it was not a slam dunk. Later versions took special pains to rein in Psychic types' overbearing advantages.
Another series with strong RPS systems is the Fire Emblem line, which has multiple cycles. The most obvious is its "weapon triangle," where Swords beat Axes, which beat Spears, which beat Swords, and attacking a unit using a weapon to which the target unit's equipped weapon is strong confers substantial bonuses. But its magic system also has a triangle. And its classes also have circular strengths and weakness from just how its battle math works: heavily armored units are very difficult to overcome unless you have an enemy with super-high attack, speed and critical hit chances like Swordsmen; Swordsmen, on the other hand, tend to be lightly armored and not super mobile, making them the prey of mounted units; and mounted units tend not to be able to overcome high armor. But later in each game you find exceptions to all of these rules, weapons that reverse the triangle, characters with atypical stats, and weapons and spells with enough power to overcome the triangle bonuses and penalties, which can give even good players a nasty surprise if they are not prepared.
The thing about R-P-S, however, is keeping in mind your player’s party composition and which monsters they’ll be fighting at different times. Except in certain cases, such as subquests and challenge areas, it’s a bad idea to make a foe a character is weak against totally invulnerable to that character. Depending on your game’s story and/or the player’s choices, the party may be composed entirely of Rocks, so to speak, at particular places. If there’s absolutely no way for a Rock-filled party to defeat a Paper-type foe, then it will really suck if there is a hard-stop Paper boss it must defeat. One way to give such a group an out in these situations is through the use of consumable attack items that can be bought or found that overturn or ignore the standard sequence, although in that case care must be taken that they aren’t so obtainable that the normal cycle can be ignored, and the party could just load up on those and coast through the game.
So anyway, how could a "Rock Paper Scissors" system look like in your game? Well for starters, this kind of system is really an analogue for a game where characters have a circular, non-transitive set of powers. "Circular" here means character A beats character B, which beats character C, which beats character A: relative powers are set up in a circle like that, where each category of character or enemy has those things it's good against and those things it's bad against. "Non-transitive" is a mathematical term: transitivity is a kind of property where, if some directional relationship is true of A and B, and B and C, it is logically true of A and C. The properties of being "greater-than" and "less-than" are like this: 5 is greater than 3, 3 is greater than 1, therefore, 5 is greater than 1. Obvious, right?
To some degree this is obvious, which means it kind of breaks your brain a little bit when it's not true. My favorite example of this is the phenomenon of non-transitive dice, which is one of those mind-warping facts of our universe that can't help but change your perspective of it slightly when you find out about it. Seriously, follow that link and try to understand it.
You could use the mathematics of non-transitive dice to run your combat, but you don't have to go that far. Basically, you can add a Rock Scissors Paper level of non-transitivity by hinging combat on values in a circular kind of way. Here's an example.
Divide characters and monsters in your game into three groups:
Group A has high attack, but low magic defense
Group B has high speed, but middling hit points
Group C has high magic attack, but decent defense and speed
So: Group A's high attack should prevail over Group B in the first round; Group B's high speed should let it beat Group C early; Group C acts last in combat, but its defense should let it survive the first attack from a Group A character and destroy it with a spell.
Now this arrangement isn't perfect, and I feel I should mention this is just a suggestion, and one where the details matter greatly. In fact, on some level, the details are the game! Just like Rock Scissors Paper, as played in a schoolyard, is more of a test of psychology than a game, so this system, if strictly applied, isn't really that interesting. Just as how Pokemon doesn't have a circular arrangement of strengths and weaknesses but a complex web, the nuts and bolts of the combat, the ways in which it doesn't line up exactly into three exactly opposed groups, the asymmetry of the system is what makes it interesting. If one group is, objectively, a little more powerful on average, then it in turn increases the power of its counter. And, of course, as characters and monsters increase in power, eventually you find cases where Group C can beat Group A if its level is high enough, and so on.
What is my point here? Although your combat system is based on numbers and their linear relationships with each other, you should find ways to use different values to upset the expectation that more in one category always equals better. In real life, people are never "greater than" each other, and you can play with the assumption of strict power superiority in your game to make its play more interesting.