Every few months, Illyriad runs a tournament. These are global challenges, where players can compete together to control territory, gather items, kill enemies, and so on.

But why do we do this? Why not just leave the players to continue on their merry way, building cities, raiding NPCs, and squabbling amongst themselves?

The simple answer is that the players like tournaments. We know this because they tell us so. We know because if we don't run a tournament ourselves, individual Alliances run little mini-tournaments themselves. And we know because the data tells us that they do.

With a successful tournament we can see a measurable improvement in player engagement. And we do measure player engagement, quite carefully.

The data that we gather is a great starting point for analysing whether a tournament worked (i.e. how much the players enjoyed it), and so what sorts of tournaments we should run in the future. And there are lots of different factors that we can measure.

We can measure the numbers of missions sent out by players from their cities, the number of military units that they committed to combat (and how many they lost), the extent to which the bigger players rebuilt or redesigned their cities or landholdings, the volume of Chat (both in the public, Global Chat channel, and within the Alliances' own Chat channels), the numbers of In Game Mails sent, the amount of Prestige (our paid-for virtual currency) bought and used, and so on.

Beyond that, we can also look at more qualitative indicators – good old fashioned “do the players say they had fun?” questions. We'll read through and discuss all the Forum posts, for example, and we can back this up by measuring the incidence of positive or negative keywords in player communications during the tournament.

What Does the Data Say?

Illyriad is still quite a new game, but we have run a range of tournaments, and looking at the way that players received these should give us a clear direction for designing new tournaments.

Unfortunately it is not quite that simple.

For one thing Illyriad is a growing, developing game, so comparing tournaments over a period of 18 months might not be comparing like with like. For example, in 2012 we introduced resource gathering a crafting, which gave players many more goods to trade; and many of those goods have a direct military use. So, if we see a tournament in 2013 triggering a big spike in volume of trade per player, whereas a tournament in late 2011 didn't, that does not necessarily mean that the 2013 tournament is more successfully increasing engagement – it may just mean that purchasing specialist weapons is now an option, and people use that option.

Likewise, the tournaments do not exist in isolation. Other factors are at play. For example, one tournament had notably low engagement, because a big player versus player war broke out one week in: it looked like some Alliances had been holding back from the tournament in anticipation, and when hostilities commenced many other alliances pulled out or scaled back their involvement.

Still, the data is instructive. And we can still see different tournament types being received very differently.

As three examples:

Bloodthirst For Knowledge was a PvE hunting tournament, where NPC groups spawned on the map and players could go out and defeat them to collect dropped items, which scored points for their Alliances.

Undead Hordes was superficially PvE, but with the potential for PvP intervention, where players (divided into leagues based on their size) set up defensive camps to fight off NPC attacks, in order to score individual points.

The Second Anniversary Tournament was an Alliance-based PvP tournament, based around holding squares (in a king-of-the-hill, capture-the-flag style).

Qualitatively, there were lots of positive comments about the Undead Hordes, and it was nice as developers to hear the players congratulating us on our great tournament design. Unfortunately, these vocal fans seem to have been in a minority. Certainly there was an increase in the number of military missions dispatched during the tournament – players got stuck in, militarily. But on every other measure that we have, the tournament actually reduced player engagement with the game. The volumes of Alliance Chat, Global Chat and In Game Mails sent all declined during the tournament – something that we've never seen in any other case. And it wasn't just that players communicated with one and other less during the tournament. As examples, Prestige spend did not increase, and new player referrals seemed to be being held down by the tournament - they rose only after it ended, as if players had been too distracted with their solitary involvement in the tournament to bother inviting their friends to play with them.

By contrast, the Bloodthirst tournament received quite a lot of negative comments, and the data suggests that some people were exhausted by it. It had been quite grinding, and this both garnered criticism and also led to some players disengaging from the game for a while afterwards. But the other data remained highly positive: players spent more Prestige during the tournament and continued to do so afterwards, and the volume of communication rose dramatically. Most interestingly, the tournament really seemed to bring Alliances closer together, making them more cohesive, and the volume of Alliance Chat in the month after the tournament was nearly three times higher than in the month before it.

Meanwhile, the much more straight forward Anniversary Tournament was moderately positive in every single way. It didn't have the huge uplift in Alliance communications that Bloodthirst did, it didn't win us the praise of vocal players as the Undead Hordes did, but every metric was positive. New player referrals, Prestige purchases, In Game Mail, Chat, rebuilds of mature cities, everything.

There is strong evidence that tournaments can increase player engagement, and that the design of the tournament will influence how players behave. These aren't statistical blips. For example, the Anniversary tournament trebled In Game Mails, and increased military participation six-fold on some measures.

Therefore in early 2013, as we sat down to discuss the design of our next scheduled tournament, we had a lot of interesting insight into how past events had run... but we had no single, simple direction suggested by the data of what a tournament should look like.

 

Designing the Tournament

We run tournaments because players enjoy them. But beyond that, what do we, as developers, actually want to achieve?

One possible answer might be that we want to increase revenue - to drive Prestige purchases. Obviously it costs money to keep Illyriad running, and we want the game and the company to grow, so of course we'd like more money. But actually, we never try to build a tournament to drive revenue.

Illyriad does not aggressively monetise players. We only make money if people keep playing, and stay engaged. If the players are enjoying building their cities and tinkering and fighting, then they stay, and we get money; if they aren't happy, they are more likely to leave and less likely to pay. We don't expect to short-circuit that – we don't try to trick players into giving us cash, because that is an experience that they will not enjoy and in the long term everyone loses. We have to keep the focus on giving the players an experience that they will find engaging.

On the other side of this, word of mouth and the referral mechanisms get us most of our new players. So, in theory, we'd love a tournament that, as a by-product, increased referrals. However, none of the mechanics that we've tried dramatically increase referrals, and it's hard to see how we'd create a tournament (without being crass and uninteresting) that would do that.

What tournaments can do, which is vital for us, is to bring people together. Statistically, the players who interact with other players (whether conspiring in Alliances, or sending care-packages to new players, or just chatting) stay in the game longer. And if they stay longer, that means, indirectly, more referrals and more income.

For tournaments to increase cooperative forms of engagement is vital. Cooperative engagement means that the players are having fun, in a way that is very healthy for the game.

This explanation is long-winded, but as a result, we have a very simple objective when we design a tournament:

“Give players challenges that they can enjoy together.”

It is as simple as that.

 

The Tournament Mechanic

Eventually the mechanic that we settled on was a variation of the camp-the-square idea that we'd used in the Anniversary tournament.

Every seven days, NPC armies would emerge from a particular point on the map, and march towards a set of squares. We would not announce the squares, but once the NPCs arrived each square would activate, and thereafter would award the PC occupying force one special unit per day, for the next seven days. The NPCs would march out four times, so that the tournament ran for 28 days. Each special unit was worth one point, and the Alliance with the most points at the end of the tournament would win.

As an extra complication each player's town could only gain one unit. With players each owning 1-10 towns, this meant that no one player could, alone, score points every day of the tournament. The logic of this was to encourage Alliances to involve their smaller members – a small player's city might not produce many soldiers, but it was as capable of scoring a point as a much more powerful city. So, to be truly effective, Alliances' large and small members would have to cooperate. After all, we want to “give players challenges that they can enjoy together,” all of the players, not just the big players.

The decision making process for Alliances should, we expected, be quite complicated. For example, because the target tournament squares changed every seven days, Alliances would repeatedly be tempted to race to occupy squares as swiftly as possible – but in Illyriad the fastest units are often poor defenders, and if killed can take a relatively long time to replace. As another example, if slow moving, strong defenders are occupying a square, should one leave them there until that square expires, even though they might take days to march home, and all but miss the next tournament square? Or should one abandon a square a while before it expires? A lot of this deviousness is down to the movement speeds, attack and defence stats, replacement rates, and availability of special unit equipment, and this provided plenty of scope for the sorts of players who really like to engineer and fine-tune their strategies.

Finally, we awarded prizes not just for a global winner, but also on a province by province basis in the gameworld, with nominal individual prizes to any individual who managed to score. In this way we ensured that this was not just a race between a few huge Alliances. Any established group of players could, as it were, choose their own difficulty level – by choosing whether to compete globally (hard), locally (medium), or just chase individual rewards (easier).

 

Watch, Adjust and Applaud

Having designed the tournament, written the code, generated the art and text, and set the whole thing live, our job, as developers, was not over. Two things remained.

First, the tournament required our ongoing attention. We had to keep an eye on things and, if necessary, respond.

Secondly, we needed to applaud peoples' successes. Illyriad does not fall over itself to pat players on the head – we generally reckon that our players are mature enough to not need constant positive reinforcement from the game, and also social enough that praise is usually given by other players. (We never cease to be impressed by how friendly and supportive our community is.) However, we still need to mark successes – it's a question of having respect for the players.

The second of these was quite straight forward. We set up code to automatically award medals to players as they scored points – and these badges are public awards, visible on the player's profile – with notifications sent to the player and also posted in her Alliance Chat stream so that her fellows would see her success. Then when the whole thing was over, we looked back on the events and wrote a Herald (in game newspaper) piece praising the winners (and runners up, as it happened); this appeared on the screen that players see when they log on, a couple of days after the tournament concluded – it was important that the piece was written discussing the tournament itself and praising people for their genuine achievements, rather than just being some stock piece of copy that we pre-wrote and set to go live at the end.

But remaining on call to handle issues as they arose was more challenging. Not least, we had no idea how much we might have to change during the tournament, so we had to stay constantly vigilant. Obviously we hoped that everything would be fine, the design was solid, the code adequately tested... but a tournament is a big, public event, so we couldn't just walk away from it. And in the end there were two things that caught our attention.

One of the surprises was entirely excellent. Illyriad prides itself on the depth of the strategies that people can evolve, and we love being surprised by our players' abilities to exploit these in interesting ways. So it was that a few days in we started seeing player armies camping not on, but close to the tournament squares.

Had the players miss-typed the target coordinates and sent armies to the wrong place? Not a bit of it! The players were using two properties of Diplomatic units (usually useless in combat) – first that they can be attached to armies, and second that they have a visibility radius around them. In this way, by camping close to a square, they could watch what happened on the square. Without this spy army close by, the square would be under a partial fog of war, with a graphic showing that a camp was present but no indication of how large the army was. With a spy army close by, these players always had an estimate of how large the occupying force was, and so could better plan when or if to attack it.

Even better, we were not the only people to work this out. Opposing players started realising what these lurking armies were for, and we started to see little battles around the squares as people attempted to drive away the spies.

The other surprise was less welcome.

At the start of the first week of the tournament, as NPC armies marched towards the supposedly secret target squares, PC armies started to camp, uncannily, on these destination squares. And all of those armies were from one Alliance. We started to get Petitions raised by players accusing that Alliance of cheating. Ouch.

With some investigation, we discovered what had happened.

We had timed all of the NPC armies to arrive at the same moment, which had been announced to the players, and they travelled at different speeds. We had also set them to move on an arc, so that players could not simply extrapolate their landing square from their departure point and a single location en route. What the devious Alliance had done was to have their members measure the armies' speeds and trajectories over a course of some hours, and had then calculated their eventual end points.

This raised a hard question for us. On one hand, the Alliance's deductions were very clever, and, as they used only data available to everyone, they did not constitute cheating. We love players who throw themselves into the game and do devious things – and this had taken hard work and intelligence. On the other hand, this meant that a single Alliance, simply because they had uncovered one trick, was going to get a big advantage, which seemed unfair on everyone else.

So we took a middling course. We absolutely were not going to punish this initiative and intelligence. But we did change the way that the NPCs marched in subsequent waves, so that their paths could not be calculated by players.

So, Did It Work?

Whatever we do, as developers, some people will praise us. It's lovely to hear, and it's useful to get a better understanding of why people say that they like certain things. This is good information for future use and on a purely human level it is, of course, nice to hear that your work is appreciated.

And what ever we do, a few people will always complain (often loudly, and sometimes entirely disproportionately), and in such cases, if there is a genuine case being made then we need to hear it. A management truism that counts just as much in game design is that people will always rush to give you the good news – what matters is how fast and how accurately you get the bad news!

But this informal feedback can only be part of the picture. Those who are quickest to congratulate or loudest to complain may not be statistically representative.

What we have to look at first and foremost, is the data. And in this case the data was broadly positive. Yes, there was an increase in military sorties, in volumes of Alliance Chat, and so on, and even a small increase in Prestige purchases. But these increases in engagement were uneven – they didn't show up across the game world – and as a result the overall increases were not as strong as we might have hoped.

What we saw was that in some Alliances, engagement barely increased, and this lack of impact was especially strong in the first week of the tournament.

Now the question that we have to ask, is why our increase in engagement was patchy. Was it a flaw in the design? Or was it something external to the tournament?

There is reason to think that events outside the tournament were causing some Alliances to sit this out. A major PvP war was drawing to a close, and some Alliances may have felt too militarily exhausted to throw themselves into a tournament. This is a plausible explanation.

But lets not let ourselves off the hook too easily. If it was simply that some Alliances were exhausted, why was engagement lowest in the first week?

Our suspicion, having read some of the qualitative feedback, is simply that we confused people. By engineering a tournament to allow the most devious players to really indulge themselves, and by tweaking it to encourage cooperation between larger and smaller players, we had simply created a system which was too complicated.

This would be an issue that we have seen before: Illyriad as a whole is extremely “deep” – which is another way to say “complicated” - and people who try to dip in and skim the surface can get bemused by it.

It could be that here we had over-engineered the tournament, that even for more experienced players it was inaccessible. This might explain why engagement picked up as the tournament progressed; perhaps some people found the whole thing hard to understand, and needed a bit of time to work out how to approach it.

Also, it might be that having the tournament squares moving every seven days acted as an impediment for players who had settled remote regions, or who had specialised in slow-moving troops. They simply looked at the travel times for their armies, and decided to hold them back.

Either way, there's a hint here that next time we might want to run a slightly simpler tournament.

Ultimately the tournament was a qualified success. There were no technical glitches, no need to panic. Player engagement measurably increased, so over all the players were having fun, which is good for them and good for the game.

But perhaps just as importantly, we now have even more data to mull over, which will better inform our future design decisions.