Daily news, dev blogs, and stories from Game Developer straight to your inbox
Sponsored By
Featured Blog | This community-written post highlights the best of what the game industry has to offer. Read more like it on the Game Developer Blogs.
Knights, Pawns, Bombs and Stars: Evaluating your competition to achieve actionable feedback
How do the features of your game fare against the competition? The "Relative Importance Matrix" is a research tool that helps determine which of your game features are the most important vs. competitor's and in which order to fix potential issues.
10 Min Read
There has been a shift in how many successful games come to market. We used to predict success based on gamerankings and “on-time & on-budget” development.
The new world of “games as services,” persistent games, “always on” games, F2P and all those games that create a readily accessible community of players have been enjoying the successes of marrying consumer research with production. The accessible data makes knowing what consumers want so much easier. So, why care if a game is developed on time and on budget if nobody ends up wanting it?
Traditional game development (like disconnected boxed product) suffered from a lack of consumer research, and when consumer research was done – it almost never yielded “actionable feedback.”
You might be in marketing, or you might be in production, but if you make games and give your consumers any importance whatsoever, you probably are on a constant quest for “actionable feedback.”
“Actionable feedback” is – literally - feedback you can act upon. It’s like when you are looking for a spot to park your car and your passenger tells you, “One there!” before you passed the spot, and not after. Essentially, it’s the useful kind feedback, the one that, given in time, makes your life or your product better.
In the digital space, as of 2013, rivers of data stream from consumers. The industry is enjoying great successes from experimenting on consumers. If there are analytics, we can focus on our own data and see what works. This is truly the great achievement of the videogame industry since its move to more connected experiences. We now know how gamers react to each one of our game designs and marketing decisions, almost in real time.
The emphasis is, however, on one’s own games, one’s own consumers, one’s own data.
Since games are now often a service, we find we invest time in keeping them alive and fighting for an ongoing amount of consumer’s time (and wallets). This is where we can take a lesson from the over-cited Sun Tzu, and figure out that knowing our enemies is good tactics. To help in this, we can use our community to develop a Knights, Pawns, Bombs and Stars matrix.
Getting your Data
Most developers and publishers constantly monitor their own data. This is great! But fewer constantly monitor their competition.
There is however a quite simple way of constantly assessing your game’s performance versus competitors: the “Relative Performance matrix.”
In its simplest form, the steps a team has to go through when creating this kind of tool are:
1) Determine what are the game’s key features:
First the team should sit down and brainstorm. The question the developer needs to answer is “what are the game characteristics/features consumers look at when playing my kind of game?”
Then the same question can be asked to consumers. These are open questions: ”what are the game characteristics/features *you* look at when playing my game?”
2) Create a first survey (with multiple but limited choices):
With what brainstormed and with the consumers responses, the developer should create a survey for customers to rate the importance of each core feature that has been defined. Customers should be able to attribute a score (normally from 1 to 7) to each item.
Also ask consumers what other competing games they play.
3) Examine the raw data obtained from the first survey and rank the game features by consumer segment.
4) Create a second survey:
Use the “top ten” most important features that have emerged from survey #1 and have consumers rate the games performance on them.
Have consumers also rate the top ten features of the major competitors (as defined by the consumers under point 2b).
5) Analyze the data obtained from survey #2 and create a “relative performance matrix” (see below). This exercise should be repeated for every segment of consumers the team is interested in studying and for every competing game that shall be evaluated.
Traditionally, this was done with paper surveys. Remember going to a restaurant and finding a scorecard? To incentivize consumers to fill out the form, small gifts were promised. Like having their name entered in a raffle. I need not say that today, delivering virtual prizes in exchange for valuable feedback is a lot simpler (and can go a long way in creating goodwill).
Developers should use their community. They probably already know what features consumers find important in their game, and which features they prefer. It’s just an extra step to see how they shape up against the competition.
Developers of persistent games (F2P games, MMOs, MUDs) also have a great advantage in creating and implementing this kind of surveys. The first resource are beta testers. They will be more than willing to fill out online surveys on the game they are “testing.” The second resource is, once the game has launched, its online community.
Feature Importance vs. Relative Importance
For simplicity let’s consider we are making an RPG called “MyGame.” The developer of MyGame sends out his questionnaire and has the top features scored for importance. Then he asks the consumers to rate how MyGame does against his arch-enemy game, the dreaded/admired “Competitor.”
Remember:
Survey #1 Asks which are the most important game features and what competitive games do the consumers play.
Output #1: From Survey #1
We build a DB of features scored by importance (from 1 to 7, with 7 being high importance)
We build a DB of our top competitors.
Survey #2 asks consumers to score how your game scores against your top competitors on the top features (features with >4 importance) determined using Survey #1.
Once the data is collected, the results for the most important features (>4) should be plotted in a table as shown below.
Relative Performance Matrix Chart
From this data we can plot out a matrix that shows the importance of each feature against the relative performance of MyGame on that feature vs. Competitor.
Relative Performance Map
The score on the Y axis is the rating your segment gives to a game characteristic/feature. The value on the X axis is the relative performance. It is obtained by subtracting the Competitor’s performance score on a given feature from MyGame’s score.
By examining point A, “franchise/brand appeal,” one can understand that:
1) it qualifies as “low importance,” with a score of 4.7. This yields the consumers point of view on the feature/characteristic.
2) MyGame, even if it possesses a good brand name due to its hypothetical success, suffers the brand strength that Competitor possesses (maybe its a long lasting franchise). Thus, MyGame’s relative performance is a -0.2 (= 6 – 6.2).
On the other hand, all the rest of the dimensions compared see MyGame as a winner. Most importantly, MyGame scores high ratings on very important RPG features such as gameplay, level of customization and number of classes/races available. Do not be misled, the performance is only “relative.” If we plot MyGame against yet another game, imagine a “Competitor2,” we could find out that Competitor2 largely outperforms MyGame both on customization and number of classes/races available. On the other hand, MyGame’s strong brand name could result in the “franchise appeal” dot in the lower right quadrant.
How to turn this feedback into “Actionable Feedback”
Once the developer knows his game’s strengths and weaknesses and how consumers rate them, he will be able to define all items as one of the following:
1) Stars: High relative performance, High importance
2) Pawns: High relative performance, Low importance
3) Time bombs: Low relative performance, High importance
4) Knights: Low relative performance, Low importance
Knights, Pawns, Bombs and Stars Matrix
Stars are features where your game is better than competition and that customers find really important. They define features where the game is on the cutting edge respect to its competitor. In a perfect world all of the items evaluated should be clustered in this zone. Items in this area should be “maintained” in order to keep the competitive advantage. Developers should also keep in mind that it’s probable that at some point a competitor with a great hit will kick their star to the left.
Time bombs are features that customers find really important but where your game is worse than competition. They are immediate problems, they are about the blow up and might have already done some damage. This is where competitors are giving your game a hard time, especially because consumers constantly notice the difference between the product and the competitors’ product (the latter being better). Time bombs are high priority and are the first situation that should be assessed and defused. Think of Time Bomb’s as a Def Con situation, the higher the number of items in this area, the higher is the Def Con number (try not to get to DefCon 5). The objective is to fix them so that they shift laterally into your star quadrant.
Pawns are features where you perform really well but to which consumers attach low importance. They should be assessed only after a developer has successfully defused all time bombs. The problem with pawns is that even if one’s product performs well with them, consumers don’t seem to appreciate them enough. The main task here is to convince consumers how these features, as a matter of fact, are important. Community building and marketing efforts should be the main course of action.
Knights have last priority. They are defined as knights since they need a “non linear tactic” in order to be assessed (the classic chess piece L jump). Knights show some problems both on performance and in perceived importance. The first task is to improve their performance, shifting them into the pawn quadrant. At that point one should work on making consumers understand how much the knight feature is important (see the Pawns paragraph).
The existence of Knights begs the question: “Why aren’t we just killing the Pawns and Knights features? Aren’t they low importance?”
The answer is no. Any feature that qualifies in the low importance quadrant should still be deemed as important. Since the matrix is built based on consumers’ second survey, thus using only the feature they deemed as most important, these features should only be considered less important. Or even just the least important of the most important. Hence, they are probably too important to kill.
To summarize, efforts should be focused in the following way.
1) High Priority – Fix the Bombs: they leak consumers.
2) High Priority - Maintain your Stars: you don’t want to lose your competitive edge.
3) Secondary Priority – Evangelize your Pawns: you want players to understand why this feature is so great.
4) Low Priority – Fix then Evangelize: don’t make the mistake of marketing a feature before fixing it.
Prioritizing actionable items can help establish and maintain a market in a world of constant change. Social gaming, online communities, lowered barriers to entry and analytics have evolved the way games are made and increased potential competition. It just takes an extra step to use all the tools that such connected experiences offer in order to evaluate, not only the game you are making, but also the competitive landscape.
Read more about:
Featured BlogsAbout the Author(s)
You May Also Like
Latest News
Trending
Featured Blogs
