Quick Tutorial on how to Set and Run A/B-tests

Why Do You Need Split-Testing  

For those who are unaware of what A/B testing is, I am putting this short explanation here.  Different components of your game can be implemented in different ways. And we, as game designers, are constantly hesitating which way would be the most fitting and effective for a specific setting, audience, or an entire game. Or we are planning to implement new features and don’t know how it will affect our game. Therefore, we split the player to form different independent groups, so we could test different varieties of new component and see how it affects the game.   

Conducting this kind of experiment on real players, we determined what solution works better based purely on KPI metrics.   

Where to Begin

First, you need to determine the list of areas you would like to test. If it is a new project with a lot of new theories to prove, you’ll probably need an expanded split system that would allow you to collect qualitative and representative data.

Areas to A/B-test:

Each of these options is measured by different metrics and the metrics is what is used to determine the effectiveness of every change. Interfaces and Mechanics are measured by the engagement KPI: target actions (their sequence), session duration and quantity, number of target actions per DAU etc. Balance and Economy are measured by financial indicators, like deficit and surplus of resources in your game. Content Progression and Game Goals are measured by the retention-related metrics which include long-term retention, number of sessions and their durations, and by other custom metrics.

In other words, the analytical systems you use in your project should be able to analyze different splits based on different metrics and KPI sets. It is a common mistake to analyze all splits using the same dashboards with the basics metrics. Nevertheless, you should not replace those metrics - it is possible that a certain feature would improve engagement metrics but kill monetization at the same time.  

How to Manage Splits

To properly work with A/B-testing tool, you’ll need clear and comfortable to work with interface, and this is the task for your server developer.

Demographics Split

When users install your game, they are divided into different cohorts, which are further used to conduct a cohort analysis. You can set any custom division setting, like 50/50, 33% of users per a single cohort etc. Also you can set the system to send, for example, 10% of new installs to a specific cohort, if you don’t want to risk critical mass of players by giving them the untested feature.

You might also want to set a targeting. It is possible to set how the split is assigned to a certain user. For example, a test might only affect iPad users, or female from the USA - then you’ll be able to split only specific users and don’t risk the rest.

Next Step

During an install or the first session, new user is assigned with certain settings, configurations, and other parameters. From now on, the user lives with these parameters and sends correspondent data to your analytical server.  

A/B-test Interface

Since you want to have different split options, you will need to create tables of settings for each splitting area and a general management table.

Interface Split Table:

Mechanics Split Table:

Configurations Table:

General Split Table:

The tables that combines all previous tables in one: 

Keep in Mind Before Conducting the Tests:

How to Make Sure Your System Works

Conduct A/A-test - the one that divides the audience in half - and look at the results. If the results are the same, it means your system works properly. If the results are different you should ask yourself, what went wrong.