Quick Tutorial on how to Set and Run A/B-tests

Thought it might seem obvious, most game designers tend to ignore A/B testing. While it is a must if you are launching free-2-play game, since it allows to apply new changes almost painlessly. That's basically the following blog post is about.

Why Do You Need Split-Testing  

For those who are unaware of what A/B testing is, I am putting this short explanation here.  Different components of your game can be implemented in different ways. And we, as game designers, are constantly hesitating which way would be the most fitting and effective for a specific setting, audience, or an entire game. Or we are planning to implement new features and don’t know how it will affect our game. Therefore, we split the player to form different independent groups, so we could test different varieties of new component and see how it affects the game.   

Conducting this kind of experiment on real players, we determined what solution works better based purely on KPI metrics.   

Where to Begin

First, you need to determine the list of areas you would like to test. If it is a new project with a lot of new theories to prove, you’ll probably need an expanded split system that would allow you to collect qualitative and representative data.

Areas to A/B-test:

  • Interfaces
  • Game Mechanics
  • Game Economy (balance)
  • Content Progression
  • New Feature of Content Effect

Each of these options is measured by different metrics and the metrics is what is used to determine the effectiveness of every change. Interfaces and Mechanics are measured by the engagement KPI: target actions (their sequence), session duration and quantity, number of target actions per DAU etc. Balance and Economy are measured by financial indicators, like deficit and surplus of resources in your game. Content Progression and Game Goals are measured by the retention-related metrics which include long-term retention, number of sessions and their durations, and by other custom metrics.

In other words, the analytical systems you use in your project should be able to analyze different splits based on different metrics and KPI sets. It is a common mistake to analyze all splits using the same dashboards with the basics metrics. Nevertheless, you should not replace those metrics - it is possible that a certain feature would improve engagement metrics but kill monetization at the same time.  

How to Manage Splits

To properly work with A/B-testing tool, you’ll need clear and comfortable to work with interface, and this is the task for your server developer.

Demographics Split

When users install your game, they are divided into different cohorts, which are further used to conduct a cohort analysis. You can set any custom division setting, like 50/50, 33% of users per a single cohort etc. Also you can set the system to send, for example, 10% of new installs to a specific cohort, if you don’t want to risk critical mass of players by giving them the untested feature.

You might also want to set a targeting. It is possible to set how the split is assigned to a certain user. For example, a test might only affect iPad users, or female from the USA - then you’ll be able to split only specific users and don’t risk the rest.

Next Step

During an install or the first session, new user is assigned with certain settings, configurations, and other parameters. From now on, the user lives with these parameters and sends correspondent data to your analytical server.  

A/B-test Interface

Since you want to have different split options, you will need to create tables of settings for each splitting area and a general management table.

Interface Split Table:

Mechanics Split Table:

Configurations Table:

General Split Table:

The tables that combines all previous tables in one: 

Keep in Mind Before Conducting the Tests:

  • It is important to determine the number of simultaneous tests based on your audience and capability - do not run more tests than you can analyze.
  • Make sure that your trackers are placed and the data is collected properly.
  • Ask your analysts (if you have one) to test splits and their settings on the internal audience before enabling them globally (on the production server).  

How to Make Sure Your System Works

Conduct A/A-test - the one that divides the audience in half - and look at the results. If the results are the same, it means your system works properly. If the results are different you should ask yourself, what went wrong.  


Latest Jobs

IO Interactive

Hybrid (Malmö, Sweden)
Gameplay Director (Project Fantasy)

Arizona State University

Los Angeles, CA, USA
Assistant Professor of XR Technologies

IO Interactive

Hybrid (Copenhagen, Denmark)
Animation Tech Programmer

Purdue University

West Lafayette, IN, USA
Assistant Professor in Game Design and Development
More Jobs   


Explore the
Advertise with
Follow us

Game Developer Job Board

Game Developer


Explore the

Game Developer Job Board

Browse open positions across the game industry or recruit new talent for your studio

Advertise with

Game Developer

Engage game professionals and drive sales using an array of Game Developer media solutions to meet your objectives.

Learn More
Follow us


Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more