informa
4 MIN READ
Featured Blog

How we A/B test game art and icons

A/B testing is often used for economy or tutorial tuning in freemium games. Yet it can be also effective for art, from icons through to game content itself. This article discusses the various ways Frenzoo tests art as part of its design and development.

When it comes to free to play design, our studio starts with creative freedom and then decides through quantitative feedback.

In practice what does that mean?   

Lots of experiments and A/B tests.  Not just for features, UI and economy - but also game art.  Here are some of the tactics we use:

App Icon 

An app icon is always lowest hanging fruit when it comes to increasing downloads.

We use Store Listing Experiments, built into the Google Play Developer Console for this purpose.

An example with one of our 3d simulation games:

image

Here we tested different icon compositions trying to “beat” the prior champion.  One, two or no avatars? Elements from the theme, the gameplay, or just focus on characters? 

After a couple of days of data we found a single avatar and gameplay background performed best, outperforming the original icon with a high degree of confidence (it’s handy to be able to see the expected performance interval directly in the results)

image

In one of our older dressup games we found an icon with a “popstar feel” improved install performance over 10% relative to the original - more than we would have guessed.

Apart from icons, we also use Google Play Store Listing Experiments to A/B test other aspects - screenshots, description etc.  A handy tool!

Testing Game Art

We also test game art in two main phases of development:

  1. First to experiment with concepts and see what works
  2. Then with produced art to guide balancing and content distribution

Testing Concepts

Early in the design cycle our designers come up with art concepts and we get feedback on style / subject matter.

For example, in a farm prototype we wanted to add more crafting, and tested to see what players preferred to make.  In this case there was a strong preference for cakes over other types of food and drinks:

image

We can view and filter the responses by different attributes - pay/free user, level etc

image

In another example we wanted to see what type of items players would want to sell next in a boutique simulation game:

image

We use the polljoy plugin for Unity and web dashboard for in-app image and text surveys.

The feeling of spamming is always a concern with popups so typically we:

  • configure surveys to be opt-in
  • don’t ask players at low levels
  • only gather minimum responses needed to gain statistical confidence 
  • configure research polls to give virtual currency as thanks 
  • upload custom graphics via the dashboard to make the poll feel native

Other times we want to ask questions of people outside of our apps' current audience.  

For this we use Google Consumer Surveys which, while not free, can be helpful especially when exploring a new audience type.  You can setup qualifying questions, for example - do you play social casino games - and only survey those who meet your criteria.

Apart from surveys we also use Facebook groups with hardcore fans for more traditional feedback and discussions:

image

Testing Produced Art

Once we have art created, we then poll the finished items.  For example shoes in a fashion game:

image

Why?  Mainly as input for balancing and content distribution.  

For example an excellent first impression in NUX flow helps next day retention, so we plan for some of the highest rated items to appear up front.  

We also want to make sure premium bundles and items have the most desirable content to improve conversion etc.

image

Source - genchev blog

Just like the level of excitement tends to cycle up and down through a movie or game, I think a similar approach works when it comes to arranging content in a mobile 3d game.  

A/B Testing Builds

Once everything is ready, we'll often A/B test the new build itself using Google Play's staged rollout capabilities.  Using this, we can decide how much of install traffic will be on a new version or the control (existing one).  

If it's a major art or feature change (for example a recent Christmas themed update during the holidays), we usually roll out 20% of our traffic and monitor over a week or more. This allows us to check at least the early retention and monetization metrics before we commit to roll it out 100%. 

Conclusion

Being able to test art has had a number of benefits for our studio.  

First it lets us try lots of ideas both mild and wild, at the concept phase which can open up more possibilities than if we just set out with a plan A.

Testing also lets us have confidence before we commit to production and launch, where the costs are vastly greater.

Most of all this feedback loop helps us get closer to the most important stakeholder of all - our players!

image

_____
ps - if you want to get in touch - email is [email protected] This article is also cross posted at my game design blog, iteratingfun.com 

Latest Jobs

Xbox Game Studios

Redmond, Washington
10.5.22
Technical Lighting Artist

Innogames

Hamburg, Germany
10.5.22
Game Designer - Elvenar

Six Foot

Houston, TX
10.3.22
Six Foot Director, Player Relations

Hometopia Inc.

Remote
10.7.22
Lead Engineer
More Jobs   

CONNECT WITH US

Explore the
Subscribe to
Follow us

Game Developer Job Board

Game Developer Newsletter

@gamedevdotcom

Explore the

Game Developer Job Board

Browse open positions across the game industry or recruit new talent for your studio

Browse
Subscribe to

Game Developer Newsletter

Get daily Game Developer top stories every morning straight into your inbox

Subscribe
Follow us

@gamedevdotcom

Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more