Daily news, dev blogs, and stories from Game Developer straight to your inbox
Featured Blog | This community-written post highlights the best of what the game industry has to offer. Read more like it on the Game Developer Blogs.
How we A/B test game art and icons
A/B testing is often used for economy or tutorial tuning in freemium games. Yet it can be also effective for art, from icons through to game content itself. This article discusses the various ways Frenzoo tests art as part of its design and development.
February 17, 2016
5 Min Read
When it comes to free to play design, our studio starts with creative freedom and then decides through quantitative feedback.
In practice what does that mean?
Lots of experiments and A/B tests. Not just for features, UI and economy - but also game art. Here are some of the tactics we use:
An app icon is always lowest hanging fruit when it comes to increasing downloads.
We use Store Listing Experiments, built into the Google Play Developer Console for this purpose.
An example with one of our 3d simulation games:
Here we tested different icon compositions trying to “beat” the prior champion. One, two or no avatars? Elements from the theme, the gameplay, or just focus on characters?
After a couple of days of data we found a single avatar and gameplay background performed best, outperforming the original icon with a high degree of confidence (it’s handy to be able to see the expected performance interval directly in the results)
In one of our older dressup games we found an icon with a “popstar feel” improved install performance over 10% relative to the original - more than we would have guessed.
Apart from icons, we also use Google Play Store Listing Experiments to A/B test other aspects - screenshots, description etc. A handy tool!
Testing Game Art
We also test game art in two main phases of development:
First to experiment with concepts and see what works
Then with produced art to guide balancing and content distribution
Early in the design cycle our designers come up with art concepts and we get feedback on style / subject matter.
For example, in a farm prototype we wanted to add more crafting, and tested to see what players preferred to make. In this case there was a strong preference for cakes over other types of food and drinks:
In another example we wanted to see what type of items players would want to sell next in a boutique simulation game:
We use the polljoy plugin for Unity and web dashboard for in-app image and text surveys.
The feeling of spamming is always a concern with popups so typically we:
configure surveys to be opt-in
don’t ask players at low levels
only gather minimum responses needed to gain statistical confidence
configure research polls to give virtual currency as thanks
upload custom graphics via the dashboard to make the poll feel native
Other times we want to ask questions of people outside of our apps' current audience.
For this we use Google Consumer Surveys which, while not free, can be helpful especially when exploring a new audience type. You can setup qualifying questions, for example - do you play social casino games - and only survey those who meet your criteria.
Apart from surveys we also use Facebook groups with hardcore fans for more traditional feedback and discussions:
Testing Produced Art
Once we have art created, we then poll the finished items. For example shoes in a fashion game:
Why? Mainly as input for balancing and content distribution.
For example an excellent first impression in NUX flow helps next day retention, so we plan for some of the highest rated items to appear up front.
We also want to make sure premium bundles and items have the most desirable content to improve conversion etc.
Source - genchev blog
Just like the level of excitement tends to cycle up and down through a movie or game, I think a similar approach works when it comes to arranging content in a mobile 3d game.
A/B Testing Builds
Once everything is ready, we'll often A/B test the new build itself using Google Play's staged rollout capabilities. Using this, we can decide how much of install traffic will be on a new version or the control (existing one).
If it's a major art or feature change (for example a recent Christmas themed update during the holidays), we usually roll out 20% of our traffic and monitor over a week or more. This allows us to check at least the early retention and monetization metrics before we commit to roll it out 100%.
Being able to test art has had a number of benefits for our studio.
First it lets us try lots of ideas both mild and wild, at the concept phase which can open up more possibilities than if we just set out with a plan A.
Testing also lets us have confidence before we commit to production and launch, where the costs are vastly greater.
Most of all this feedback loop helps us get closer to the most important stakeholder of all - our players!
Read more about:Featured Blogs
You May Also Like
Accessibility and fancy footwork with GLYDR's John Warren - Game Developer Podcast ep. 40Feb 28, 2024
Exploring the 2024 State of the Game Industry report - Game Developer Podcast ep. 39Feb 2, 2024
Phantom inspiration and the ethical auteur with Xalavier Nelson Jr.Dec 8, 2023
Designing Killer Queen: from playground experiment to modern arcade sensationOct 18, 2023
Get daily news, dev blogs, and stories from Game Developer straight to your inbox
Subscribe to Game Developer Newsletters to stay caught up with the latest news, design insights, marketing tips, and more