For over two months a group of over 200 players from all over the world tried out early development builds of Earthcore: Shattered Elements collectible card game for iOS. We have learned a lot from our beta testers and I thought it would be useful to share our experience and findings with other devs.

With the help of people who joined our Closed Beta program we identified quite a few gameplay issues and managed to solve them during development. We also decided to optimize art, user interface and user flow at the same time. We managed it without stopping the work on other features and bugs, all of it planned in our development schedule. We also kept updating the game regularly both for the beta testers and Soft Launch players in Canada and the Netherlands (in fact, the local SL started at the same time as the global Closed Beta). It took a lot of effort and sacrifice from our team but the result is a great payoff.

I hope the following findings will be useful for other indie developers, who just like Tequila Games, are thinking about distributing their game through TestFlight Beta Testing before it’s officially released.

Recruitment and selection

We decided to recruit beta testers worldwide, we also asked for help a number of online magazines and blogs. For that we prepared a signup form with MailChimp and posted it on a special landing page.

The exclusiveness of a beta program is enough to get the attention of editors, especially if they can offer their readers a guaranteed access. We only gave this right to the websites with an audience similar to our target - tabletop and digital card gamers. We also advertised the beta program with a banner on Touch Arcade’s forum, which brought us some additional core players.

We had to reject people with incompatible devices and select only those who possessed iPads 3 or newer, with iPad 2 and iPhone not supported back then. This left us with a beta tester base of around 360 people from around the world.

Click here to open the full Player Profile infographics

Those people received an invitation to download the game through TestFlight Beta Testing. We also provided them with a short document with easy-to-follow instructions on how to install and how to play sent through MailChimp. Later we used MailChimp for sending newsletters with updates and links to online surveys.

According to our first survey, our final group of beta testers consisted of a mix of casual players, core mobile gamers, fans of tabletop games as well as hardcore PC and console players who don’t play a lot on mobile devices. Age-wise, we had a big cross-section with the youngest player at 12 years old while the oldest ones were in their late 40’s. We managed to keep a broad variety of geographical locations and nationalities as well. It allowed us to gather a lot of varied data which is very important for a free-to-play title like Earthcore.

Gathering feedback and findings

During the closed Beta program we used two sets of tools for gathering feedback, which gave us two different sets of information.

The first tool was our in-game analytics software, running in the background. This is the same software we’ve integrated in the versions soft-launched in Canada and the Netherlands and that we’ll use in the final version of the game. This middleware allows us to track players’ progression and see how they react to with certain elements of the game. It also allowed us to compare our selected group of testers with regular players who downloaded the Soft Launch version.

One of our objectives was to understand which of the single-player missions were too hard or too frustrating - failing them could cause in players quitting the app altogether. We cross-referenced the information with our technical logs from TestFlight reporting that included all the crashes happening due to excessive memory or power usage - especially on lower-end devices. We used this raw data to work on gameplay progress, balancing and optimization issues.

The second set of tools we used to gather feedback included online surveys, forum posts and all other channels for our testers to send us direct messages. We were very pleased to find out that most of our beta testers enjoyed our combat system, which is based on a three-element mechanics, and the strategic depth of Earthcore.

We started with something easy to understand (just to remind you: the mechanics of Earthcore operates on the Rock-Paper-Scissors game but with elements) and built on top of it is a game that offers a lot of tactical possibilities via the Card Crafting system and a large variety of skill combinations. At one point we were afraid it will be too difficult even for core players. However, our beta testers enjoyed (at least most of them) the rather high difficulty level of the single player campaign.

If we were to brag a little, we would quote testers that “loved” our “gorgeous” visuals. This feedback ratified the choices we made in terms of card illustrations, animations as well as overall style.

Top findings

As it is, the most valuable lessons come always from constructive criticism: you have to embrace negative feedback and listen to the ideas for improvements!

During the Closed Beta we faced a high dropout rate at the beginning of the game. Only about 30% of players (sometime even less!) finished the tutorial mission and advanced in the game. Thanks to the suggestions from our beta testers we decided not only to rebuild the tutorial missions in the game, but also to enhance them with some additional short (and skippable) videos about basic rules and tactics: what to look for on the game table, what to take into account when choosing your next card, and so on. This way we could explain our gameplay mechanics in more details and squeeze more fun from strategic choices.

Right now, after the Closed Beta, our Soft Launch version in Canada and the Netherlands have over 80% completion rate on the tutorial. This means we’re getting better and better at retaining players for longer than the first 15-20 minutes required to finish this stage of gameplay. We believe that this first experience is also crucial for further player retention, as it’s when you build a connection with the gamer. Our experience and in-game data from the last few months proves this approach right.

After our Closed Beta we also worked on balancing the challenges in the adventure mode (not making them easier to win, but showing players more skills and combos). We also improved the rewards for winning a battle, giving gamers better cards for completing missions, especially when they achieve three stars (maximum) rating. On top of that, we improved the booster packs - increasing both the number of cards and their rarity.

Another important feature that was requested by our amazing testers was a mulligan for the first draw of cards - from your deck to your hand. Players wanted to have more control over the cards they start their battle with, and we felt it was a fair request.

The experience with the new TestFlight Beta Testing system has been a little painful at times: receiving and activating the download link for an early version of the game is still a rather complicated process for players. Even some more experienced game journalists didn’t know how to avoid a redirection loop that happens if you open your emails with anything but iMail and use other web browsers than Safari. We had a dropout rate of 40% caused by this process, which was surprising. As said before, it’s a new tool and as they are working on it, we expect this issue will be resolved soon.

Slightly less surprising was the number of testers who didn’t provide us with any active feedback, less than half of the people who actually played the game didn’t bother to fill the feedback forms or send us a message. Good we at least had our analytics for them.

Conclusions

The list of feedback, bugs and suggestions for improvements that we received from the testers was of course far longer than what we actually implemented. Having a large beta group though, gives you a sample that’s big enough to understand what the majority wants and what are the issues that you absolutely have to fix.

Overall, our Closed Beta Program for Earthcore CCG helped us to identify new issues and confirm the list of changes that we already discussed within the team but needed a fresh perspective for. It was a very helpful and illuminating experience which we can recommend to all indie developers that want/need to understand their players.

It’s always good to ask what do players expect from you and your game (and hopefully confirm your initial assumptions), but one needs to be ready for surprises.

With only a few weeks before Earthcore: Shattered Elements launches globally on the App Store, me and my team really can’t wait to share it with you in its improved form!

Earthcore: Shattered Elements is currently available in Soft Launch on iOS. For updates about the game and the global release visit the official website www.earthcoregame.com, or follow the game on Facebook and Twitter.