Testing for Success: Devising a Mobile QA Strategy

The highly competitive mobile games market grew to $25 Billion in revenue in 2015. The market has matured and the stakes are getting even higher for game developers competing with thousands of developers while contending with rising development costs. One way to manage mobile game development spending is by building a Quality Assurance strategy. The right strategy can minimize resource commitments and reduce the overall product development cycle.  

Every mobile game developer shares at least one critical success factor for their project: cover the largest possible market segment – or – make sure your game is compatible with the most popular devices on the market. Building a smart QA strategy is part and parcel to this objective. 

While Apple has significantly less marketshare than Android, Apple is a good operating system on which to start building your strategy. Why? It’s fairly straightforward from a QA perspective. To date, there have only been 13 models of the iPhone, 6 iPods, 6 iPads, 4 iPad Minis plus the iPad Pro devices. Apple devices total 31. While it’s important to cover the older version of the OS, Apple has been very persuasive when it comes to software updates. For the most part, a majority of iOS users are running the latest OS, which is great news for iOS app developers.

Apple’s 31 devices are nothing compared to the massive number of Android devices. There are dozens of manufacturers of the Android device, each with numerous versions. Samsung has released ten models of the Galaxy in the US alone. Motorola, LG, Nexus, HTC, ASUS, Sony, and others each have numerous product lines that must be taken into consideration. The sheer number of devices on the market combined with the different operating systems available can make testing on Android extremely complex, creating an unnecessary resource burden - resources that can and should be invested in the game experience or marketing. The need for a hardware strategy is an absolute necessity on Android.  

The challenge mobile game developers face today is the same challenge PC developers have been facing for decades: How to test software with the infinite combinations of video cards, sound cards, CPUs, RAM and operating systems. Mobile devices are somewhat easier to deal with because manufacturers have standardized the building process. The CPU and GPU are part of a set package called a System on a Chip (SoC). If a device is advertised as having the Snapdragon 801 SoC, then we know that the CPU is the Krait 400 and the GPU is the Adreno 330.

A well thought out QA strategy should account for all of this information for all relevant mobile devices. Manufacturer, model, OS, kernel, chip family, CPU, processor speeds, cores, GPU, resolution, and memory should all be recorded by device in a QA testing database. This should be kept in a database that is a living tool, evolving with the introduction of new devices. Every new device launch marks an opportunity to compare it to similar devices. For example, the LG Nexus 5X uses the Snapdragon 808 SoC. That’s the same SoC as the LG G4. If budgets and timelines are tight, as they often are for mobile games, testing on one of these devices should mean that the product will run on both (barring significant differences with the device OS).

Any software developer will attest that hardware issues are often the hardest for them to fix. One of the first tasks when a game enters QA should involve installing it on a large assortment of devices as quickly as possible. This is to insure that the game launches on the hardware before more time and money is spent doing deep functionality testing. It all comes back to money and time. Based on the sheer number of devices, it’s simply not practical to install software on every Android device. In order to get as much coverage as possible, the devices tested should be chosen based on a wide range of SoC specifications thereby minimizing overlap. If an issue is discovered in this initial batch of devices, the test can be expanded to encompass more devices using similar hardware. This saves precious time, helping developers hit their target launch marketing windows, which is increasingly critical to success on the App Store and Google Play.  

As an example, a recent, initial hardware device test on a game found that the app failed to launch on the Samsung Galaxy Tab 3 10.1. Further investigation revealed that it wasn’t launching on any devices running the Intel Atom SoC's. The test quickly identified a critical issue with a large subset of hardware on the very first day of testing, saving the developer many, many hours chasing down a bug. That’s what a good mobile QA strategy is all about: allowing developers to focus their resources on what they do best - creating great content. 

Building a well thought-out QA strategy is the best way mobile developers can mitigate some of the resource burden and risk associated with that last, critical phase before bringing a new app to the marketplace. It can mean the difference of success or failure in the highly competitive mobile marketplace.