Testing Games with AI Bots

An honest “help wanted” ad for today’s quality assurance (QA) professional would include phrases like, willing to be on-call 24/7, must love repetition, must have a perfect memory, must have multiple heads and arms to flawlessly accomplish a myriad of tasks daily.

Everyone knows QA testing is a crucial step in the game development process because it ensures the deployment of a high-quality game experience free of inconsistencies, glitches or bugs. To get there, QA is responsible for planning tests, risk analysis, smoke tests, exploratory tests, bug triage, regression tests, testing every possible combination…and then analyzing the test results and filing bugs with reproduction steps.

The quest for adequate automation is understandable and completely necessary. A key element of this is unit testing, where the smallest testable parts of an application are individually scrutinized to determine whether they are functioning properly. Software developers and QA specialists carry out various automated tests, including unit tests, during the development process. Although some of these tests could be performed manually, they are often automated to save time. However, it is important to note that automated tests are not a silver bullet. If the current solutions for automated testing actually solved QA’s problems, there wouldn’t be the level of energy devoted to AI in testing that we’re seeing today.

QA testing is a job that requires a high degree of imagination, creative analysis, inventiveness and strategic thinking. In other words, it needs human creativity and intelligence to be successful. Unlike yesterday’s automation tools that made incremental advancements, artificial intelligence, particularly AI bots, are rapidly evolving game testing for developers and QA professionals because it is finally fulfilling the promise of freeing up their time to focus on what matters most. 

Exploratory Testing with Bots to Uncover the Bugs

AI-driven exploratory testing is one of the most exciting and truly transformative advancements the QA field has seen in recent years. By leveraging AI bots that know how to explore the entirety of the game, large amounts of game content can be tested with just a few clicks. Exploratory bots go anywhere and everywhere a player can go. This advanced technology enables QA professionals to allocate their time more effectively and focus on other essential tasks, ultimately streamlining the overall game development process.

This provides an expansive and incredibly powerful approach to uncover bugs within a game. By deploying multiple exploratory bots, each autonomously navigating the map, they can find even the tiniest bugs and flaws by walking, running, jumping, climbing and crawling anywhere and everywhere the character could possibly go (and sometimes even ending up places where a character shouldn’t go, revealing bugs with the level layout). A horde of QA bots can identify bugs swiftly, potentially saving days, weeks or much more than by manual and other methods of testing.

Exploratory testing starts out by deploying the bot or bots in a secure pre-defined position within the level. They are then instructed to explore the game environment through exploratory movements. The bot resets exploration upon meeting specific goals or criteria, such as falling out of the level or encountering anomalies. A bot can provide a heat map of the areas it has explored. Multiple bots can play at a speed that is many, many times faster than a human player would. All the bots are reading from and adding to a heatmap, which delivers quick results. Not only does this lighten the load for QA because they don’t have to visit every coordinate in the map, but it also automatically gathers game data for each individual event that was fired. Also, this type of testing does not need to be done during working hours. The bots can be unleashed after hours, so when the humans get back to the office in the morning, a cup of coffee and a full report from the bots are waiting for them.

Along the way, developers and QA pros receive continuous insights into the bots performance through detailed reports. When this data is combined with the knowledge and experience of the QA people, they obtain an in-depth view of the game’s mechanics, structure and functionality like never before.

Bots can also be used in smoke testing to check the essential things like loading into a map, and they’ll report any null references and exceptions. The map in the test report can also be used to identify collider issues. All of this can be combined with custom event reporting, so that the issues you care about are surfaced clearly in the test report, alongside helpful debug information like the bot’s position, game state variables, and actions taken before the event fired. Besides logs, video recordings of each test can also be delivered to visualize and analyze crashes, errors, exceptions and warnings. Additionally, the test report contains performance monitoring graphs, allowing the developer to scrutinize memory and find leaks.

By harnessing the power of bots, QA professionals can unlock unparalleled efficiency, empowering them to achieve comprehensive testing coverage, expedite error detection, and deliver higher-quality gaming experiences. What’s left for the human QA pros is planning the tests, risk analysis, bug triage and analyzing the results—in other words, the strategic elements of QA.