Game Testing with AI Bots
An honest "Help Wanted" ad for a modern QA (Quality Assurance) specialist would include phrases like willingness to be on call 24/7, must love repetition, have a perfect memory, and possess multiple heads and hands to flawlessly carry out a huge number of tasks daily.
Everyone knows that QA testing is a critical stage in the game development process, as it ensures the creation of a high-quality gameplay experience free from inconsistencies, glitches, and bugs. To achieve this, QA is responsible for test planning, risk analysis, smoke testing, exploratory testing, bug hunting, regression testing, testing all possible combinations... and then analyzing test results and fixing bugs with reproduction steps.
The drive for adequate automation is quite understandable and absolutely necessary. A key element of this is unit testing, where the smallest testable parts of an application are individually checked for proper functioning. Software developers and QA specialists perform various automated tests, including unit tests, during development. While some of these tests could be done manually, they are often automated to save time. However, it is important to note that automated tests are not a silver bullet. If existing automated testing solutions truly solved QA problems, we wouldn't see the level of energy around AI in testing that we observe today.
QA testing is a job that requires a high degree of imagination, creative analysis, inventiveness, and strategic thinking. In other words, human creativity and intelligence are necessary for success. Unlike yesterday's automation tools, which advanced incrementally, artificial intelligence, particularly AI bots, is rapidly advancing game testing for developers and QA specialists because it finally delivers on the promise of freeing up their time to focus on what matters most.
Exploratory Testing with Bots to Discover Bugs
AI-driven exploratory testing is one of the most exciting and truly transformative advancements in QA in recent years. By using AI bots that can explore the entire game, a large amount of game content can be tested in just a few clicks. Exploratory bots can go everywhere a player could possibly go. This advanced technology allows QA specialists to allocate their time more efficiently and focus on other important tasks, ultimately streamlining the entire game development process.
This provides an extensive and incredibly powerful approach to identifying bugs in the game. By deploying multiple exploratory bots, each autonomously moving across the map, they can find the tiniest bugs and glitches by walking, running, jumping, climbing, and crawling everywhere a character can reach (and sometimes ending up in places the character shouldn't be, revealing level design bugs). A horde of QA bots can quickly uncover bugs, potentially saving days, weeks, or much more time compared to manual and other testing methods.
Exploratory testing starts by placing the bot or bots in a safe, predefined location on the level. They are then instructed to explore the game environment using reconnaissance movements. The bot resets its exploration when it reaches certain goals or criteria, such as falling out of the level or encountering anomalies. The bot can provide a heatmap of the areas it has explored. Multiple bots can play at speeds many times faster than a human player. All bots read data from and add to the heatmap, enabling rapid results. This not only reduces the burden on QA, as they don't need to visit every coordinate on the map, but also automatically collects game data for each individual event that was triggered. Additionally, this type of testing does not necessarily have to be done during working hours. Bots can be run after hours, so when people return to the office in the morning, a cup of coffee and a full report from the bots await them.
Along the way, developers and QA specialists receive constant feedback on the bots' activity in the form of detailed reports. When this data is combined with the knowledge and experience of QA specialists, they gain deep insight into the game's mechanics, structure, and functionality like never before.
Bots can also be used in smoke testing to verify crucial things like loading into the map, and they will report any null references and exceptions. The map in the test report can also be used to identify collider issues. All of this can be combined with custom event reporting so that the test report clearly displays the issues that matter to you, along with useful debugging information: bot position, game state variables, and actions taken before the event was triggered. In addition to logs, video recordings of each test can be provided to visualize and analyze crashes, errors, exceptions, and warnings. Moreover, the test report includes performance monitoring graphs, allowing the developer to closely examine memory and find leaks.
By leveraging the power of bots, QA specialists can achieve unprecedented efficiency, enabling them to achieve full test coverage, accelerate bug detection, and ensure higher game quality. The human role then becomes test planning, risk analysis, bug fixing, and results analysis—in other words, focusing on the strategic elements of QA.