Why it’s important for QA Engineers to know about game KPIs

This text is a translation of an article originally published on DOU.ua.

My name is Hryhorii Baturin, and I’m a QA engineer at Plarium Kharkiv. I currently work on Mech Arena, a mobile, multiplayer, third-person real-time action-shooter. The game has recently been released globally.

Thinking back to when I was preparing for the interview for my job, I noticed at the time that there was very little information available online about testing games. I’d like to fix that - and hopefully make life easier for those looking to work as QA engineers - by talking about one thing in particular that can make you stand out among other candidates: an understanding of KPIs - key performance indicators.

As a QA engineer, when you know the basic KPI metrics, you’ll be able to prioritize tasks and analyze test results more effectively. Plus, when you have a grasp of the relevant terminology, it will be much easier for you to build an effective dialogue with other teams. What’s more, if, during an interview, you're asked to prioritize a hypothetical workload (a classic QA engineering task), you’ll be in a position to use the metrics to justify your decisions.

Apart from going into more detail on all of those points, in this article I’ll also use a feature from Mech Arena as an example to illustrate which KPIs are especially important for QA engineers to pay attention to.

What’s it like to work as a game QA engineer?

Just like with any profession, working as a QA engineer in the gaming industry requires a specific skill set and knowledge. It’s worth keeping that in mind not only when preparing for an interview, but also when planning your professional development.

Below, I’ve listed some of the qualities of great QA engineers, particularly as they relate to the development and implementation of test cases for game projects.

1. Experience with the game engine This is one of the core technical skills that differentiates a professional QA engineer from a beginner. A good specialist understands the principles of the engine being used, and can create specific test scenarios for it.

2. Extensive gaming experience Do you have “extensive” gaming experience if you’ve played Dota 2 for 8000 hours? What about if you’ve been playing World of Warcraft since the day it was released? Exactly what defines “gaming experience” can be hard to pin down, but for a QA engineer in gamedev, what’s crucial is not the number of hours spent playing. Instead, it’s important to have played a variety of games across multiple genres.

3. Deep knowledge of the target platform and operating system Nowadays, most projects are created for multiple platforms. This means that a QA engineer must understand the specifics of the operating system they’re working with, be it Android, iOS, or Windows, or a native operating system such as, for example, the Nintendo Switch. When you have that knowledge, you’ll be able to make the most of an operating system’s functionality.

4. An understanding of the specifics of testing games This is really an overarching point that connects all the ones listed above. It means things like being able to manage teamwork and testing priorities, divide up tasks, and estimate how much time tests will take.

All the skills above are much easier to acquire and develop when you have an understanding of game KPI metrics. Knowing the metrics, a QA engineer will be able to do things like accurately assess what the impact of bringing something new into a game will be, regardless of if that happens at the initial or final stages of development. In addition, they’ll also be able to suggest changes that will simplify development and testing processes without impacting target indicators. As to what those target indicators are, let’s take a look.

What are game KPIs?

The fundamental game KPIs can be systematized in the AERM model – Acquisition, Engagement, Retention, Monetization. The model illustrates the user's journey through a game.

There are other models, such as AARRR – Acquisition, Activation, Retention, Referral, Revenue, but we’ll stick to the AERM model, since it’s easier for beginners. Here’s an overview, then we’ll look at each stage in turn:

The AERM Model

Acquisition

We don’t need to focus much on this first stage as it largely concerns marketing and game design departments. Still, as we will touch upon it later, it is worth knowing about its key aspects. They are shown in the table below:

The various aspects of the “Acquisition” stage of the AERM model

There are three key metrics associated with the Acquisition stage:

1. New Users - These are users who come into the game for the first time. Here, it’s important to separate the concepts of an account and a person. One person can have multiple accounts, and each account they create will be counted as a New User. Depending on the type and purpose of the project, instead of “New Users”, the terms “Downloads” or “Installs” might be used.

2. Total Users - As you might have guessed, the total number of users in your game. This indicator includes both active accounts and inactive ones until they are deleted.

3. Cost Per Install (CPI) - This is equal to the marketing costs for attracting one user. The term Customer Acquisition Cost is sometimes used as an alternative.

Engagement

Engagement and Retention are often combined because their indicators are directly related to players’ interest. These are the most sizable stages because they represent the behavior of players in a game.

For QA engineers, it’s important to note that the quality of testing has a big influence on the indicators of these stages. If a game contains bugs or is difficult to master initially, users will not want to return to it. Therefore, a QA engineer must stay on top of regression testing and be ready to suggest solutions that will improve the gaming experience. Engagement breaks down like this:

The various aspects of the “Engagement” stage of the AERM model

The overall engagement indicator is Average Session Length. This indicator is sometimes referred to as Session Duration, Visit Duration, or Average Session Time. It is calculated using the formula:

Another key engagement metric is Participation Rate, which is often used to assess the impact of bringing something new into a game. As an indicator that measures the share of players who participated in a gaming activity or event, it allows for the tracking of overall player involvement in any aspect of a game.

Retention

The indicators of the Retention stage reflect the extent to which users are interested in returning to a game. Most updates to games are designed to improve this KPI, which means it can be used when analyzing test results. This is how Retention breaks down:

The various aspects of the “Retention” stage of the AERM model

The Retention Rate can be considered the main indicator here. It is the percentage of users who login to a game on the Nth day after creating an account.

Retention Rate is often divided into:

• Short-term Retention – Day 1, Day 3, Day 7
• Long-term Retention – Day 14, Day 28/30, Day 60, Day 90

In relation to Retention Rate, these abbreviations are often used:

• DAU (Daily Active Users) – the number of users who enter a game at least once over the course of a day
• WAU (Weekly Active Users) – the number of users who enter a game at least once over the course of a week
• MAU (Monthly Active Users) – the number of users who enter a game at least once over the course of a month

When discussing retention, it is also worth mentioning Churn Rate. This indicator is used to estimate the number of users who have left a game over a certain period of time (a two-week period is often used). For a QA engineer, it’s a key indicator because if, for example, Churn Rate suddenly increases, that suggests there are errors in a release.

Monetization

Monetization is a key stage in the development of any game project. Among its chief benefits is that it can fund the creation of new features that diversify gameplay and improve the overall game experience for players. It also demonstrates which aspects of a game players find most interesting, by providing data on things like which game modes or characters enjoy the most demand.

When a development team has access to that kind of information, they are able to understand what to prioritize for future releases. Taking Mech Arena as an example, if we see that lots of players are purchasing a particular Mech, the development team can add skins or quests for it to the game, or make it easier to get—for example by making it part of the rewards for taking part in a certain in-game event. That would also have the effect of increasing player interest in the event, which is no bad thing.

Returning to our AERM model, here is a summary of the most important aspects of the Monetization stage:

The various aspects of the “Monetization” stage of the AERM model

When looking at a game’s monetization, it’s important to take into account the ratio of CPI (Cost Per Install) to LTV (Lifetime Value). The latter is also sometimes called CLV (Customer Lifetime Value), and measures the income generated from one user across their "life" in the game.

For monetization to be a success, it’s important that the two indicators stay in a state whereby LTV > CPI. And while the relationship between LTV and CPI doesn’t tell the whole story (it doesn’t take into account the cost of development and testing), it’s still very useful for making judgements about a game’s monetization overall.

But if, on the other hand, we want to evaluate the monetization performance of individual game features, there are a number of other metrics that can help us do just that, such as Paying User Share. This is the percentage of players among the entire player base who have made at least one payment over a particular period of time. It’s calculated like this:

Other useful metrics here include:

Conversion Rate – This tells us what share of DAUs (daily active users) have become paying users. This indicator is also used in assessing marketing performance.

Average Revenue Per User (ARPU) – A measure of the income that one active user generates on average over a given period of time. It is calculated according to the following formula:

Average Revenue Per Paying User (ARPPU) – A measure of the income generated on average from a user who has made at least one payment over a given period of time.

So, now that we’ve been through our model and become familiar with all the terms, let’s see how they apply to testing a feature in Mech Arena.

Using game metrics to evaluate a Mech Arena feature

One of the features we have in Mech Arena is what we call an influencer link. This is a link to download the game that social media influencers can give out to their followers. Anyone who uses the link to install the game receives a reward, which looks like this in-game:

Before we test any feature—while we’re still at the stage of planning and analyzing its requirements—we need to understand which stages in the AERM model are influenced by the feature, and why. Let’s take a look at how this works in the case of the influencer link:

Acquisition - The influencer link is for attracting new players, so the key metrics here will be New Users, Total Users, and Cost Per Install.

Retention - Since players receive their reward immediately after passing through a tutorial, if there are errors in how the reward is displayed, that could significantly affect player interest. So here we need to think about DAU and Churn Rate.

Monetization - While this feature isn’t linked to monetization directly, the size of the reward can affect the number of people who will continue to play and become paying users, so the New Paying Users metric is important here.

Once you think you’ve identified a feature’s key metrics, my recommendation is to have a discussion with the game designer responsible for the feature’s implementation. Don’t be afraid to ask questions and suggest changes at the planning stage. This will give you the best understanding of whether or not the metrics you’ve identified are the right ones.

In addition, don’t forget that an important stage of planning is to evaluate how much time you’ll need to run tests. In the case of the influencer link, we had to keep in mind that the feature is a marketing tool, so we had to allow extra time to get feedback from the marketing team.

Testing of a feature should only begin once you know the full timeline for the feature’s implementation and have identified the key metrics it will affect. Knowledge of the metrics is crucial, as it will help you defend your point of view when suggesting improvements in a game’s balance or UI elements.

Once a feature is released, that’s when you should ask questions about how the metrics have been changed. That way, you’ll be able to suggest improvements and innovations for future releases. In the case of the influencer link, for example, one improvement could be to provide more unique rewards based on what players have said they want.

Learn more about game KPIs

I recommend regularly checking out a range of resources to keep your knowledge on KPIs up-to-date, and learn about new approaches. Below I’ve collected together several useful links:

Video:

Major Mobile Application Metrics (devtodev)

Mobile Game Development Explained [2020]

Articles:

12 Key Mobile Game Metrics That Developers Must Track

KPI approach: Metrics and stages of the game project

Glossary of game metrics (devtodev)

Knowledge of KPIs and their underlying metrics is important for everyone involved in the creation and development of a game, QA engineers included. In fact, as QA engineers, it's crucial for us to be able to identify patterns and trends when the metrics change; only then can we argue our point of view and push for improvements.