ISTQB Game Testing (CT-GaMe) : A MGT Review
I recently passed the exam* for ISTQB's new game testing certification. It's one of the first internationally recognised certificates for the game testing profession, something which was long overdue and desired by game QA teams worldwide.
So, the question which you all want the answer to.
Is it any good?
In this article I review the syllabus as a standalone published work, how effectively it teaches game testing skills and how closely it matches the pragmatic day-to-day work in modern game companies. Finally, I'll recommend whether it would strengthen a CV, regardless of the content inside.
The intent of this review is not to critique the work of individual contributors to the syllabus, but instead to evaluate the extent to which the published syllabus and paid-for exams, books and courses support the game testing industry. Because many game testers will invest in this certification, it’s incredibly important that the content within is accurate and representative of the real work of modern game testers.
Let’s get started.
*Full disclosure of my score at the bottom of the article.
Syllabus overview
I can’t help but eye roll at the first thing I see when I open the syllabus. The title, Certified Tester Game Testing, shortened to GT-GaMe. Wait. Shouldn’t that be CT-GaTe? Of course they had to crowbar in the GaMe acronym.
Rising above my petty grievances I move onto the syllabus overview. It totals 76 pages, covering these chapters:
Specificity in game testing
Game mechanics testing
Graphics testing
Sound testing
Game controllers testing
Game level testing
Localisation testing
I cast my eye down the table of contents, taking in the chapter headings and recommended study time for each chapter. Okay, not bad. A reasonable spread of the knowledge buffet. I'm happy to see terms that I recognise and the focus on game content has been identified in the graphics chapter. A little surprised localisation made it onto the list, particularly considering this is a relatively short document.
Hungry to understand where the syllabus puts the biggest focus, I look closer at the study time for each chapter, identifying the meatiest parts. This is where things start to come undone. The biggest chapter by study time is -wait for it- sound testing!
Not what I would have chosen.
My expectation is that this document is a holistic introduction to game testing, and at only 76 pages will need to cover a lot of topics. Sound testing isn’t unwelcome, but I was surprised to see it at the top of the list. More on the contents of that chapter later.
Chapters by study time:
Sound testing - 190 mins (20%)
Game mechanics testing - 180 mins (19%)
Graphics testing - 160 mins (18%)
Localisation testing - 155 mins (17%)
Game controllers testing - 95 mins (10%)
Specificity in game testing - 75 mins (9%)
Game level testing - 55 mins (7%)
The other chapter sizes are more what I might expect, with the exception of localisation. Game mechanics (any and all in-game logic) is a massive area for testing and the focus on graphics is certainly very unique to games.
Regarding the total scope of what we’re about to learn, I can’t help but compare this to my own list of game domain areas that I cover in my book (shown below).
Platform Integration
Third-party SDK integration
Platform certification requirements
Regulations and age rating compliance
UI, menus and navigation
Game environment
Content (dragons, items, etc.)
User generated content (chat, skins, art, etc.)
Game modes (camp, levels, events, etc.)
Multiplayer (matchmaking, modes, leaderboards, etc.)
Game progress and retention
Online services
Live events, sales and AB tests
Audio, music and sound effects
Player profiles and file IO
Identities and cloud save
Hardware and peripherals compatibility
Tutorials
Telemetry and analytics
Anti-cheat and security
Settings and configurations
Performance
Tooling and debug
Upon further scrutiny of reading the syllabus through, there’s unfortunately a lot missing from it, which brings us to my first major feedback point.
#1 - The scope of the syllabus is poorly defined
It’s not made clear what is and isn’t included in the syllabus. This is true for the coverage of game domain areas above, but is equally true for other areas of game testing work. For example, the correct application of different testing techniques and types (exploratory vs. scripted, verification vs. failure, etc.) is only casually referenced in parts of the syllabus. Equally, nowhere does it reference who this syllabus is aimed at, only implicitly referring to the QA tester role, as opposed to the work of QA analysts or QA managers. You won’t find any guidance on test planning and higher-level strategy here, everything is written from the point of view of an individual tester. While some project milestones and phases are defined within each chapter, the syllabus doesn’t paint a holistic picture of how each feature timeline fits into a typical game project lifecycle or what a test schedule looks like.
The decision to exclude some parts is acceptable, even necessary for the sake of conciseness. The problem is it's not explained anywhere. The reader is thrown into the gory details of the first chapter without context to guide them. Senior QA roles already in the industry can use their experience to frame the content, but aspiring testers will struggle to understand the wider context, and unfortunately they're most likely to take the exam.
Let's move onto the details of each chapter.
Specificity in Game Testing
This introductory chapter (6 pages total) aims to tell us why game testing is different to all other software testing through a brief overview of game product risks, typical bugs found in games and why testing games is different to playing games. The content presents a mixed bag of usefulness, with mostly valid points, but coupled with misplaced statements and a few incomplete lists.
Most of the content is well-placed as introductory topics. The callout to the reality that testing is not the same as playing the game is a welcome necessity, setting the expectation to the many aspiring game testers who will undoubtedly take the exam to learn about the industry. The callouts to the specifics of negative testing and multi-platform specific complications in games are brief, but accurate.
However, the section on game product risks and common game bugs feels hastily created. The lists provided are incomplete and the items within randomly selected. While the syllabus is quick to tell us that the lists aren’t exhaustive, I can’t help but remain confused by them.
“Such [game product] risks include, but are not limited to:
- Gaining an unfair advantage
- Dependence on market success from the subjective opinion of the players
- Multi-platform issues
- Complexity of predicting the effectiveness of game mechanics
Testers can find defects in the following areas:
- In the architecture of gaming software early in the development lifecycle
- In the sound design
- In the text of a video game that is already available for release”
The second list here is particularly confusing; both the intent of the list and items within remain a mystery to me.
This is unfortunately a trend that I see throughout the syllabus, lists that appear out of place or poorly defined. If you're going to provide a list of items you need to have a solid crack at making it as exhaustive as possible and ideally place the most prevalent items at the top.
Is 'Gaining an unfair advantage' the biggest and most common risk in video games? I don't think so.
This point applies doubly so if the contents will form an exam question. If you asked me to blindly list common game product risks, the list might be correct but still different to the one provided here. My difficulty memorising the sporadic set of topics in this introduction and the contents of each list was reflected in my poor exam performance for the chapter, 25%! Oh well, moving on.
Sound testing
Despite the fact I think this chapter is too large for the syllabus, the content within it is actually very insightful and well-written. I observed that the syllabus was written by multiple members of the Russian ISTQB group. The writing style, depth and clarity of this chapter appears different from the others and I would guess to be the work of a single audio QA specialist who won the competition of who could write the most content.
The chapter goes into detail about the different types of audio content, how they are created and the different roles that are required to integrate them into the game. It does a good job of describing how sound effects are attached to objects and game areas, and the difference between bugs with the source file versus how they are set up in the game, then goes further to define who might fix those bugs, a sound engineer, a level designer or a developer.
All this is very welcome if, like me, you haven't worked on a project where audio was a high priority or you haven't been the audio QA specialist to work on it. For the same reasons, the application of this chapter to your own project is likely to be very binary, either being very helpful or not at all.
Lastly, the chapter also does something that none of the others do, it provides a substantial list of examples. Praise Cthulhu!
A good time to raise my second feedback point.
#2 - The syllabus lacks examples, pragmatic context and feels intentionally dry
Aside from the sound testing chapter, there are very few examples provided to elaborate on the points made. There is also very little project context provided to help the reader understand how each point fits into real game project work. This is a huge flaw considering the number one feedback point for my own talks has been on the helpfulness of specific examples and stories from real projects. The syllabus (like others from the ISTQB) is a dry read and feels like it was written to necessitate purchasing a study guide book or a tought course. However, the sound testing chapter has examples, so why not the others?
Game mechanics testing
This chapter could be huge. I was curious if they'd attempt to categorise common game mechanics across the major game genres and how they're going to approach such a large area. Unfortunately we don't get anything so specific.
The chosen breakdown is to classify game mechanics via: gameplay vs. non-gameplay mechanics, core loops vs. meta loops and client vs. server mechanics.
The categorisation itself is reasonable, but I got lost in the fog when reading through the details of this chapter. The level of detail provided varies wildly and when coupled with questionable word choices and sentence structure; the resulting thread of narrative guiding you through the content is extremely slack. The best example of this is the client<->server section which delves into a gory explanation of server authoritative logic, showing what happens if a player is able to modify their local game to cheat and what effect that has on single-player and multiplayer games. Conversely, the section on gameplay mechanics provides only a high level summary using generic terminology.
The epitome of this generalisation can be read in the first paragraph on common defects in game mechanics, which provides such a broad explanation that it loses nearly all meaning.
"Game mechanics involve influences on game objects and feedback signalling the result of these influences. Taken together, this creates the unique character of the game, with suitable dynamics. Most often, games use a wide range of influences and feedback elements".
What?
This brings us neatly to my third feedback point.
#3 - The syllabus feels tailored to a specific type of game
The areas with the biggest focus and explanation are the most relevant to the multiplayer shooter genre and similar games. This means your own mileage of syllabus applicability will vary depending on what project you're working on. As a result, it also provides a slightly biassed view of what is and isn't core game testing knowledge. The client<->server section is both accurate and insightful, but I think the space could have been better used for more foundational, must-have information.
Lastly, this bias sets the reader on a rollercoaster ride through the content, skimming some topics briefly before plunging abruptly into a deep rabbit hole of complexity and explanation. It can be very disorientating and necessitates the need for a study guide to set it straight.
Graphics Testing
The third course, let's goooo!
The size of this chapter is well-placed and the content within it detailed and relevant. With a few minor exceptions, it does a good job of explaining the different components of game graphical content, all of those parts that come together to make a game world and populate it with objects and characters. Like other chapters, it's half explanation on graphical content itself and half explanation of how to test it. Topics include: 3D models, 2D models, textures, lighting, animations, character rigging, visual effects, collision geometry vs. visual geometry, level-of-detail, collisions and hit-boxes.
It’s widely applicable to many games and insightful for any testers who haven't specialised in art and content testing. Having worked on some AAA projects, I was aware of these components but I was never involved in the gory details of their testing, making the content here a nice refresher. Equally, it's explained clearly enough that newcomers will be able to learn it and apply it in their work.
I also appreciate how this chapter explains the process and roles involved to create objects, environments and effects in various creation tools, then import them into the graphics engine editor to be placed in a scene. The context of how an artist, world designer and developer work on different parts of the creation process is helpful context for testers when finding and categorising content bugs. Additionally, this is exactly the type of context that many testers are not exposed to due to their distance from the development team, making this syllabus a welcome supplement.
So, what’s not to like?
Historical accuracy. Both this chapter and others state that part of the tester’s job is to test the game against historical accuracy and authenticity. This is both a very specialist area and a highly controversial responsibility for testers.
“To test graphics in games, the tester must have knowledge of physics and optics, have an understanding of color rendering techniques, and knowledge of history”.
“Typical aspects to consider when testing graphic content for historical accuracy are:
- Accuracy of reproducing weapons, equipment, vehicles, clothing of a particular era and their characteristics”
I’m sorry, but I can’t get on onboard with testers being responsible for historical inaccuracies.
In terms of RACI (responsible, accountable, consulted, informed), testers are certainly consulted and provide input into historical inaccuracy bugs, but the syllabus is repeatedly suggestive that testers should read up on their knowledge of history as a prerequisite to testing work, which is an unrealistic and unfair expectation of new testers reading this syllabus. It’s far more likely that the bulk of historical accuracy work will be tackled at a much earlier stage of development in a conversation between art and design project leaders. They are accountable and responsible, testers are simply consulted.
This chapter also suffers from the aforementioned incomplete lists and rollercoaster of variable detail. There’s a particularly puzzling paragraph on computer generated images which would be at home in one of my too-early-in-the-morning lectures back at university, sitting in a dusty lecture theatre and listening to some dry topic that no one has found a practical use for in the last 10 years.
“Any image of the monitor, due to its flatness, becomes a raster, since the monitor is a matrix consisting of columns and rows. 3D graphics exist only in our imagination, since what a human sees on the monitor is a projection of a 3D figure, and the viewer creates the space by themselves. Thus, visualisation of graphics can only be raster and vector, and the rendering method is only raster. The way of defining the image depends on the number of these pixels”.
WHY!?
Is that detail really necessary?
This question is even more puzzling when we remember that any part of the syllabus can be included in the exam.
Localisation testing
The last big chapter, 17% of the syllabus.
Why so big? Their definition of localisation has a broad scope and includes targeting specific audiences and regions by adapting the game to align with the cultural, linguistic, religious and political characteristics of each audience. This cultural adaptation considers many facets: humor, religion, historical accuracy, peculiarities of national culture/worldview and legal restrictions.
The full-fat explanation has some interesting sections and does a comprehensive job of detailing all the nuances of preparing a game for the global market and trying to penetrate difficult markets. The content is very relevant in specific scenarios where entire sections of game content need to be replaced, like when launching your game in China which forbids certain topics and imagery, and requires a submission to the government.
However, the chapter leaves me with some challenging questions.
How relevant is deep localisation to the majority of game projects?
How much overlap is there between the functional testing skill set and a localisation expert?
Should localisation be included here as a subset of testing or is it a discipline in its own right?
It’s no secret that many game QA folks enter the market through functional localisation (or translation) testing by being multilingual, then move into other functional testing roles. The chapter includes some good detail about the different game areas where translation is required and paints a picture of the type of work a functional localisation tester might carry out. The details about testing translation of different types is accurate and well-placed.
However, I struggle to see how most of this chapter will be applicable to the day-to-day work of a functional tester within games, particularly the parts about deep cultural adaptation. I also know many localisation managers who would enthusiastically argue that the work is specialist enough to require its own job family and not be bundled within the QA discipline. In the Venn diagram of localisation and testing, there is overlap, but they remain two distinct career paths.
As for the breadth of applicability, I return to my previous comment that the syllabus should be a foundation of must-have knowledge which applies to the widest possible group of projects. While the practical details of functional localisation testing are welcome, the rest of the space would be better served with more holistically applicable knowledge.
Game controllers and levels testing
The two final chapters are smaller and cover -you guessed it- testing of different types of game controllers and testing of levels (game environments).
At only a few pages each, these chapters succinctly describe the main testing approaches and common bugs of each area. They’re both very unique to games and serve as a good addition to the syllabus. There is less variation in the level of detail provided here, which is a welcome change.
The section on game controllers is mostly what you would expect: button mappings, in-game legends and controller swapping; covering a wide variety of common and specialist input devices including touch and motion controllers. There's a strange paragraph on security testing of input devices, which recommends checking that certain controllers can’t hack the console or gain unauthorised access to developer mode. ‘Hacking’ the console would be outside the scope of developing an individual game and I’d be very worried if a game enabled debug in a production build but hid it behind a button combination. That would be a very risky choice and an unlikely bug. However, you have test it to eliminate doubt, so I’ll categorise that as a valid, but unlikely point.
The content on game level testing is brief but well-constructed. It does a good job of describing the key phases of level (or environment) design, the role of testing at each stage of creation and the categories of common level bugs. This is exactly the kind of art and design process that is completely unique to games and which is miles from any software testing approach. This chapter is useful if you haven’t been exposed to the details of environment testing or you haven’t worked on a project which has a substantial 3D environment. It’s a thumbs-up from me here.
Conclusions
The syllabus is a solid attempt at defining a foundation of game testing knowledge. It provides detail about testing processes, approaches and trends which are unique to games, all of which is very welcome knowledge. It also does a good job of describing the wider creation process for each game area, which I agree is necessary context to understand how testing is organised. In that respect, large sections of the syllabus are essentially describing how games are built. Finally, QA managers, analysts and testers may find the syllabus helpful to fill in gaps in their knowledge, since no single person can specialise in all game areas.
The problems I have are around the size, structure and scope of the syllabus. It needs to be larger to provide the pragmatic details necessary to support real testing work, at least twice the size. There are many smaller areas of game development and testing which aren’t included in the syllabus or referenced at all. This is made worse by the inefficient choice of topics and unnecessary detail where it isn’t required.
The insightful sections are entwined with irrelevant or out-of-scope sections, making it difficult for novice readers to pick out only the valid content. Furthermore, some sections are outright misleading, which I think is damaging to the expectations of aspiring game testers. Due to the widespread popularity of the ISTQB organisation, they have a responsibility to publish content that is authentic and substantiated, a mark which is missed in parts of the syllabus.
Would I recommend the syllabus to existing game test teams? Yes, it's free and worth a read.
Would I recommend enrolling on training courses for this certificate? No. I don't think they're worth the financial investment.
Would I recommend the syllabus and the exam to aspiring game testers and junior testers with little experience? Yes.
The content within is too high level to be useful to existing teams and most existing QA folks are more likely to pick up the same knowledge through their work. The syllabus does a good job of summarising some key game testing topics if you don't already have experience of them and may be enough to equip junior testers with the knowledge they need to move into more senior or specialist. Lastly, if you want the certificate under your belt, it is internationally recognised and is worth the financial investment regardless of the content within.
That's it. My review of the ISTQB Game testing syllabus.
*Here is my exam score and breakdown
Pass score: 65%
My score: 72.5% (29/40 marks)
Specificity in game testing : 25%
Game mechanics testing : 83%
Graphics testing : 100%
Sound testing : 75%
Game controllers testing : 60%
Game level testing : 40%
Localisation testing : 100%
Enjoying my content?
Consider buying my book to keep reading or support further articles with a small donation.
