Qualicon '23 : MGT Coverage
Now in it's third year, Qualicon again brought us a treasure trove of game testing and quality insights in this year's conference. Describing itself as created by game quality professionals, for game quality professionals, it's focused entirely on the QA discipline. This certainly shows. Every single talk and discussion was directly relevant and insightful. A refreshing change after attending general game development conferences in the UK like Develop: Brighton which don't include a QA stream and pose topics which are interesting but not actually relevant to my work ( returning from a conference with no learnings does not provoke a positive response from your team! )
Qualicon is remote and well-priced, resulting in an easy sell to your manager for budget approval. The downside is that we don't get to shoot the shit over evening beers with our fellow QA chums, or lose terrifically at outdoor laser tag - which is my favourite part. Beers aside, the conference also starts at 4/5PM for European folks, which is convenient for not ruining your already-full working week.
So, the talks! What did we learn? I'm not going to give a blow-by-blow of every one, but let's take a look at some of the highlights.
Opening remarks
I enjoyed the brief opening remarks from Jimmy Bischoff, Head of Quality at Xbox Game Studios. He sensibly addressed the trite topic of recent AI advancements and their impact on game quality and testing, saying that we're likely to see more content, code and art within games that have been generated automatically and that it will create new unique risks and testing approaches which we'll have to adapt to. Some examples he gave included:
Procedural generation of game content through AI
Personalised gameplay experiences based off player behaviour
Adaptive difficulty levels based on live player behaviour
AI created animations which are more realistic
AI generated code
With that out of the way let's agree not to talk about AI for the rest of the conference or this article. Agreed? Okay, great.
4 Key Strategies to Balance Work and Life, while Advancing Your Career in a Fast-Paced Environment
Michael Boggs, QA Manager at Epic Games.
Day one kicked off with a talk from Michael about personal improvement and career advancement, with the lens that it's difficult to 'find time' to make conscious efforts amidst a fast-paced game dev schedule. A well-positioned topic and a harsh reality for many people in the industry, QA or otherwise. The pragmatic approach taken in the talk identified the common difficulties and provided practical solutions.
His talk was split into four main parts:
Path of self-actualisation
Find your OWN authentic problems
Building your personal brand
Practical tips
The common theme running through the talk was of self-evaluation and personalisation of your career. Identify your own authentic problems and build your personal brand around solving those problems, instead of trying to focus on what other people care about and the problems they have. I can really relate to this and have found that other people become interested over time in my own problems and solutions.
It seems that your own problems are often also new problems, and in solving them, sharing your work with your team and the public naturally follows.
Michael also called out some books for us to check out. "One Minute Manager", "Extreme Leadership" and "Fostering Innovation".
Standout quote: "Show, not tell. I wouldn't ask anything of my team that I wasn't able to do myself".
Soft Skills and the Role They Play in Your Career Development
Jennifer Monaco, Senior Quality Lead, Xbox Game Studios
This talk from Jennifer hit a lot of the same notes as Michael's talk but with a heavier focus on communication and emotional intelligence - a topic which is often missing from recruitment and performance processes in the workplace.
There were many great reminders and tips in this talk pertaining to self-improvement, like the fact that soft skills apply holistically to all work, making everything you do that little, bit, easier! We also had a reminder that soft skills are transferable between jobs and will be helpful to you wherever you go.
Standout quote: "Having good emotional intelligence, the ability to effectively communicate your needs, and solve problems can contribute to your overall fulfilment and happiness - both inside and outside of work"
Achieving effective test automation scale through mindful partnerships between technical QDs and QE
Andrian Maroiu, Automation Architect at EA, DICE
Cristian Deri, Group Technical Director at EA, DICE
If you want to feel simultaneously dumb and amazed, look no further than the latest presentation from the automation leadership at EA. Their talk will make you realise that your QA org is about as mature as a toddler throwing a tantrum. It focuses on the problems and solutions trying to scale their automated end-to-end tests, 3000 of which currently run every day. A side note before we begin, listening to Andrian’s dulcet tones explain complex topics never gets boring. I could listen to that shit all day. I highly recommend attending if you see these guys present in the future.
For some background context, the central automation teams at EA support many projects across multiple game studios owned by EA. In my view, the maturity of their automation comes from the conjunction of building franchise games which benefit from year-to-year incremental automation investment and the benefits of shared company technologies that can be used by many game teams.
Because of this maturity and scale, Andrian and Christian explain to us that their ‘quality verification’ (QV) automation team is split into two sub-teams, an autotest team (QD) and a quality engineering team (QE). The former is a team of quality designers who are responsible for writing automated tests that serve a variety of studios across the organisation, whereas the latter is a team of quality engineers who are responsible for the infrastructure and tooling required to support the automated tests. They also have an engine QV team which focuses more on engine performance and stability assessments.
The talk gave a vertical slice of two main parts of the organisation.
The first was how the team dealt with the increased scale of running tests and specifically triaging the failing tests. In the old system, making sense of failed autotests required a huge amount of tribal knowledge and a broad and deep skillet in different areas. The number of failed tests was also increasing as the automated tests scaled, making the triage burden worse. As a solution, they developed a post-process service which scanned artefacts from automated test results and tagged them if they contained known reasons for failure. Failure categorisation matured over time and new categories were defined.
The second vertical slide gave us insight into the relationship between the different QV teams and with the project dev teams that they serve. They described that the early relationship was transactional between the autotest team and the quality engineering team, and that there was too large a barrier between the QV team as a whole and the dev teams they served. The solution was to bring the different automation teams closer together and foster a heavier overlap in their skill sets, which resulted in greater synergy with multiple positive outcomes.
Faster, more precise and much more efficient
More empathy and maturity
Higher specialisation ceiling for QA
Higher quality ceiling for dev
Repeatedly avoiding instances of tunnel vision
Fix now, new standard for the future
United strategy and vision
Standout quote: "For automated exploratory tests, we have a scriptable AI that we used to control the bots / autoplayers in the games and this behaviour has become more complex over time".
Transforming Bungie QA from Risk-Based to Value-Driven Testing
Leland Dantzler, Senior QA Discipline Manager, Bungie
Next up we hear from Bungie who are delivering new seasons of Destiny 2, a AAA games-as-a-service product. The content of the talk centres around sustaining regression tests for an increasingly large and complex game.
The talk focused around the number of tests being run for each new season of Destiny 2; around 18k for season 8 which ballooned to around 109k by season 16. Their team identity was to ship (almost) flawless games and what they were calling risk-based testing was actually fear-based testing. Everything was a risk, therefore everything needed to be tested.
The solution was to change their identity and mindset to a value-based approach; 'we ship the right value and the right risks'. By focusing on value, the team were able to focus on the right risks and not be driven by risk as a default. More specifically, they categorised different values that were important to the team to help drive decision making.
Player value
Business value
Development Pipeline Value
Process Improvement Value
Cross-Discipline Collaboration Value
I really like this quantifiable approach to defining value. I’m sure this helped them get buy-in from non-QA stakeholders and navigate specific cases after the change in direction.
This all ultimately led to the massacre of a huge number of valueless test cases; a ‘testocide’ if you will. May they never be forgotten.
The team identified and cut almost 100k test cases that didn’t find any bugs in their previous release, resulting in a 55% reduction in test cases without a reduction in the coverage or confidence.
During the talk we also learn about their 'QA discipline manager' role which sits outside the structure of game teams and are line managers to the QA team members, craft leads and are responsible for hiring and training. I really like the presence of this role and the intentional separation from the often hectic pace of project work. Judging by the Bungie presentations, it allows them time and freedom to shape and grow the QA discipline, and make changes to the vision (as they did in this talk). Take a mental note here of the benefits of being in a company large enough to sustain such roles.
Standout quote: "Who we are drives what we do"
From defect detection to defect prevention: Voices from the industry (A journey through shift-left quality practices)
Ivan Doumenc, Senior Engineering Manager, EA
John Santamaria, Development Director, EA
Tyler Moeller, Principal Software Quality Engineering Manager, Xbox Game Studios, Mojang
Matt Ng, Director of Franchise Quality, Xbox Game Studios, Mojang
Gareth Rogers, Senior Quality Lead, Xbox Game Studios, Rare
Kudos goes to Tyler for having the longest job title. You’ll get your medal in the post!
It was difficult to choose which talks to include in this overview, but this one really stood out to me. In this panel discussion we heard stories from quality leads at Microsoft and Electronic Arts about their successes implementing quality at source initiatives (shift-left) and heard candid accounts of the varying success each team had.
The panel kicked off by identifying why shift-left is so important by highlighting problems that many of us can relate to.
You can’t inject quality back into the game after the fact
We’re always operating at dev capacity
We’re always focusing on feature delivery
How do we justify investments with no clear ROI?
Culture - Quality is someone else’s problem
First we heard from Gareth on the approach that Rare took during the inception of the Sea of Thieves project. They’d been burned by a previous project which created an appetite for change across the entire project team going into the next project. This resulted in a culture shift whereby the dev team would not have a black box test team to support them and they would have to own the quality and testing of their own work from the outset. Quality engineers were hired into the development team and over time they ‘established small beachheads’ in automation and tooling, then held them.
When I asked about specific processes and tools they developed to support shift-left, the response from both Rare and Mojang was that shift-left was a people and culture problem, not a technical hurdle - which I found really insightful. “No one intentionally checks in bad code” was a good response that I got. This goes some way to explaining why it's so bloomin' difficult.
Today, the Rare team do have a black box test team supporting their GaaS game Sea of Thieves, but it’s only around 30 people.
Interestingly, EA and Mojang reported a more difficult journey with their established projects. One of the problems adopting shift-left at Mojang was their current popularity and success, which I've personally seen in successful mobile games. Success hides many development sins, it seems!
A few more info bites from the talks were:
Definition of done checklists for dev commits
Dedicated quality engineering team at Rare lowered the bar to adopting better quality practices for the rest of the dev team
Unit testing is a big ask and not the first thing to employ. Start with static analysis and code coverage analysis
Pairwise dev and testing is great
My final comment here is that I've had repeated recommendations to check out previous talks from Rare at Qualicon and GDC. So, if you have access to those vaults. Go check them out! Some of it may even be on YouTube.
Standout quote: "For shift left, a lot of the conversation and success comes from people and culture, NOT from process and tech".
Conclusions
This was my first year of Qualicon and the content was top tier. I look forward to seeing what next year brings and I'll spend the interim watching the talks from previous years.
I've only had the space to include some of the content here but I hope it's given you a taste of what you could see in future years. Being an event focused specifically on game quality and testing, every talk is directly relevant to QA folk, resulting in a very high 'hit rate' of productive points.
Peace and love until next year.
Enjoying my content?
Consider buying my book to keep reading or support further articles with a small donation.
