The ‘Proof of Accountability’ Problem in QA
The proof of accountability problem within testing describes the difficulty in making QA testers and QA analysts accountable for the work they produce and how this impacts the discipline’s performance across the industry. If we’re unable to accurately and fairly assign accountability, poor quality work goes unchecked and great work goes unrewarded.
So, why is it so difficult to assign accountability to QA members?
If testing misses a severe bug which is not discovered until a release is live, is it possible to attribute the root cause to a mistake made during test planning that can be used in performance evaluations? Put another way, is it possible to separate an unlucky bug missed in a solid test plan, from genuinely poor test planning that should have caught the bug? The answer, not easily. It requires a great deal of careful consideration from the line manager.
Another question, is it even right to assign accountability to major missed bugs? I think so, yes. The team needs to identify the root cause of the problem and take action to prevent it happening again. The ‘water under the bridge’ approach that many game studios choose to adopt is as equally damaging as the polar opposite of publicly chastising your QA team every time a major bug escapes to the live game. The problem with these extremes is that QA are never held accountable for the effectiveness of their test plans and are neglected this feedback in their performance reviews. This applies to both QA analysts for their test plans and QA testers for their testing.
Why “Should testing have caught the bug?” is so difficult to answer
The problem with this question is the large number of variables that need to be considered or disproved before an answer can be found. Let’s look at some possible accountability scenarios.
Ineffective testing: The test plan could have identified the scenario correctly that would have caught the bug, but the tester tested it incorrectly or ineffectively. The line manager would need to identify that this area was not a gap in the test plan coverage and then assess the quality of the writing in the test to determine if it was reasonable that the tester misunderstood it. If the test is well-written, the accountability for the mistake would lie within the tester team, otherwise it’s an action for the QA analyst to improve their test design.
Ineffective communication: The test plan could have identified the scenario correctly that would have caught the bug, but miscommunication between the QA analyst and the test team caused the bug not to be found. The nuances and complexities that arise during testing provide fertile ground for small details to be missed. Tester teams need to be setup to succeed in test execution by having all contextual and test environment information provided to them by the QA analyst owning the feature. Line managers would need to assess the complexity of the test scenario as well the format and frequency in which the QA analyst is providing information to their team of testers. If something was missed due to a communication error, could the QA analyst say that they had made ‘reasonable effort’ to keep the testers apprised of the latest information? This could either be an action for the testers or for the QA analyst.
Lack of sources of truth: The QA analyst who wrote the test plan might have had limited or incomplete information from which they based their test plan on, causing the buggy scenario to be missed in the plan. Are they to blame if they were not setup to do a good job? While an experienced test planner is more likely to identify that information must be missing and seek it out, we can’t expect this natural wariness from more junior team members. This becomes a very blurry area as the information is often available if you know who to speak to and what questions to ask. The line manager must review the documentation directly and assess what they would have done themselves in the same scenario. Failures caused by mistakes of this type are not a cause for concern but simply part of the learning journey of the QA analyst as they gain experience in the work. However, effort should be made not to repeat the same mistake twice!
Lack of time: The test analyst could have been under time pressure to write and execute the tests because earlier development phases overran, providing an excuse for the mistake which led to the bug being missed. This is a very common cause of poor test planning and the reason some QA analysts can go many years without improving their work or writing well-structured tests. Top performing QA members can become stunted in their career growth and poor performers can remain hidden among continuous excuses of time pressure. I’ve seen many poor performers hide under this umbrella, many of which are so accustomed to handing in partially completed plans that they follow the same pattern even when they do have enough time. They simply work more slowly.
There are many ways to combat time pressure and the QA analyst needs to have shown that they’ve communicated the shortfall early and have made a reasonable effort to mitigate it. The key to not repeating this mistake is improving the estimation and communication skills of the QA analyst so they are able to communicate to their team how long they need to create the plan and the impacts if that plan isn’t completed in time. It’s also important that the tasks of test planning and test execution are recorded separately on the project plan to increase the understanding that planning is a separate activity to testing. With this scenario, many times the accountability is entirely on the QA analyst, while other times they are only partially responsible. This tangle needs to be successfully unpicked by the line manager to seek a fair resolution.
Low priority tests not run: The QA analyst could have been forced to only run a subset of their planned tests and had assigned an incorrect priority to the test that would have found the bug, providing an excuse for the mistake. Sometimes, the difference between a critical bug being missed is the difference between making a test priority one or priority two. If the buggy scenario was identified within the plan and the team made a joint decision to only run a subset of that plan due to time constraints, then the accountability is with the entire team. Even if the QA analyst incorrectly assigned a single test to the wrong priority which led to the buggy scenario not being tested. This is a small and unlucky mistake. Instead we would highlight all the correct steps that the QA analyst took to get to this point, like prioritising their tests and proposing a priority-first strategy when met with time constraints. They also took the correct action to raise it with the team and get their approval. These steps show a reasonable effort to mitigate for time constraints despite the major bug still escaping to the live game and so it would be unfair to assign entire accountability to the QA analyst.
Complex bugs missed in a good plan: The test plan could have been very effective but still missed the bug simply because the bug was complex, difficult to find, very unpredictable, very low repro or occurred in an area which nobody identified as an area of change.
This scenario is the real sticking point.
The reality of software -games particularly- is that complex bugs appear which are unpredictable to even the most senior team members. Often, these bugs aren’t caught during testing and when they do appear, their root cause takes a long time to diagnose and fix. It’s not a reasonable expectation that a test plan catches such a bug. Indeed, several of these bugs may have been successfully caught during testing, but it only takes one to escape and cause havoc. As a result, we (rightly) don’t hold QA analysts accountable for every bug missed. If the team and the line manager agree the bug was complex or edge case and it wasn’t reasonable that the QA analyst could have predicted the bug would be there, then this scenario wouldn’t be a strike against their performance. The problem here is, ofcourse, identifying if it was ‘reasonable’ for the QA analyst to have predicted the buggy scenario.
Other reasons: There are also many other reasons why major bugs might escape to the live game. The complexities and games development and the huge number of people involved in their making mean that mistakes will be made. Like the ones described above, each scenario needs to be analysed, accountability assigned and actions recorded.
Why is proof of accountability a problem?
Without assigning accountability, QA analysts and QA testers are either unfairly targeted when major bugs escape or they’re given a reusable ‘get out of jail for free’ card. Neither of these approaches enable the discipline to grow because they treat poor performers and high performers equally. I’ve seen many QA team members continually underperform because of this grey area, sometimes knowingly, other times not. QA analysts that coast through their work without progressing to more senior roles because there are no repercussions. Without strong QA line management, untangling the mess of accountability and identifying specific examples that can be used in a performance review or personal development plan is extremely difficult. So the problem goes unchecked.
There is a side topic here that the solutions-orientated, nicey-nicey approach to QA accountability is an overreaction to the historical problem of QA team abuse by company leadership. There is an equilibrium that must be found here for the QA discipline to grow.
To successfully evaluate the effectiveness of a test plan, QA managers need to be skilled in test planning themselves. They must be aware of all project influences and determine what portion of accountability lies on the QA analyst and list actions they could take to improve their performance in the future.
A segue to the ‘scope of work’ problem
This topic I’ve been discussing is closely coupled with the separate topic that the scope of work for a test plan is self-defined by the author of that test plan.
There is no clear or consistent measure for the definition of done when it comes to test planning.
When a developer or an artist executes a piece of work, the output of that work is visible to everyone and the definition of done clearly defined. The work often has acceptance criteria and any missed areas are picked up during testing. This is a stark contrast to self-defined limits of any test plan and the favourite test question, “How do you know when you are done testing?”.
Because QA analysts define the scope of their own work, they are in control of how much or little detail (and therefore effort) they put into any test plan. Every test plan is a balance of achieving confidence while maintaining efficiency. Overtesting or undertesting tips the balance too far to one side of this scale. Because the ‘correct amount of testing’ is so difficult to define, this also is a heavy contributor to QA analysts underperforming and putting insufficient detail into their plans. Many QA analysts write insufficiently detailed test plans which goes unchecked because they are not held accountable to any external definition of done. I’ll talk more about The Scope of Work Problem in another article.
If the testers are the watchmen who test the definition of done for everyone else, who watches the watchmen?
Enjoying my content?
Consider buying my book to keep reading or support further articles with a small donation.
