A perennial question I get asked is how do you measure the effectiveness of a tester?
I respond with “Why do you want to measure the effectiveness of the testers in your team?”
I do this to try and understand the motives behind why measures of individual’s performance are important to managers (as it’s typically managers that ask me that).
It is indeed important to know how someone performs against their job/objectives, but the motives behind working this out are important. Is it to improve the person, provide training and support, or simply to measure it for budgetary reasons.
Don’t get me wrong, I like measures. We measure all sorts of things here, but we’re careful about the conclusions we take from the measures.
The typical response I get to my question above is that managers (especially test managers) are trying to justify to those holding the budget the value a tester adds (or doesn’t add) to the team.
They are often trying to measure two things*
Firstly, whether Tester 1 and Tester 2 add the same value as each other. I.e. Can we switch them around and still fulfill the business objectives? Is one better than the other?
Secondly, they are trying to measure whether Tester 1, or Tester 2 are even adding value to the team. (what would happen if we took them away?)
This often leads managers to measure crazy numbers like test case completion, or defect detection rates as a way of measuring an individual’s performance.
As crazy as this sounds it’s incredibly common. Managers are after simple metrics to inform decisions about individuals.
I can see why. In fact, I know one blazingly obvious reason why this is so prevalent — it’s because managing people is really hard!
Managing people and understanding the value they add takes time and, in software development at least, cannot be measured by a single number (despite what many managers believe).
Managing people takes patience. It’s about building relationships. This is hard. Good management reveals truths about yourself as a manager that can be painful to accept.
So managers dig around for metrics to work out an individuals performance. Relying on a number is easier than relying a wide range of team numbers (naturally influenced and affected by more than an individual), your own observations and the feedback from others.
The most common measure I hear back from test managers when it comes to measuring people’s performance is the Defect Detection Rate.
It’s an interesting measure in its own right – interesting as in “does it actually tell you about the quality of the process and testing?”
It’s especially interesting when you start to use this measure as a way of measuring an INDIVIDUAL’S performance. Can a single tester (or any other team member) control how many bugs are in the product and the rate at which the team flush them out?
Someone the other day emailed me a massive calculation that she uses to measure her tester’s performance. The calculation was a combination of test case completion rates, the number of bugs detected, the period the bugs were found in, the speed of bug resolution and the number of tests created. Bizarre. It was so complicated and so open to gaming/misuse/inaccuracy/pointlessness that she floored me.
How astounding that a person’s future and career progression is literally put down to a calculation as bad as this. But that’s how it is in some companies.
It’s a common story to hear of testers (and their future prosperity) controlled by bug counts and test case completion rates. These measures are often taken within a business system that wouldn’t allow good testing to happen anyway. The system often doesn’t even support the goals and business objectives.
- What about the team’s results?
- What about the business value the software adds?
- What about revenue generated by the value the software adds?
- What about the teams ability to solve problems?
- What about the cycle time of work?
- What about the feedback from the customer?
- What about the person’s ability to grow and improve?
- What about the feelings, feedback and emotional reactions of others to this person?
Well, — these other ways are often hard to measure and don’t provide a single answer on their own. Combined together though and the above tell a compelling story…. that’s powerful stuff, but it’s hard for managers to gather this.
So here it is.
I provide you with a calculation to rule all calculations. A calculation that renders all other calculations of team/person performance redundant.
Trust me — use this calculation and it will give you a number.
This number is the right number.
It’s the most important number you have.
It’s the only number you need.
It’s the most complete number possible.
I will add this though:
PLEASE don’t use this number to make a decision about someone’s career and future.
Instead, do the work. Build the relationship. Understand the person.
Manage people, not just numbers.
Why not work out your number and leave it in the comments.
* Managers are often trying to measure many other things, and sometimes fall in to the trap of using a single measure to inform a bewildering array of decisions