I’ve just spent the last 10 minutes trying to control what I believe are the emotions of rage, laughter and despair.
I’ve just got the results back from my participation in the Advanced System Testing Groups “Software Testing Skills Assessment” pilot. And boy did I laugh. I laughed at just how badly I had done. I got just 39%. Maybe I should consider a new career. After 10 years I’m obviously not cut out for the world of software testing. Well, the world of certified, standardised, poorly structured, no customer contact and no business contact software testing.
After I managed to control my laughter I entered a small, but perfectly agreeable fit of rage. I’d realised just how little information came back to me from the questions I fired off during the test and just how many assumptions Advanced System Testing Groups had made in creating these supposedly insightful assessments.
I then entering a stage of despair as it dawned on me that this way of testing the tester could well be adopted and embraced by unknowing industries and create yet another exam and certification.
So why do I disagree so strongly? Well here’s my reasoning’s:
- The test had no real world time limit which resulted in no risk based testing (even though they claim to have assessed against that….Advanced System Testing Groups – how were you assessing my ability to do risk based testing when I had plenty time to complete all tests and questions with no commercial pressures or reporting deadlines?)
- The pilot had no point of contact for questions or feedback or concerns (I fired some off via email and got nothing back)
- The test results sheet only allowed for one defect per feature (even though some testers were reporting more than one, this was not counted (or appeared not to be))
- The test application itself was so archaic and old fashioned (written as an MS access app) that in today’s modern world it seemed inappropriate, certainly for my context. It also didn’t open properly in my latest version of MS access.
- The instructions were not very clear
- There was no opportunity to test or report on performance, networks, security etc etc
- They were measuring using ISEB/ISTQB/IEEE techniques, which although very valid techniques, are not the best measure of test effectiveness (I worked with a tester once who spent so much time preparing flow, loop and data state diagrams that he left just 3 days for testing the app….)
- There were no measures for good communication, passion and pro-activeness, usability, accessibility, test case quality, exploratory testing charter quality, defect reporting, ability to learn etc
- The measures are the assessors measures of what makes a competent software tester, not mine, not my peers or colleagues – well of those I know well anyway. Is their assessment right? Can anyone truly assess the value of a tester when the industry is so varied and complex with so many overlapping roles?
- They bring each and every person down to one level. They assume a tester on the floor using the assessed technique checking all day will also be competent in a highly volatile, commercial decision meeting where a strong personality and commercial clout is needed. They bring a one level assessment to every single tester, in every single role, in every single company – and this is flawed.
- They ignore the commercial and market pressures found in the environment
- They ignore test data, test environments, accessibility, usability, performance and load etc.
- They ignore the human traits so very much needed to work in a testing role.
- They essentially apply a best practice to software testing. And surely we all know by now this simply does not exist.
- They ignore the communication skills needed to truly reflect the defect, report the metrics, communicate to the developer, raise the right level of priority etc
- And no doubt more I can’t bring to mind at the moment. Someone want to help me out?
Anyway. Enough ranting. I’ve decided I’m going to create my own assessment, pilot it, sell it, make millions from a certification and then retire safe in the knowledge I’ve brought standards and Best Practices to testing. Or I could just continue doing the best job I can, offer my services for mentoring, continue to offer hands on exploratory testing sessions and help to build a social community in testing where real value can be gained from sharing experiences and ideas. Millions….or career integrity……….
Tough one.
Rob..
Could you tell us how you really feel? <g>Your points are all significant and relevant. It sounds like the assessment is certifying at most a minimal level of mediocrity.From a different perspective, I can’t tell you how many testers (and developers) I’ve interviewed how they would test the triangle problem (3 inputs representing sides of a triangle, program outputs whether it’s a triangle, and if so, is it scalar, isosceles, or equilateral), and had it be a teaching moment (and generally a #fail).Would the above certification solve that issue?
Hi Earl, thanks for the comment. I’ve sat that same triangle test many many times and each time I know the right set of answers. I can prepare for it. It’s no longer new. It’s also a fairly tricky way of gauging effectiveness.I look for good innovation, intuition and communication skills coupled with a passion to learn. I don’t look for testing skills as such other than the general ability to understand testing concepts. One of the best tests I ever sat at an interview was a simple print out of a UI with some basic requirements. The guys interviewing gave me 10 minutes to point out as many defects and suggested tests possible, However at intervals they kept disturbing me, starting conversation with me and adding new requirements. They also weren’t looking for the average tests or defects, they were looking for outside testing such as accessibility, consistency, usability and supporting information like help text and guidance.It was a revelation in interview tests. It tested me and my reactions, not the standard iseb responses in my head. I couldn’t revise and hold in my head the answers these guys were after. It was an fun interview too.Thanks again for the feedback.Rob..
But does that 39% result mean in a test without a time limit, you found only 39% of the bugs (that they knew about)? Do you know how to interpret the result?
Hi Jared,Thanks for the comment.It’s not clear how they work it out to be honest. I’ll see about publishing some of the stats although not sure on copyright. I found somewhere in the region of 60% of the bugs that they knew about. I also found around 10 functionality bugs they didn’t recognise along side about 10 usability ones I would have raised in the real world to. Again, these were not counted.It’s really tricky to work out how they got my effectiveness to 39% as no-one observed me, gather feedback from me (other than the results sheet) and no-one fielded any questions from me.They do need to be more transparent in their results so I can work out what areas I need to improve in to pass the test. Some people are 80+% effective but most of these (according to the results) have only 1 or 2 years experience. This is quite telling as maybe these people are straight out of ISEB and applying all the techniques they’ve learnt. ??Thanks again.Rob..
Hi Rob,nice rant :-)The first question I have to ask is, why did you sit the test in the first place? What did you want to get out of it?What I find a bit sad is that you’re obviously very passionate about testing and that this is making your chances on the job market harder rather than easier. I’m not sure if you’re looking at the moment, so this is just a general comment.What I try and convince my team of that the single most important trait of a tester is their integrity. You can do a good or a bad job, but you can not knowingly hold back information or glaze over things that you know are wrong. If a tester wants to hold on to their passion for testing integrity is a must IMO. Of course, you can always switch off your brain when you get to work in the morning, do some checking, then go home in the evening knowing you haven’t made a difference but also got your pay cheque..Back on track… You said that this was a pilot. Were your comments taken on board and do you expect the course to change? Or is this it? A good course should have more (read several hours) of the type of questions you had in the interview you described. This could spark a discussion and show the trainee where they’re missing things and to understand why they’re important. So an oral test with a discussion, meaning two human beings exchanging ideas and points of views, would be a far better test for the tester in my view. This has it drawbacks as well but personally I’d rather employ someone who I know has set a test with one of the people I respect than someone who can tick the boxes.
I fully understand your anger.They have reduced your effort to a number, and lost everything that was important.Software is made for people, and therefore is more subjective than objective. The same goes for tester effectiveness.This means that we can’t, and shouldn’t, measure who is better than another.
I’ve sat that same triangle test many many times and each time I know the right set of answers.You’ve never sat through the version of the triangle program that James and I teach, and I’ll guarantee that you won’t know the right set of answers. We’ve been teaching it for years, and we still don’t know the right set of answers, except within our specific context for teaching it. And then, maybe.But we’ve got some important questions about it.This, to me, represents the big flaw in the way many people think about testing—that there are universal and context-free answers to the triangle problem. One of the reasons that the certification is bogus is that it makes the claim that there is one and only one right answer for the questions they put on the exam. Testing doesn’t work like that. For the same reason, I’d suggest that you not fall into their trap of thinking that your success as a tester is measured solely in terms of the number of bugs that you find. Can you communicate what you’ve found? Can you talk about risk? Can you identify tests that you’re not going to do, and why those tests wouldn’t be important in a particular situation? How are your note-taking skills? Are you handy with tools? Which ones? And so on…Which reminds me… when am I coming to Southampton? [grin]—Michael B.
Hi Thomas,Thanks for the comments. I sat the pilot in the belief that it *could* be different. I really wanted it to be different. It was described as being different. I did have doubts, like some others who took part, but I *really* wanted it to work.As soon as I started it though I knew it would never be able to assess me, as a tester, but more importantly; me as a person. Testing is more about soft skills than technical skills. It’s about being personable, communicating effectively and building bridges and relationships. And this is the part that no remote, single answer test can ever assess in combination with my personal work environment.The skills I need to do my testing role now are very different to those I needed about 5 years ago. Not that the testing world has moved on, but I’ve moved industries and companies and things are different. This is a good thing. We need the diversity. I was just hoping the test pilot would address this in some way. It didn’t.I do hope they take on board some feedback but so far I have received no responses to my initial set of questions, no feedback or evaluation form and as far as I am aware, there is no forum for discussions.I’ve yet to see a single way of assessing a tester completely. I’ve been interviewing people for a while now and the only things I really look for are passion, commitment and communication skills. The rest they can learn.Check out my iMeta blog http://blogs.imeta.co.uk/RLambert/Default.aspx for a post later today about the SIGIST event where Isabel Evans raised some interesting ideas about testing as a profession. Despite the heavy BSc push for control of this profession the points were quite valid. I disagreed with most of them though….. 🙂
Hi RikardThanks for the comment. Objective and subjective – I like it – wish I’d thought of those words when writing the original piece. Rob..
Hi Michael,Thanks for the comment.I’m in absolute agreement that their is no one correct answer. I’d also go so far to say that really effective testers have really low bug counts. Especially in my experience. If they are involved at the right stage, can build the communication bridges and can instill some thoughts for quality early in the process then there should be fewer bugs. I certainly don’t measure my success in terms of bug numbers. And yep. No doubt your triangle test would be indeed like nothing I’ve seen before. I’d love to sit the test. Unfortunately, here in the UK there seems to be a standard triangle test that can be downloaded and used. Hence, everyone knows the answers, well, those who do their homework and research.We really should organise a rapid software testing course here in the UK. I’ll gauge some response from the community here as to when would be a good time (budgetary and time) and we’ll get something sorted.ThanksRob..
Here’s the problem with the triangle test (again, as it’s usually presented): it’s somewhere between 30 and 50 years old!The “standard” version of the triangle test that you were doubtless given appears in The Art of Software Testing by Glenford Myers. That was published in 1979. The test itself was developed by Jerry Weinberg or Glen Myers or (I think) Harlan Mills, who were working with IBM at the time, quite a while before that. Jerry can’t remember who; doubtless there was some collaboration at some point.But consider what computing was like in 1979. At my high school, programs were still being submitted on punch cards. Around that time, our first green-screen terminal appeared. The school (one of the teachers, actually) had a Commodore PET. The Apple II was available, but relatively new. The Lisa was in development; virtually no one knew about graphical user interfaces, mice, multitasking (except on big machines), event-driven programming… The first IBM PC was two years in the future.The triangle problem, as it’s usually posed, is still based on functional correctness. As it’s usually posed, it posits a ludicrously simplistic view of boundary testing. It ignores problems associated with human factors and user interaction, security, performance, misinterpretation of requirements and specifications, testability, and context. As Rikard points out, it ignores a more important point that Weinberg was making in the 70s–that quality is value to some person–and that therefore the nature of what constitutes a bug depends upon whom you’re asking.The triangle problem, (again, as it’s usually posed), does have one salient feature: it shows how little the people delivering it haven’t changed the way they think about software testing for the last 30 years. And you wonder why your Windows laptop and your mobile handset and your printer drive you crazy?—Michael B.
Michael,Absolutely. Superb comment “The triangle problem, (again, as it’s usually posed), does have one salient feature: it shows how little the people delivering it haven’t changed the way they think about software testing for the last 30 years.”That is too true.Rob..
I pretty much ask the classic Weinberg/Myers triangle question. I have waited in vain for someone to mention its origin. The framing ‘statement’ is “I want you to go test this program, and come back and tell me with 100% assurance that it is 100% correct. Tell me what you’ll do to achieve that.” I word the requirements vaguely (“inputs”, for example), to see if they’ll ask clarifying questions. For those that dig into the inputs, I’ll even negotiate down to integers, and even to integers that are positively signed and <=255.The idea that 100% assurance is a canard has not been blurted out by *anyone*, nor has anyone run from the room screaming at the thought of working for such a clueless manager. Some come to that realization as a ‘light-bulb’ moment. In short, what I’d like to see is either a laugh (“that old question?”), a bang-bang-bang answer to get this obvious warm-up question out of the way, or a springboard to the substantial discussion that someone like Michael could engender.As a young developer, I always wondered why the test managers were at best, ah, testy, and on average curmudgeonly cranks. As one often said, “Quality goes out the door with every product we ship”.I appreciate them more every day.
Hi Earl,You’re spot on. Glad you appreciate them more each day :)Rob..