No Test Case, No Bug

A while ago I remember Phil Kirkham mentioning that he’d found a bug that “fell” outside of the Test Cases he’d been given and someone was arguing that it wasn’t a bug. I found it incredibly interesting that someone would dismiss a serious bug because it was not found as part of a test case.

The test case is an abstract, it’s something we create to guide us (mostly), yet too many people take the test case to be the testing. The testing is a human process. The test case is a guide, an artefact, a crutch, something we may *have* to create, a container of information, a checklist or any other supporting element to our testing. The test case itself…is not testing.

About a year or so ago I wrote a short eBook on the problems with Testing. I set the main Test Manager character in a company who valued process over actual results. That process over results was similar to what I believe Phil saw.

This was originally going to be a decent sized blog post about process and test case management. Instead I thought I’d spent 5 minutes hacking together an Xtra Normal video to see if I could get the point across with animated characters instead (I also wanted an excuse to have a go at an Xtra Normal vid). (BTW – The Evil Tester did an Xtra Normal video here : http://www.eviltester.com/index.php/2010/03/03/hey-you-are-you-a-tester-the-m…

My video is by no means an award winning video, but it was actually quicker than writing out my thoughts. I think it conveys a similar meaning.

I’ve always wanted to be a film director too. I’ve got a lot of work to do to make that a paying reality :)

http://www.xtranormal.com/xtraplayr/13134755/no-test-case-no-bug

Tags: , ,

9 comments

  1. Phil Kirkham

    I’m proud to be the inspiration for that video – and it is almost word for word how the converstion wentYou could also do a similar one with Test Manager Droid 2000 where his jaw drops open after hearing I found a defect with no test case and then smoke coming out of the top with ‘Does not compute, does not compute’…

  2. pmberry2007

    Of course a bug is a bug, however found. One way around this blind insistence on procedure, though, is to simply reverse-engineer a test case from a bug.

  3. Rob Lambert

    Phil – I did originally script this as something I would animate and the robot did indeed explode because it couldn’t compute the fact people could test without a test case, but it was beyond the ability of the tool…and my skills :)Thanks for commenting.

  4. Rob Lambert

    Hi Paul,Thanks for commenting. Agreed. The easiest way is to create a test case and add it to the stack…..the problem with this though is that the metrics will be out now. Instead of 100 tests to run, at ten per day, this poor tester now has 101 tests to run. The plan has been agreed based on 100 test cases. This is a major problem now for Test Manager 3000 :)Cool idea though. It’s certainly what I would do in this situation….before looking for another job :)ThanksRob..

  5. James Christie

    Dearie me, Rob. You’re such a troublemaker. You just don’t get it do you?I’ll try and make this simple.Ok, it was a bug, BUT, bugs have no meaningful existence during test execution unless they’re found. If we insist, as a matter of both professional thoroughness and pragmatic management, that test cases flow directly and solely from the documented requirements, and we also insist that testing is restricted to those test cases then we can be sure that we won’t find annoying time-wasting bugs like you’re talking about.And if we don’t find them then for all practical purposes they don’t really exist. So we can go ahead and ship on time. Of course, the bug will probably manifest itself when we go live, but that’s not a problem. Well, it’s not a problem for us.We’ve got our traceablity matrix, and our mandatory standards. We tested properly. The bug must have been down to poorly specified requirements, so it was the users’ fault. Best not to blame the BAs. That’s bringing it uncomfortably close to home.So now do see why you’re just being a nuisance dragging up these wretched bugs that only exist because you’re one of the awkward squad?Get a grip and start managing the testing process, and stop getting all precious about managing the testing!

  6. Brilliant animation. If you have been in the software industry for a few years you will have come across variants of this conversation, usually the test manager, or project manager is staring down the barrel of the “will not make shipping date” gun if they recognise the bug and so try to use every angle available to them to avoid having to go up the line to announce a delay.NB: Not all test managers/PMs are like this, only the minority IMO.

  7. Rob Lambert

    Hi James,I like to question things :)I like the deeply philosophical angle you’ve brought to this. If it wasn’t on a test case (and hence not considered as part of the requirements) then did it exist at all. Deep.At least with this approach though, you’re right…we’ve passed the blame to someone else…the people who specified this stuff in the first place.Great comment and great little story about the effects of this -thanks for commenting.Rob..

  8. Rob Lambert

    Hi SheyMouse,Many thanks for commenting and glad you liked the video. I hadn’t thought of that scenario when I did the video, but absolutely. I’ve heard horror stories of managers finding ways to avoid the reality when faced with a tight deadline. Interesting point and one I shall ponder some more. That must be a tough situation to be in.Thanks for commenting.Rob..

  9. James Christie

    My post wasn’t much of an exaggeration. I’ve seen test strategies (never mine!) that state quite explicitly that the only source for test cases will be the signed off requirements. Once you set off down that particular road to damnation you’re putting huge pressure on the testers to execute only pre-written test cases.I’ve also just been looking at a Master Test Plan from a very large company that states quite brazenly that there will be no non-functional testing because the user hasn’t specified any non-functional requirements. This isn’t any old MTP. It’s an official exemplar, an example of “best practice”.