Professional Skeptics, Dispeller of Illusions and Questioner

Bear with me as a clear a few posts out of draft. This is my last one for a few weeks. Promise.

This one came about as response to James Bach’s excellent Open Lecture presentation. I took away a number of lessons from that lecture. (including the title – Professional Skeptics, Dispeller of Illusions and Questioner)

Off the back of that I decided to post some questions about “Testing the Spec” as an intriguing LinkedIn forum post got me thinking about why “Testing the Spec” is so common. “Testing the Spec” is where a Tester takes the spec and the system and then validates/verifies that the spec is correct. Yes, you read that correctly….that the spec is correct.

It got me thinking and asking a couple of questions:

What benefits will you receive by testing the system against the spec?
What don’t you know about the system? Will the spec help you?
Do you have any other information sources or Oracles?
Is the information sufficient? Or is it insufficient? Or redundant? Or contradictory?
Would a system diagram suffice?
At what point do you know you are complete? Once every page of the Spec has been “tested” – what about other parts of the system not covered by the spec?
Have you seen a system like this before? Will it help you?
Have you seen a slightly different system? Will it help you?
Who has asked you to do this?
What value do they see in you doing this?
Does the spec accurately depict the system? (I guess this is what they were testing for….)
Would it matter if the spec didn’t match the system? (which one would give you most value; the spec or the system?)
Is it essential to bring the spec up-to-date to match the system?
Will the spec tell you everything you need to know about the system under test?
Would the spec tell you anything that the system wouldn’t?
Do you know how out-of-date the spec is?
Is the spec important as a communication medium? Or just something that gets produced?
How would you communicate your findings? In Bug Reports? Or in a report?
How would you know what was a bug and what wasn’t?
Could the spec confuse you more than simply exploring the system?
Are you using the spec as a crutch or a guide or an Oracle?
How much of the unknown can you determine?
Can you derive something useful from the information you have?
Do you have enough information to model the system in your mind?
Have you used all the information?
Have you taken into account all essential notions in the problem?
How can you visualise or report the results and progress?
Can you see the result? How many different kinds of results can you see?
How many different ways have you tried to solve the problem?
What have others done in the past that might help?
Where should you do your Testing?
When should it be done and by whom?
Who will be responsible for what?
What milestones can best mark your progress?
How will you know when you are successful?
How will you know when you are done?

I’m not doubting the fact that testing a spec against a system is valuable. It *could* be the best thing you can do. But I would ask a lot of questions first. The system is always different to the spec, but in context, your spec could be used as the main reference point, so only you will know whether “Testing the Spec” is valuable for yourself.

I rattled out this list through a combination of the Phoenix Checklist but also after watching the very excellent video of James Bach doing an Open Lesson on Testing.

As with all things, I tend to sketch and draw as both a record of my thoughts, but also as a way of distilling some ideas. I read better in visuals so these rubbish doodles aid my learning and increase the chance I will revisit these points. You might struggle to see some of the text in the image but there are plenty of tools for opening images and zooming.

Note: The diagram is my take-aways from James’ lecture. There are many more lessons which I’ve taken away earlier from James’ blogs and talks. I *may* have interpreted some things differently to how you would, or even how James intended, but they are my take-aways. I share them here just for completeness and urge you to watch the video

One thought on “Professional Skeptics, Dispeller of Illusions and Questioner”