Consistent User Actions

This morning I was in my local Tesco supermarket and noticed a classic case of inconsistency between the message and the action.

I am a seriously big fan of the self service checkouts in Supermarkets. Not only have they reduced queues but they’ve made it entirely possible to dehumanise the entire experience of food shopping; which, when the customer service is often so bad, can actually be a positive thing.

I’m also a fan of using my own bags, so I press the “Are You Using Your Own Bags?” option. At this point I receive two messages which don’t match the action:

A voice suggests I place my bags on the packing section and then press “Go”

The screen suggests I place my bags on the packing section and then press “Go” but shows me a button labelled “Continue”

It may seem trivial to point this out, that both mediums suggests “Go” and the actual next step saying “Continue” but it had me scanning the real estate for a few seconds before realising it was a mistake. Other users may not find this so trivial.

And in many contexts, this simply doesn’t matter, but as a Tester it’s important to check for consistency between prompts, advice, help, information, guidance and the actual action or process.

In some contexts this consistency could be very important.

Push The Button

During a conversation with a group of testers at an event I soon found myself outnumbered in my views around “enhancements” to the product. I was the only one who saw a Tester’s role as more than just verification.

I was a little amazed at how this group of Tester’s (or shall we call them Checker’s?) narrowed the focus of their roles down to simply following a script, clicking buttons and ticking boxes. They were also incredibly passionate about not suggesting improvements to the process or product. It’s not so clear to me though where the line is between enhancement and bug. It’s not always easy to categorise.

Through sheer persistence and the use of pursuasive language and examples of real bug stories I convinced them that there was more to Software Testing than checking. I also bought them all a drink, but I don’t suspect that had anything to do with it. 🙂 I’m not saying they weren’t doing a good job, far from it, but to not look at the wider context around your checks is to ignore the purpose of the system.

I promised to blog about the things we talked about, and here it is, about a year later than it should have been.

This post stemmed from a point one of the guys made “A button is a button. It’s the same in every system. We test it does what the spec says it should do.”

At a basic level this is absolutely correct, but to look at a button with this narrow focus is to miss the point of Testing (as I see it). The button is just one action, or element, or interaction in a much bigger process or system. It may fire off something or trigger an event but the interesting questions for me are:

  • Why would I click the button?
  • What are my expectation?
  • Do I have any choices around clicking the button?
  • Will I like what happens?
  • How will you encourage me, force me or convince me to click the button?

And a whole lot more. To ignore the context of the button is to not question the system it lives in. To check the button does what the spec says, is to put too much trust in the spec. So I blurted out a few ideas about testing, which actually cross over in to design. I’ve included as many as I could remember here in this post. Some may be helpful for you, some may not.

  Is the button accessible to all users?

  • Should it be?
  • Do certain users get more or fewer buttons? Why?
  • For example, admin users or permissions based systems.


Why would someone click this button?

  • Are there other choices?
  • What will drive someone to click this button?
  • What happens in the grand scheme of things if they don’t click it?


How will your users “feel” before and after clicking the button?

  • Do you know?
  • Does it matter?
  • Can you redesign to cater for this “feeling”?
  • For example, submitting your tax returns will give your users a different feeling to submitting a newsletter subscription form


Is this behaviour monitored or detected or regulated?

  • Does this change the button form?
  • Does this change the behaviour?
  • Who detects it and why?
  • Are there privacy issues or data protection concerns?
  • What is done with this data?
  • For example, audit trails in systems.


If your users access the site via a mobile device, does the button still have the same look and feel?

  • Does it still do the same thing?
  • In the same way?
  • Should it?

Do previous choices or data entered alter the button choice and action?

  • Should it alter it?
  • Could it alter it to get a better outcome for all?
  • When does it alter it, and when doesn’t it?


Does the button have the potential to do damage and therefore require more consideration or a confirmation?

  • Status updates?
  • Sending personal details?
  • Firing missiles?
  • Turning off life saving machines or devices?

Do you need further steps in the process or is clicking this button enough?


Is it possible to design the overall page and the button to deter or encourage certain behaviour?

  • For example, to eliminate the “Donkey” vote the candidate voting lists are often randomised

Does anything surrounding the button affect the choice or decision?

  • Should it?
  • Could it?
  • Can we affect choice by anchoring adverts or information or other users choices?
  • For example, peer review scoring on sites like Amazon

Could the button or process be disabled but visible to encourage upgrades, add ons or extra steps?

  • Should it?
  • Does it fit with the business model?
  • Would your users appreciate this?

Is the action the button performs pleasurable for the user? Or negative?

  • Could this process be sweetened if negative?
  • Or rewarded if good?

Is the button consistent with other buttons in the process or system?

  • If not, why?
  • If yes, should it be?
  • Would it be better with a different stand out maybe?

Does it matter whether you are the first person to click this button?

  • Do you get extra features?
  • Do you disable a process for other users?
  • Is the outcome clearly communicated?
  • For example, one time downloads or data collections.

Can you click the button more than once?

  • Why?
  • How?
  • Does it function consistently?
  • How does this affect the system or data or process?
  • For example, submitting a form more than once


Does the button support multiple users?

  • Can more than one user click their own button at the same time?
  • At different times?
  • At all?

Can you double click the button?

  • What does it do?
  • Should it be possible?
  • Could you deter this behaviour with warnings or informational text?
  • For example, some users click hyperlinks twice.

Is the design of the button modern, or old?

  • Can it be skinned?
  • Does it look old for a reason (functional, to encourage upgrades etc)

Do you need supporting help text?

  • Is it obvious it’s help text?
  • Can it be on or off?
  • Or on all the time?
  • Is it obvious the text is relating to this button or process?
  • Is the help system being designed against a recommended compliance like the elmer guidelines for form design?

Can you create an emotional attachment when the user clicks on the button?

  • Can you get buy in, repeat business or a brand fan?
  • Do you want to?
  • Do you need to?
  • For example, the Like button on Facebook

What action does the button perform and is that consistent with what your users want to do?

  • Submit, next, back, details, open, close, print, etc
  • For example, the final submission button in a wizard should not be called continue, unless it has supporting text to inform the user what that button will do.

Is there a time limit surrounding the button?

  • A limit before clicking or does it start a timer?
  • How would this affect the design and testing?

Is the button grouped with other like minded buttons, or dejected and lonely?

  • Is the like minded group logical?
  • Does it convey the right message?
  • Is the lonely button there for a draw attention?
  • For example, all text formatting buttons should be logically grouped, rather than spread across a busy toolbar.

How does the button perform with no supported visuals?

  • Text only
  • Screen readers
  • Accessibility

Are you using colour to symbolise or signify meaning?

  • Red for danger
  • Green for go
  • Are these signs common amongst your users?
  • Ambiguous?
  • Should you be using colour? Accessibility? Colour blind?
  • Do they display well on different monitors?
  • If you had no colour could you still use the system?

Does the button actually do what the user expects?

  • Regardless of the spec, is the button action logical?
  • Are the expectations wrong?
  • Is the action wrong?
  • Is your oracle wrong?
  • Is it by design, fake affordance maybe?

Is the button part of a sequence?

  • How is the sequence represented?
  • Is the sequence logical?
  • Can the sequence be tricked, skipped or broken?
  • Is the sequence communicated well?

Does the action of the button mirror actions or features in other systems?

  • Will users see a link between systems, for example a status update in Facebook?
  • Could you redesign to take advantage of that association?
  • Would you want the association?
  • Is it worth redesigning a common process, or will the baby duck syndrome kick in

Is the process surrounding the button common?

  • For example installers.
  • Does the process require much thought or is an almost natural process of unthinking clicking.
  • For example, the browser add on tools that are hidden in the middle of a software installer. They work on the basis that you will not read it and hence install the browser add on.
  • What could you do to encourage more thought around the process? Make it harder?

Does the button itself represent the action?

  • Delete icon, rubbish bin icon, cross, plus sign.
  • Are they logical?
  • Are they repeated on the same screen adding ambiguity?
  • Are the symbols universal to your audience?

If there is a breadcrumb trail, how will clicking this button affect that?

Does the button require data fields to be completed?

  • Should the button be visible until all fields are complete?
  • Is that clearly communicated?
  • Greyed out? Disabled? Button text change? Disappear/Reappear
  • When does it validate? Where? How?
  • Is it consistent?

Does the button recall previous data?

  • What happens when clicked more than once?
  • How is the data represented or affected

Is the button generating real time feedback?

  • How real time?
  • Does this affect performance?
  • Is it required?

Does the button work under multiple instances?

  • Multiple browsers and/or tabs?

What lies beneath the button?

  • Can you glean any further information about the button by using a tool like Firebug?
  • Does it have alt text?
  • Is it logically labelled and suitable for automation?
  • Are there any comments in the code or other clues?

Is the button the final action?

  • Does the user need to know anything before they click the button?
  • Do they get a confirmation?
  • Are they reassured enough to click it?

If the button is part of a process, would it better to push the process through a wizard?

  • Will the user experience be better? Less prone to mistakes?
  • Will the process be done in one go, or can they save and return?

Does the button need an undo?

  • Does the action need a reverse or opposite action?

Is the button multilingual?

  • What happens to the button when it tries to show different languages or locales?
  • Does it still make sense?
  • Does it hold longer text?

Can the user press this button without using the mouse?

  • Mobile devices. No right clicks. Etc
  • Shortcut keys? Tab order?
  • Should they be able to?

Can it be made easier or harder to press the button depending on your objectives?

  • Many large organisations appear to be making it hard to find the button which leads the user to the telephone numbers or listings.  Self service is cheaper but is it always better?
  • Big red buttons with lots of “click me” appeal
  • Button in the middle of the page, or hidden in the top corner.

If you removed the button and that part of the process how would it affect your system?

  • Would it make it better?
  • Unshippable?
  • Un-usable?
  • Better?

A lot of these ideas come about when reading about design ethnography and UX, particularly around lenses, but for each one I’ve always been able to relate this back to a bug or “enhancement” I’ve seen in systems I test.

Questions about the bigger process will lead to more test ideas about the button (and the bigger process also). The button is a part of a bigger system, which is operating in a context. I think it is at this level that individual actions start to make more sense as a whole, and you can begin to question who, what, why, where and when.

If you find yourself working in an environment where you’re not allowed to raise questions about the checks you are running and you really want to (and attempts have fallen on deaf ears), then it may be time to move on and see what else is out there.

Professional Skeptics, Dispeller of Illusions and Questioner

Bear with me as a clear a few posts out of draft. This is my last one for a few weeks. Promise.


This one came about as response to James Bach’s excellent Open Lecture presentation. I took away a number of lessons from that lecture. (including the title – Professional Skeptics, Dispeller of Illusions and Questioner)


Off the back of that I decided to post some questions about “Testing the Spec” as an intriguing LinkedIn forum post got me thinking about why “Testing the Spec” is so common. “Testing the Spec” is where a Tester takes the spec and the system and then validates/verifies that the spec is correct. Yes, you read that correctly….that the spec is correct.

It got me thinking and asking a couple of questions:

  1. What benefits will you receive by testing the system against the spec?
  2. What don’t you know about the system? Will the spec help you?
  3. Do you have any other information sources or Oracles?
  4. Is the information sufficient? Or is it insufficient? Or redundant? Or contradictory?
  5. Would a system diagram suffice?
  6. At what point do you know you are complete? Once every page of the Spec has been “tested” – what about other parts of the system not covered by the spec?
  7. Have you seen a system like this before? Will it help you?
  8. Have you seen a slightly different system? Will it help you?
  9. Who has asked you to do this?
  10. What value do they see in you doing this?
  11. Does the spec accurately depict the system? (I guess this is what they were testing for….)
  12. Would it matter if the spec didn’t match the system? (which one would give you most value; the spec or the system?)
  13. Is it essential to bring the spec up-to-date to match the system?
  14. Will the spec tell you everything you need to know about the system under test?
  15. Would the spec tell you anything that the system wouldn’t?
  16. Do you know how out-of-date the spec is?
  17. Is the spec important as a communication medium? Or just something that gets produced?
  18. How would you communicate your findings? In Bug Reports? Or in a report?
  19. How would you know what was a bug and what wasn’t?
  20. Could the spec confuse you more than simply exploring the system?
  21. Are you using the spec as a crutch or a guide or an Oracle?
  22. How much of the unknown can you determine?
  23. Can you derive something useful from the information you have?
  24. Do you have enough information to model the system in your mind?
  25. Have you used all the information?
  26. Have you taken into account all essential notions in the problem?
  27. How can you visualise or report the results and progress?
  28. Can you see the result? How many different kinds of results can you see?
  29. How many different ways have you tried to solve the problem?
  30. What have others done in the past that might help?
  31. Where should you do your Testing?
  32. When should it be done and by whom?
  33. Who will be responsible for what?
  34. What milestones can best mark your progress?
  35. How will you know when you are successful?
  36. How will you know when you are done?

I’m not doubting the fact that testing a spec against a system is valuable. It *could* be the best thing you can do. But I would ask a lot of questions first. The system is always different to the spec, but in context, your spec could be used as the main reference point, so only you will know whether “Testing the Spec” is valuable for yourself.


I rattled out this list through a combination of the Phoenix Checklist but also after watching the very excellent video of James Bach doing an Open Lesson on Testing.


As with all things, I tend to sketch and draw as both a record of my thoughts, but also as a way of distilling some ideas. I read better in visuals so these rubbish doodles aid my learning and increase the chance I will revisit these points. You might struggle to see some of the text in the image but there are plenty of tools for opening images and zooming.


Note: The diagram is my take-aways from James’ lecture. There are many more lessons  which I’ve taken away earlier from James’ blogs and talks. I *may* have interpreted some things differently to how you would, or even how James intended, but they are my take-aways. I share them here just for completeness and urge you to watch the video

Please replace me, let me go

One of my favourite blogs ( carried an essay/article on humans and machines by Marina Gorbis. It’s a breezy article but makes some awesome points and each point Marina made rang true for what we are seeing in the Testing world.

Marina makes a point that machines are replacing the mechanistic jobs traditionally done by humans, which is allowing humans to concentrate on critical thinking and situational response and other “thinking” activities.

This is true in the Testing world where many people utilise machines to perform the jobs machines are good at, leaving the humans to do exploratory testing, critical thinking and creative endeavours.

I believe Michael Bolton summed this up perfectly when he made the critical distinction between Testing and Checking. Checking could be considered the mechanistic tasks, potentially best performed by a machine. Testing is the human task best performed by a….human.

Marina lists out a number of ways in which Machines are becoming included in everyday work. I’d like to briefly list her ideas out here and draw the parallel to what I see happening in the Testing world.


So what are machines good at?

Machines are awesome at repetitive, mechanistic tasks.
Those tasks that are mechanistic and repeatable. The tasks you do daily by rote are perfect for a machine.

If you can programmatically check something, aim at getting that code written.

So if you spend your day ticking boxes, checking hyperlinks work or simply clicking through the same User Interface over and over again, then you need to think about using a machine to do this.

Machines are awesome at doing the things humans cannot do.
You want to deliver 100,000 requests to a server at one go – Use a machine.

You want to check how well a device works in sub zero temperatures whilst being bombarded with Acid Rain – use a machine. (believe me, someone I know tests products in these conditions)

If you need to “test” something but it’s impossible for a human, then a machine could aid you in achieving your goal.

Machines are awesome at analysing complex data and applying rationality
A quote from Marina’s article :

“For centuries, economists have built models based on the assumption that humans behave as rational economic actors. But thanks to advances in neuroscience and behavioral economics, we’ve come to realize that humans aren’t good at thinking through probabilities and risks and making rational economic choices based on those probabilities. While we don’t want to use pure rationality when making moral or ethical decisions, more rationality would be helpful in situations such as when making financial decisions.”

Should we start using machines to aid us in decision making?

Could we start using machines to inform our decisions about what and where to focus testing? You don’t already?

How about using Pairwise tools to filter down your combinations?

How about using machines to number crunch previous system usage or bug counts to find high risk areas to test?

How about using machines to analyse performance metrics gathered from a Load Test?


Machines are awesome at doing tasks which are too large or too small
Those pesky tasks that are really small or really large could be completed by a machine.

Filling in compliance reports with test data.

Adding all of the change sets to the release notes.

Producing reports on Test Coverage and Results.


So what are humans good at?


Humans are awesome at Thinking
A quote from Marina’s article:

“Thinking and computation are different processes, and machines are not good at thinking”

So we have these machines that can do loads of cool things, but as of today, they cannot really think. Testing is a very human process as we aim to discover more about the system under test.

In order to understand more, we need to peel back layers to discover new insights. These insights require a human to think about them. To understand them. To respond to them. To base new Tests on them.


Humans are awesome at Social and emotional intelligence
Being able to connect to your domain and subject matter, in my opinion, leads to better software testing. I’m a firm believer that a Tester can do great testing in any domain and on any product, but I believe they will do awesome testing when they have an interest and an emotional connection to the product or systems they are testing.


When you are emotionally connected, at any level, I believe your senses are hightended and you become more channelled and focussed. As of today, there are some emotionally aware machines, but they are still in early development.


“Feeling is as complicated as thinking, if not more so, and just as the machines we’re building aren’t thinking machines, the emotional and social robots we’re building aren’t feeling machines—at least not yet.”

Without emotions how can you understand why your end user is finding the UI so frustrating?
How can you connect and build relationships with your colleagues and stakeholders?
How can you begin to understand the contexts in which your customers work?


Do your automated tests design themselves in this way?

Do they have an awareness of feelings?


Humans are awesome at creativity, intuition, and improvisation

A quote from Marina’s article:

“……. the comparative advantage of humans is in doing things spontaneously, responding to unique circumstances of the moment, and making decisions accordingly.”


Do your automated checks respond to a new error and design a new test on the fly?
Do your automated checks create a new hypothesis on spotting a path through the system not previously taken?
Do your automated checks make use of other tools (that you hadn’t told them to use) to test a new path of interest?

Replacing the Tester

A manual tester can be replaced by a machine. They absolutely can. But that wouldn’t be a good strategy if that Tester thinks, connects emotionally and in general, uses their skills, experiences and crucially, their brain.

So you were worried about your job being replaced by a machine? You should be if you do rote checking with no thinking, but not if you do Testing.

A great quote from the essay tells a better story than anything I could cobble together with tenous links and parallels.

There hasn’t yet been a technology that has resulted in our working less. This is because machines don’t just replace what we do, they change the nature of what we do: by extending our capabilities, they set new expectations for what’s possible and create new performance standards and needs.


Although I’m not so sure that sentence is absolutely true, look at how the unemployement rate went up when robots entered the car industry, it does offer a really interesting way to frame the use of machines; as amplifiers of our skills, abilities and approaches.

But does the risk of replacing menial, rote jobs mean we shouldn’t advance the use of machines and robots in more domains, like Testing? I’ll leave that for your own judgment and feelings but it’s clear machines are empowering many to achieve new heights of Testing ability.

I’ll leave you with a great quote from the article, which I would encourage you read.


“The combination of humans partnering with machines and using superior strategies opens up new worlds for exploration.”

The basement is much crowded, but there is plenty of room up-stairs

The other day someone sent me an email asking me how to stand out from the masses in the Testing world. I responded by suggesting they engage in the community, join groups that interest them, read about any other subject that interests them but isn’t directly considered a Testing information source and to start learning and practicing Testing.


Image courtesy of :

There was some quote or reference that was nagging me though which I felt would sum up this delima.

The following popped in to my head this morning at around 4am..I believe this was the quote I was searching for. It was from P.T.Barnum in his book “Art of Money Getting Or, Golden Rules for Making Money” which I think sums up a commonly held view about Testing. It is a conversation between two people.

“I have not yet decided which profession I will follow. Is your profession full?”

“The basement is much crowded, but there is plenty of room up-stairs,” was the witty and truthful reply.

As more people flood in to this industry we are seeing a devaluing of our role and are a-wash with testers, many of which appear to all have the same skillset.

I know there will always be a market for interchangeable test “resource” but believe me, many people I speak to are having serious problems recruiting good testers.

It’s not from a shortage of applicants, but more from a shortage of talent.

So as we see the “basement” being filled with even more testers, with an even more bewildering set of certifications and best practices I can’t help but wonder what people will start to do to differentiate themselves from the masses in the future.

Do you think our profession is getting full?

Do you think there is a lack of talent or are companies becoming more specific?

Do you think it’s a lack of talent, or an unwillingness to be open minded about Testing?

Do you agree that there is a shortage of excellent testers? (I know this is subjective)

How do you think Testers could start to make a difference to their skillsets?

Duplicate Post. Duplicate Post.

The other day I posted out a blog called Duplicate Conversation. It seemed to hit a chord with a few people, which was great and a number of tweets/emails/comments came in thanking me for some advice on how to deal with the duplicate conversations.


But I’m afraid to tell you I was wrong. That post was always going to be a lead post to a more complex series of posts on communication. As Michael Bolton rightly pointed out in a comment is that communication doesn’t take place unless the intended recipient is involved. 

And that’s why the last post is wrong. I mixed the words communication, message and conversations intentionally, but at a deeper understanding level, they are two different things.


The communication is not taking place more than once, because we are conversing with different people. Therefore it is a different communication thread, stream or session. In a sense, my last post technically talked about repeating the same “message”, not the same communication.


When we start to take the analysis down to a level below what most people typically care about we start to see some interesting, and sometimes complex, processes and influences taking place. I also need to tidy up the language use and start to label things in a more, dare I say it, academic, fashion.

It’s not my intention to do too much of this sort of “academic talk” here; I have those types of conversation in “communication groups” on different forums. They make my head spin, and I intend to keep this blog above that level of discussions…mostly. But I do feel it would be important to clarify a few things about the intended message of the last post.

I have very few teachers and lecturers that stand out from my formal education but one of the most respected and trusted was my Communications lecturer and all round Guru – Noel Williams. He described every communication as having a “Purpose and Audience”.

He is a fantastic thought leader and his knowledge and insight to communication studies opened my eyes to a whole world of research and influence. He described every communication process as having an audience and a purpose, whether we know about them or not, or care about them also.

I’ve recently started adding “context” to that too, as I believe the environment/device/emotional state when you receive the communication is also a major factor in the success of that communication.


And so when I write, I define my target audience and my purpose, and then I use the appropriate language (as I perceive it) to describe my thoughts. This works fine when I get my target audience right. When I have multiple audiences with different levels of expectation…well, this is a lot harder to get right.


But that’s what fascinate me about communication. There’s more to it than just putting some words together and transmitting then in a medium.


You only succeed in communicating a message when the intended audience receive and understand your message. So in my last post I talked about repeating the communication. Not strictly true. I was talking about repeating the message to different people. This forms different communication threads or instances.


To complicate matters further the message you send is also susceptible to noise. But even more importantly is the fact you have to encode that message in the first place. And guess what? The person receiving the message also has to decode that message.

With encoding, transmitting, noise, receiving and decoding, there’s an awful lot of potential for ambiguity, lost messages and incomplete meaning. Specs? Tests? Reports? Meetings? <– They all suffer from this problem.


Here’s a diagram I like to use to show communication at a simple flow level. It’s schramm’s Model of Communication..there are others.


When I mentioned that Key Stakeholders were not present in meetings and other people took away different meaning, I was talking about exactly the above. Michael rightly pointed out:

“If the key stakeholder wasn’t present, communicaton didn’t happen for him or her. Meanwhile, the people who take away different actions and conclusions do so because they believe that communication has happened, but they interpreted what you said differently from your interpretation, apparently.”

For sure, the right audience wasn’t there and the message you thought you communicated, either got lost, got altered through influences of noise or was simply decoded in different way, by different people. Michael is right, communication never really took place with the stakeholder; the person you actually needed to receive that message.


Even the above skips out loads of technical stuff and I’ve simplified it far too much still but I hope it clears up a little about communication. I’ll be exploring a few elements of communication and sociology over the next year as I start to build my learning in this areas in to Testing.

Work Experience and Work Placement

I’ve always tried to appreciate ‘context’ when I talk about Testing and also when I Test because ‘context’ is a very real thing. I’m an avid campaigner against Best Practices in Testing and I take every opportunity possible to question blatant “context unaware” statements about testing, especially so when they are communicated as “law”.


Yet it’s frightening how few Testers are ‘context aware’ when fine tuning ideas and talking about Testing. There may be many who never need to care about any other contexts or environments. There are some who know about other contexts but really don’t care. Then there are those who know about other contexts but don’t have a chance to explore any further than that.

Awareness is a good start. Appreciation is a further step. But what about actual “experience”?


Wouldn’t that be cool if you could “experience” another context for a number of days, maybe weeks or months? Experience first hand what it’s like to Test in this environment alien to yours, whilst you work alongside other testers who work in these contexts everyday? A kind of Tester swap?


Here’s why it *could* work.

  • Testers could gain a massive insight in to other contexts and other ways of working
  • Testers could gain experience working in contexts that they would never normally get to work in
  • Both sides would benefit. One side would get experience. The other side would get an extra set of hands, or some fresh eyes for a few days.
  • It would be a fun and interesting way of sharing knowledge
  • If you have a large company with many Test teams then it could work internally as some sort of exchange process. (Thanks to David, in the comments, for suggesting this one)

Here’s why it *might not* work

  • Confidentiality, privacy and the fact it’s a new (and potentially scary) idea for some
  • The costs involved. (Travel, accomodation, etc) <– or should this be self funded?
  • Regulatory (induction, health and safety, security compliance etc)
  • Logistics (self organised, run by a community like The Software Testing Club or ad-hoc?)
  • Some companies *could* take advantage of the scheme to get extra resource for a period of time
  • Lack of benefit from those placed if they get lumbered with sketchy jobs and checkbox testing
  • The system under test may be complex and/or complicated enough that a Tester may not have the chance in just a few days to add any real value. (Thanks to Kate, in the comments, for suggesting this) – I still believe the Tester would get to see another context, but the company offering them this opportunity may see little value..maybe.

No doubt I’ve missed some blindingly obvious pros and cons and I suspect a project like this would be bigger than I expect to get started….but it would be cool…right?

Your Numbers Up

There comes a time when we all use numbers to represent some fact or information. These numbers *can* be highly effective at communicating your information.

number buttons


Image courtesy of :

These numbers could become the defacto representation of some outcome, fact or finding. These numbers could become the metric you make a decision by. These numbers could start to guide your day to day work.


At work, we use numbers like these all of the time. We use the project velocity. We use the number of open bugs. We use the number of tests that fail testing. We use a whole army of number to represent information like the top ten applets used in our system to the number of end user using a particular browser. We use them as indicators, not the law. They guide, not force.


It’s important to remember though that these numbers are just that. A number. It’s not the number that’s important, it’s how you feel about that number that matters <– I believe it was Michael Bolton I first saw using that phrase.


Sometimes I look at a number and wonder why I use it. Because I always have?


I’m no longer accepting that so I’ve dropped or changed a large number of metrics recently because they are no longer adding value.


Yet many people use numbers, with little meaning, to guide their testing, their releases, their company strategy.

Like my metrics and numbers, they may once have had meaning. They could once have been a way of locking down a fact, or a snippet of information or even a whole business process. Then our numbers grew. We collected more of them. The data got bigger. The variance even larger. They lost any symbolism they may have once held. They lost their context and no longer describe the idea or concept they once did. Time and business moved on and the numbers didn’t keep up.


But they have become the norm. They are the defacto number. They have not been re-factored, or re-worked or even questioned. So they still guide testing. They still inform key decisions. And the scary part, they may be wrong.


As Testers I believe it is our role to treat any metric or number with a little skepticism.  We should question it’s use or at least ask more about its meaning.

Duplicate Conversations. Duplicate Conversations.

In almost all places of work, there exists a level of wasted time and effort when communicating a message multiple times.

In almost all places of work, there exists a level of wasted time and effort when communicating a message multiple times.


These “duplicate conversations” are wasteful and ultimately take people away from the work they are trying to complete.


Close up on the pic 79/365 playing the Whisper Game

Image courtesy of :

We need to communicate to our “communication stakeholders” (team members, project, sales, marketing etc) but this communication should not need to be repeated multiple times.


Duplicate conversations can occur for a vast number of reasons, but often it is as simple as key communication stakeholders dropping by for updates. Only to be shortly followed by another stakeholder. And then another. Soon you find the whole morning has gone and all you have done is update people. This is especially so when the management style is to “manage by wandering around“.

Meeting are a ripe source of duplicate conversations, typically arising due to a key stakeholder not being present in the meeting or people taking away a different set of actions or conclusions from the other attendees.

Different teams working on the same problems can often result in duplicate conversations, especially when they aren’t talking to each other during the development and are both updating the stakeholders individually.

No designated “Panic Manager” or “Point of Contact” can result in the whole team being involved in a bug investigation when it could easily be dealt with by just one person.


Repeating the communication once is bad enough, but to repeat it several times is very trying.

Even worse is the predictable amount of panic these multiple conversations can cause. This is especially so when it reaches a person of “power” after travelling through a series of Chinese Whispers (other people repeating the same conversation but missing out details or adding in details not included in the original message).

It can sound like a full scale project meltdown by the time the last person in the chain hears about it. Emergency planning meetings are called. People are furious. You’re wondering what went wrong. Panic and flapping over nothing. A storm in a tea cup. A miscommunication.


Communication failures I believe cause an unprecedented amount of wasted time, unneccessary panic and general grief for all involved.


So what’s the solution?


Each situation will be different and there is unlikely to be any one single answer to any communication problem but what follows are a few suggestions.

1. Hold a daily stand up meeting (where applicable with team size) and encourage all members to be clear and accurate in their description of what they are working on, what they are blocked by and what they have completed.

A standup (like any meeting) is only valuable if people are going to actively communicate. Too much information and people switch off. Too little and you’re losing the value. And no, they aren’t exclusive to agile projects.


2. Take “minutes” in meetings with clearly defined Actions. Read the actions out at the end of the meeting to ensure everyone understood them and took on board those actions they were tasked with. Communicate the minute notes immediately after the meeting in your preferred format of communication.


3. Use information radiators (low tech boards or areas or pages used to communicate information). Make sure the information is simple and easy to understand. Making your information radiators tricky to read is no good. They need to communicate the correct information, to the correct audience with the least chance of noise creeping in or misunderstanding of the message.


4. Dedicate someone (or more than one person) from your team to be the main point of contact for issues, questions and panics. Issue do happen so it’s prudent to make sure you are set up to handle them in the most appropriate fashion to avoid distraction and interuption and also to ensure the best possible response to any customer issues.

It’s also incredibly expensive to have a large team investigating an issue, especially when that issue turns out not to be an issue.


5. Organise cross team meetings or standups or chat channels. Use whatever communication medium floats your boat to ensure different teams are talking to each other when working on the same area, or problem or feature set. It’s crucial to start the communication process and try to keep the channels open.

Otherwisey you could find you end up with a situation Melvin Conway was describing in Conways Law.


6. Fine tune your process. If you find yourself repeating the same message to multiple stakeholders then stop doing it and work on the cheapest and lowest tech solution to the problem.

In my experience, the more expensive, integrated and purpose built a software communication tool is (like Test Management Tool), the least likely it is to solve your problem.

You cannot beat face to face communication for gathering the non-verbal subtleties and clues, but if that’s not possible then what about a good information radiator like a status written on a white board?

Or a Chat Group where the discussions and outcomes are already audited and exportable for future reference?

Or a free wiki? Or a blog?

Look for the most cost effective way to solve the problem and keep iterating around the problem until it goes away.


However, none of the above will solve your communication problem if the right people aren’t kept in the loop and the people involved aren’t communicating the right messages.

This is a much harder problem to solve.





Game Storming and Testing

I’m reading an excellent book at the moment called “Game Storming” by Dave Gray, Sunni Brown and James Macanufo.


It’s one of those books that’s given me a “lesson” or insight in every chapter.


In essence Game Storming is about generating new ideas and novel solutions. It’s a way of getting from Starting Point A to End Point B, where B is unknown or “Fuzzy”.


If the journey from A to B is quantified, mechanical, industrial, then a series of processes can be used.I guess you could call these Best Practices or Business Process. This is not what Game Storming is about. Game Storming is about exploring and journeying from A to B.


The authors talk about a Game World coming to fruition as:


  • Imagine the world
  • Create the world
  • Open the world
  • Explore the world
  • Close the world

A defined and repeatable game would just be:


  • Open the world
  • Explore the world
  • Close the world

I’m not doing the book justice in these brief descriptions. I’m just setting some context.


Every game has an opening, a game/exploration and a closing.


What’s this got to do with Testing?


Well I believe it has a lot to do with Testing, especially if you do a lot of Exploratory Testing and/or Problem Solving.


I urge you to read the book for it holds many insights, ideas and a much more detailed description of the elements of Game Storming.


The authors very kindly provide ten toolbox items to aid in designing Games. I’ve headlined the ten here, but again, the book actually offers exceptional descriptions and insights, which I’m not going to talk about here. Here’s the ten ideas with how I believe they relate to my (Exploratory) Testing.


1. Opening and Closing


In a nutshell Opening is about defining the Goals and Objectives. These should be tangible, emotionally connected to myself (i.e. in line with my motives and passions and skills) and have tangible outcomes (defects, further test ideas, information, videos, screenshots, notes).


The Opening part is Divergent. It’s where I list all of my ideas and thoughts about what to test. It’s about coming up with things to try, areas to explore and hypothesis about the system under test. It’s where I do a brain dump of ideas. It’s a melting pot of suggestions. Some good. Some bad. It’s the definition of the Charter.


Closing is the end part of the game. It’s where I pull all of my data together and define some actions, some further steps and some outcomes. It’s my Test Report at the end of the session. It’s the defects, further tests and other artefacts.


The Closing part is Convergent. It’s where I bring all of the information and insights together around the Goal and look at who to communicate these findings too, if necessary.


2. Fire Starting

Fire Starting is about igniting your mind and imagination to get those ideas flowing.


I typically use a Mind Map for this stage. The brain dump happens in the Mind Map. It’s a great medium for jotting ideas, diverging the ideas and linking topics together that traditionally have been kept separate.


I typically create a Mind Map during story kick off meetings; jotting down ideas, scenarios, questions, hunches and the different elements involved (performance, usability etc). I also use Mind Maps with an opening question to generate ideas. Something like “How many different ways are there to end a phone call?” or “How do Statistics cope when I transfer the call to multiple end points?” (I test Call Centre software).


3. Artefacts

These are the objects that hold information about my game (session). These include my original Mind Map of ideas, the test charters and notes, the videos I collect of the session, the defects and associated reports, the information I find and the conclusions I come to.


The mediums for these artefacts are not really important, but having an environment set up with all of these tools and mechanisms available instantly is essential. I don’t want to be breaking away from what I’m doing whilst I find a tool or medium to store the appropriate information.


The video recording is particularly useful as I branch off down new routes and ideas, feeding off the feedback that the system gives me. I have an end goal, which at times is Fuzzy, but the journey to the endpoint is important. It needs documenting.


4. Node Generation

The ideas that I create naturally tend to fall in to Nodes in the Mind Map. But a Node could be a container for post-it note ideas for example. A Mind Map naturally brings out the Nodes and associations, but as with all ideas generation, it could be that an idea sits under more than one Node.


Artefacts would also fall under the Nodes as I create further Test Ideas, Test Data and new information.

5. Meaningful Space

“It’s a way of framing any space to make the relationships within it more meaningful” A great example is given in the book about Chess. The pieces and players in a chess game are important but without the meaningful space of the board (grid) it makes no sense.


For me, the meaningful space when I create my Test Ideas is the Mind Map (or other medium for ideas generation). It could be a white board with stickies, or even an electronic system where each idea is deemed meaningful compared to other ideas, or customer reported defects, or business critical features etc.


When coming up with ideas you need to restrain your meaningful space to allow your creativity to flourish. This is especially so in bigger teams where a lack of restraint can take you miles away from your Goals. I still prefer a Mind Map for ideas generation, even in a collaborative team setting. The medium of Mind Maps seems to naturally create meaningful space.


Each day I view the meaningful space that is our Kanban board with our tickets and work on it. Take away the board and associated columns and it’s just a selection of stickies. The meaning comes from the columns and headers and how the stickies are located. A Test Idea on it’s own is valuable but combined in the meaningful space of a Test Lab or associated with other Test Ideas it can become ever more meaningful.


6. Sketching and Model Making


I work in pictures. I like to Visualise everything I’m brain/game storming. When I’m working with my Test Team or the Programmers I tend to draw pictures. Sometimes the drawing are very useful on their own as a piece of communication. Other times they would not work on their own as a single piece of communication, but they do aid in clearing my thinking allowing my verbal language to become clearer and more concise.


Here at work, when we talk about process flow or design or even deadlines we doodle, sketch or draw. We describe process ideas to each other in pictures. In fact, we even have a small selection of drawings that represent ideas and process.




Yes, that is a picture of some cats. No, it’s not a conveyor belt. And yes, we do have obsessions with Pie Charts.

We sketch out designs, relationships between components and user interaction. When I’m testing Call Centre flows I tend to draw pictures including people in them. This helps me see the flow of the calls more clearly. It allows me to use the image for guidance freeing my mind for further ideas and divergences. I can therefore see the flow and concentrate on other things.




Again, Mind Maps are diagrams that help to visualise the relationships between Test Ideas. I can mark Static Tests (Checks) and Exploratory Tests (Tests) easily on the Mind Maps so I know where we need to concentrate automated Checking and where Testing is needed. They form a natural Sketch of the ideas and work.


7. Randomness, Reversal and Reframing


If we look at the world in the same way, all of the time, we will develop a blinkered view, biases and consistent patterns that may generally serve us well. This is good for many circumstances, but when we are thinking of new Test Ideas and looking to diverge from the “Norm” it can help to reverse these views or reframe them. It tricks our mind in to thinking differently.


One of the ways I do this in an Agile environment is to look at the user story and reframe it.


We design stories like this:


As {Someone}
I want to do {Something}
So that I get some {Value}


What about reframing the story as {Someone} else?
Or mixing stories by combining the {Someone}, {Something} and {Value} from different stories?
Or how about thinking about all the other ways you could obtain {Value} without doing {Something} or doing something altogether different?

You’d be amazed at how many test ideas this can generate.


8. Improvisation


At first I disliked this suggestion in relation to Exploratory Testing because I didn’t deem testing to be improvised. Yet upon reflection and observing my own Testing I noted that I do improvise a lot. If I have a “hunch” I will use a tool to aid my testing. If the tool doesn’t exist I’ll try to make one or get some help from the Programmers.


I will react to new information at all times, plotting my journey as I go, always mindful of the end Goal or Charter. And I guess, you could call this improvisation, but I suspect it’s more exploration.


Yet I improvise with tools, note taking and ideas generation. Drawing a line between actual Exploratory Testing and Ideas Generation is a tough one for me, I think the ideas and the Testing combined is the true value for me.


Without good ideas I’ve always felt I’ve been left with Checks and not Tests. I’ve always felt like I’m confirming, rather than exploring.


9. Selection

As the authors point out “you can’t do everything” so a way to prioritise and select is required. When I list out my Exploratory Test ideas in a Mind Map I see patterns and groups starting to emerge; these could naturally be a selection criteria.


I believe in adopting a Wide Awareness Field which means opening my horizons to other elements of the business, the testing world and the subjects that interest me outside of work.


That’s why I try to find out what’s happening in Sales, Marketing, Support, Delivery and Operations at work.

  • Are we seeing some trending user patterns from sales enquiries?
  • Are we seeing a growing number of usability issues through support?
  • Are the delivery team seeing problems with providing excellent service?
  • Do we have an area of the product that consistently proves troublesome?
  • Have we had some changes in an area that would warrant more Testing?


There is something the authors call Forced Ranking where you are forced to put the ideas in an ordered list with each idea having a unique rank. This is exactly the same technique many people use managing their story flows and product backlog. A list of the things we want in the product, ranked with the most important at the top.


I’ve tried this with my Exploratory Testing, but I’ve not yet found the best mechanism for handling this. Pivotal Tracker is excellent, but I’ve not yet found the right flow with this tool with respect to Exploratory Tests.


I suspect that with a growing Test Team we’ll start adopting a white board and sticky note approach to make it all visible to the whole team and business.

10. Try something new

The authors suggest simply trying something new with each Game.


It’s something I try to do, whether that be a new Heuristic or Mnemonic or a new Testing tool. I may try a new User Scenario or a new browser, or simply just approach my testing at a different time of the day. It’s amazing how your mood and energy levels effect your Testing.


I like these ten points as a way to start generating new Test Ideas and Conditions, as well as giving me a basic set of tools to analyse how I play games and how I use some Game Storming techniques during my Testing. They seem like a great starting point and the book goes in to much more depth about what is a Game and how to manage your Games. I found lots to learn in the book. It gave me a lot of ideas for creating the right environment to let my creativity flow. So….


How do you generate Test Ideas?








I hate it..but I must admit I’ve not tried it

It seems the more that something becomes popular the more that people start to hate it, often without even trying it. Look at what’s happening with Agile in the testing community. A lot of negativity from people who’ve never tried it. The lashing out is usually based on false ideas of what it actually means to be a tester in an agile context. A good example is a classic and incredibly misguided BCS blog post here. The comments are invaluable…and funny.

I see the same thing in the social media arena too. Lot’s of people berating twitter, blogging and facebook as nothing more than a passing trend. Yet for many it’s bringing them closer to their real customers and end users. For some it’s building long lasting friendships and relationships. Giant circles of contacts. Extending peoples reach, making the wider community available to us all.

The detractors seem to be those who haven’t tried it, or dabbled and didn’t like what they saw. It is those who see it from a distance and critique it, often with misplaced or over hyped information, often without understanding how contexts and environments play a role.

Look at how some people react when you mention acceptance test driven develoment or exploratory testing.

They offer the same look my careers advisor once gave me when I told her I wanted to be a Ventriloquist when I leave college.


Some people go mad at how ridiculous these ideas are. “Where’s the quality” they shout?

New communication techniques and social media channels give us a platform to challenge the one sided stories we’ve been hearing for years and years. The empty arguments and global testing norms can be challenged, globally, locally, glocally. We can discuss these so called Best Practices and expand our sources of information to get a better picture of what may or may not work for us. We no longer have to go to the mainstream for our information.


I heard someone at a Testing Conference criticising Virtual Machines when used for Load Testing because they felt they weren’t reflective of real systems.

Yet they spent thousands of pounds on expensive build environments, big teams to manage these environments and they then had to wait two days to rebuild this system after a failure.

The interesting part of this story was that the “real” system the presenter was Load Testing was indeed virtualised anyway. What? A live system being run on virtual machines??? Never.

It’s a bit of a rant this post, but really……if you must criticise, reject or belittle something, then criticise something you have at least tried. 

Communication. Signs. Symbols.

I’ve studied communications for years now but today was the first time I encountered the video from that I’ve included in this post. Which is a shame because it’s fantastic. At the same time I’m glad I’ve stumbled across this video now so that I can share it with those who are interested.

Communication underpins everything we do. In a sense, we cannot do anything without communicating something, so it’s important we understand the implications of this.

For those that follow this blog you’ll know I’m interested in simplifying the discussions around communication by elevating the deeper rooted theory in to a simple three thought process. Purpose. Audience. Context. It’s a starting point to think about communication and how well we communicate but is no means complete.

This video does an excellent job of explaining the deeper rooted theory, particularly focussing on Shannon’s model of communication but also touching on signs, symbols, culture and noise. It’s a great video and carries that typical 1950’s haunting aspect of informational videos. I love the images, editing techniques and footage and the soundtrack just tops it all off. But despite all of this retro greatness there’s nothing more important in this video than the content.

It’s a long video at 20 mins but well worth sticking through. Some highlighted phrases stood out from a testing point of view.


English language is one half redundant


Computers are often referred to as brains….The greatest fallacy in the comparison is one of degree.



The Peltzman Effect

I find The Peltzman Effect incredibly interesting from a Testing point of view.


The Peltzman effect is the hypothesized tendency of people to react to a safety regulation by increasing other risky behavior, offsetting some or all of the benefit of the regulation” (Wikipedia)


I find it interesting because I wonder whether we see this (or some other closely related theory) when we build secure systems or create safe ways for people to interact with our applications. If we provide an application that is secure and/or communicate outlandish safety and security claims, do our users exhibit less secure behaviour when using it?


I have a friend who interpreted “The most secure ISP provider on the market” to mean that he no longer needed Anti-Virus or a Firewall on his laptop. He had his credit card details stolen 2 weeks after signing up.


I know of someone else who fell for the same phishing attack 5 times. He put his trust in the new “Incredibly secure online banking portal” to protect him.


I know of systems that are incredibly secure yet send out all login details via just one mailing through the post. Get the mailing, get the credentials, get the money.


I know of people who keep their PIN numbers in their wallet with their cards.


Some people use obvious passwords like 1234 or 0000 or Pa$$word – check out the iPhone password stats here.


And so as a tester I always take a step back when it comes to security and look at the wider picture.


Experience has shown me that it’s the human that’s typically the weak link in any security process or application usage. And I can’t help but wonder whether this same human, when bombarded by claims of high security and safety, becomes an even weaker link in the chain.


Maybe it’s the Peltzman Effect? Maybe it’s something else? Maybe it’s nothing at all?


But I reckon all testers would benefit from drawing/sketching out the whole process (including the human elements) of your system under test and ask yourself one question:


“How can the human parts of this process compromise the security?”


Some light reading :


Are we halfway there yet?

I got asked a question the other day about metrics.



“If you don’t use metrics to assess test completion, how do you know when you are half way complete?” 


I won’t go in to all of the details surrounding the discussion that ensued, but I thought I would share with you the two stories I use to help testers understand the potential flaws with using metrics to assess completeness and project deadlines. 


Story 1 – The Regression

I used to work in a large team that would have a massive regression testing phase at the end of each monster Waterfall project.

The way this was managed was by printing out every single test case we had in the “regression pack” and putting them in “feature” stacks on a long table.  

The test team would then come in to work and grab a load of test cases and blast through them. The management insisted on everyone completing around 10 tests per day. 


The deadline was therefore estimated on the number of tests we had, the number of testers we had and the fact each one could complete 10 tests per day.

Unfortunately, it never quite worked that way. Here’s what would happen:

  • Some testers would roll in to work at 6:00am to grab all of the easy test cases. 
  • Some testers would roll in to work at 9:00am and be left with the rock hard, complicated or tedious tests.
  • Some testers would complete their 50 test cases (they would grab an entire weeks worth) in one or two days and spend the rest of the week exploring or learning or surfing the web.
  • Some testers would struggle to complete more than 1 or 2 tests per day because of the complexity or setup time.
  • Some testers would find a Boat Load of bugs from exploring which would bring the whole project release in to doubt.
  • Some testers would pass tests without even running the test case. After all, a bonus was paid out to those who completed 10 per day!

The whole process was flawed because it gave testers a metric driven system to game.


The management were not overly concerned with good testing and instead craved metrics to report further up the chain. It therefore didn’t work.

The above story shows a few things:

  • Not all tests are created equal. Some are harder, more complex, more tedious or more time consuming that others. 
  • Metrics will very rarely tell you how complete you are. 
  • Regression testing by simply re-running a load of already executed test cases is a flawed idea of regression testing. Automated tests and Talented testers doing exploratory testing is better <– I’ll save that one for another post.
  • The “switched on” testers will always find a way to game the system, especially if you add incentives based on numbers alone.

Story 2 – The Fuel Tank

 I used to own a tidy little Toyota MR2 Mk1. I loved it. A Classic. 

One thing I noticed about the MR2 (and every other Toyota I owned after this) was that I would get fewer miles from the top half of the tank than the bottom half.

“How can that be a half?” I hear you shout.

Well, technically, it wasn’t. But to be precise with petrol tanks and mileage is to assume that the fuel gauge in the car is 100% accurate…and needs to be.

I now own a Seat and I get fewer miles from the bottom half of the tank.




Well here it is. Each car does around 300 miles from a full tank.


When the MR2 said half full on the dash indicator I would have covered about 100 miles of the 300.

When the Seat said half full on the dash indicator I would have covered about 190 miles of the 300.


Yet both would do 300 miles. For those who care, this is down to the shape of the petrol tank. For some reason, in some cars, it is easier just to say you are halfway down the tank height, than to actually work out how much petrol has been used. I’m sure some cars are very accurate thought by the way…



But it doesn’t matter, because I know how many miles I can get from a tank (roughly) and I get a light on the dash indicating I have roughly 50 miles left before I run out of fuel.

They are all indicators. They are all guides for me to make a judgement.

And used in that way they are very useful indeed. 


And this is the same as defect and test case metrics. They are a good indicator, but in most cases should not be used as an absolute.

Metrics aren’t always evil. In fact, they can be very useful indeed.


But I would always suggest you think deeply about what your metrics are reporting, to whom they are being reported and about whether or not that message could be misconstrued.That way you may find that you can drop some metrics, fine tune some others or maybe start collecting a different set all together.

Mastering Testing


In the Testing world, as in any other domain, there are those who are passionate about their work, those who go to work to get paid and those who just don’t care (and a number of people at varying points on this extreme sliding scale). Scarily I see a significant number of people looking for the quick win approach to obtaining skills and excellence in testing.


In reality there is no way anyone will ever attain complete mastery. There is always more to learn. But to not even head down the path to this elusive mastery is sad. To think you can attain mastery through short cuts, certifications and snake oil training courses is tragic.


To obtain excellence and to head down the path towards mastery will mean different things to different people, but I’m fairly sure that it’s only ever attainable through hard work, practice and more practice. Or is that just a little old fashioned?


Image courtesy of :