How to run a proof of concept

When seeking out new tools to aid your testing I strongly believe in doing a proof of concept. It needn’t be anything heavy weight.

In fact, the more concise the better.

What is a proof of concept?

I use the term “proof of concept” to describe the process of evaluating a tool (or service) for suitability against a number of criteria.

This criteria should be specific to your context, although some of the criteria would probably apply across many contexts.

Note: from this point on I will use the word tool to encompass both tools, services and other paid for systems.  Why perform a proof of concept?

I believe it’s important to run a proof of concept for a number of reasons.

Short term reviews of products and more specifically, product specs/marketing material, can lead to a snap decision that might not consider a wider view. This is even more a problem when you have a salesperson pushing at you to make a decision.

You don’t have to look far to find many companies and organizations trying to sell you a solution to each testing problem you may (or may not) have. Some of these solutions are great and right for you. Others are not.

If you don’t spend time trying the tool to see if it solves your specific problem then how can you be sure the product is right? How can you even be sure that the claims in the literature or sales meeting are even true?

Tool usage can often involve people from other departments, either directly or indirectly, so it’s only right to collaborate with them also.  (for example, a bug tracking system is not typically used just by testers).

The initial pricing descriptions or quotes may not actually be the end price you pay after adding on extensions and support contracts, etc. This is a very compelling reason to find out what the tool does and what it doesn’t do before eagerly signing on the dotted line.

The tool needs to solve a problem, but it might cause more in the long run. Snap decisions to solve one problem could leave you with a bigger problem (i.e. -training needs, hardware requirements).

Comparing a number of tools can often lead you to greater insights in to the problem you are trying to solve and you may generate creative solutions to it which might not have come about without proving the tool.

My memory isn’t what it used to be so documenting the proof of concept and having comparison data to refer back to leads me to a more accurate decision.

There are many other reasons why someone might perform a proof of concept, the above are just a few that immediately spring to mind.

When should you run a proof of concept?

In my opinion any tool adoption that impacts a wider audience than yourself should have a proof of concept process.

For example, my decision to start using Rapid Reporter rather than NotePad only really changed my working arrangements. So I didn’t feel the need to perform a proof of concept.

However, a decision to start using a new Test Management tool might impact the wider Test team so this could be a good candidate for a proof of concept.

How long should a proof of concept take?

There is no right or wrong answer. It depends on the tool, the number of people involved, the cost of the tool and the infrastructure to name a few variables.

I try to keep my proof of concepts as short and time-boxed as possible. This focuses the mind and encourages better planning in advance.

I would suggest that if your proof of concept is taking months then you have a problem. Either you are not pro-actively involving the right people and making use of the time available, or you’re trying to assess every angle, feature, component and pricing model of the tool. This *might* be a good thing when opting for the big vendor wallet splitting tools, but for most tools I’d suggest it may be overkill.

How to prepare a proof of concept Scenarios and Problems

My first step is to write down the problem(s) you are wanting to solve. Writing them on paper helps to articulate clearly the problem space. It helps you to work through the problem in your mind.

For this I typically use a mind map, but that’s my own personal preference. Use what works for you.

Once you have articulated your problem clearly you will probably find a natural set of criteria emerges that you can use to judge the tool against.

For example – you might want a tool to aid your test management. You might have a desire to report the number of bugs raised versus the number closed per sprint, or other time scale.

From this problem statement you automatically have some checks to run. (i.e – does the tool generate that report?)

If the tool does not solve your problem and meet your requirement then either move on or re-assess how important that criteria (in this example, the report) is to you.

In defining the problem and some typical uses of the tool you will start to build a handy checklist. Don’t go mad though, you could spend months assessing a tool against lists.


It would make sense to find out what your budget is. Your budget constraints will naturally lead you down certain routes.

There is no point reviewing a big vendor tool unless you have a stack of cash to throw at it. As a manager though, be cautious about blowing all of your budget on a big vendor test management tool.

Firstly, there are many tools for less (or free) that do the job just as well (if not better).

Secondly, there will always be something else to spend your money on further down the line.

Is support included?

It is always worth checking if support is included in the price. And if so, what level of support that provides.

Are there any SLA (service level agreements) against this support contract?

Do you need support?

Are upgrades and maintenance releases included in the initial cost, or charged per upgrade?

I once worked with a Test Manager who bought a giant vendor tool for many thousands of pounds but failed to consider the upgrade licenses. The tool vendor released the newer version about 4 weeks after we started using it, and the newer version obviously had more glossy features and many bug fixes…which we then had to pay extra for.

Always check the pricing structure clearly.

Are you going to need any plug ins or extensions to make the tool work?

If so, how much are they?

How easy are they to use?

Are they widely supported?

Can you try these as part of the free trial?

Do I, or the team, need extra training?

If so how much and is there a cost associated with this?

Would a book suffice?

Are there any user groups or online communities that can help?

What skill sets do we currently have?

Are there open source tools available?

You might find that there is an Open Source tool available that does 99% of what you want with no license costs.

Could you roll your own tool from the Open Source code?

There may be three or four tools that solve your problem available with open source licenses. Could you plug them together with some bespoke code to create a good solution?

Do you have the required infrastructure and hardware?

If not, how much will it cost to get it?

If yes, what is involved in getting the product running? Do you need help from others in the business?

Are other departments or teams looking to use this tool too, or are they impacted by it?

If not – are you sure?

If yes – have you spoken to them? Are they involved in the review?

How long are you planning on keeping the tool?

Could you find a creative work around if your problem is short term?

Are you 100% sure that you want to sign a contract for many many years? What about a SaaS model? Or one year contracts?

Does the tool support multiple ways of working (methodologies/approaches) or does the tool enforce a certain way of working?

It is always best to find a solution to allow for future growth and change.

Many of the old-school testing tools enforce a certain order or structure which could hinder your growth or stunt your productivity.

Be sure to assess the tool from your point of view, not the point of view communicated by the literature, sales person or wider community. The tool needs to work for you.

What is the deadline for deciding?

Do you have a deadline that needs to be met. This will give you a constraint, which will help to focus your attention.

Can I try the software?

If not – why not. Come on, it’s 2012, it should be possible to try before you buy, even products that need to be installed client side.

I would suggest you download or sign up to as many tools as possible and run each one against any acceptance/proof of concept criteria you have.

Final Thoughts

The above might help trigger some thoughts for you, or you may already have your own list, but I strongly believe that time invested in the proof of concept could be time saved (and/or cash saved) in the long run.

On a general note I would look for a wide selection of tools and try as many as you can. I wouldn’t just follow what the industry mainstream is doing. Focus on what you need and you’ll certainly find the right tool.

Good testers look behind the screen

The other week I found myself in an exquisite room with a group of like minded testers talking about testing. Exploratory Testing to be precise.

I’m not sure whether I’m allowed to spill the beans on why we were all there, so I wont..just in case.

Suffice to say though I was with some of the most interesting minds in the UK with reference to Testing. Throughout the day I jotted notes and took away a tonne of actions and things to research, but one statement from Steve Green (of Test Partners fame) really struck a chord with me.

We were talking about hiring testers and what traits/skills a good exploratory tester has.

Steve gave a great example of how many testers he meets are stuck looking at the screen and running tests to check/verify/test the capabilities and elements they can see on the screen. This is fine, for some.

For Steve though, he looks deeper than this.

He prefers people who look at the screen and then beyond it.

Behind it, under it, away from it.


For example, what happens to the overall system when I click X?

How is Y working?

What happens if I tweak Z?

I thought about this comment a lot and wondered whether this was actually an example of the difference between Black and White (or pink, grey, blue, tangerine, Alizarin crimson, black olive or …) box testing.

However, I think it runs deeper than this. I think that looking behind the screen is more about curiosity and intrigue (and therefore great testing?) than it is knowing about what the internal system should do.

It made me think hard about the different traits we all look for in a Tester and the words we use to describe these testers.

It was an illuminating day. I learned a lot.

An Introduction to Mind Mapping for more than Test Design

An introduction to mind mapping

Mind maps have long been an important part of my work, right from organising management tasks through to test design and ideas generation.

I’ve learned a lot about how my thinking unravels by using mind maps. Despite my success with them I’m conscious that not everyone works well with a mind map. Here are a number of ways in which mind mapping has helped me. I’d be interested to hear about other ways people are using mind maps, or other techniques for organising thoughts, data and information.

A sample mind map
A sample mind map

Mind maps for brainstorming, thinking and writing

I’ve been using mind maps for about 15 years now for brainstorming and aiding my thinking. I’ve never moved away from them. They’ve been invaluable in helping me to generate ideas and connections.

Most of my blog posts, books and other content comes from an idea which forms the central node of a mind map. The outline of “Diary of a Test Manager” was done in a mind map. I’ve got another book I’m working on that’s been outlined in a mind map also. Some of my more complicated blog posts start out as a random thought that is then structured, explored, exploded and refined in a mind map.

For those new to mind maps they work (at a basic level) like this – from Wikipedia.

mind map

 is a 


 used to represent 



, tasks, or other items related to a central key word or idea. Mind maps are used to 




, and 


ideas, and as an aid to 





solving problems

making decisions

, and writing.

Most electronic mind mapping software allows you to move thoughts (nodes) around, add notes, add hyperlinks, cut/paste and create relationships (links) to other nodes. If you are using paper then this flexibility is less of an option. It is possible to export (from electronic system) the map in a number of format such as jpg or txt.

For me the central idea for my writing typically forms the central point in a mind map. From this central point I can spawn a number of child nodes exploding the idea. I typically don’t self edit at this point, instead I choose to let the ideas flow.

After a while I will start to drag and drop the ideas in to logical groupings before applying edits to the spelling, typos and removing ideas that no longer make sense. New ideas will obviously come to light and can be added quickly and easily to the mind map.

I can quickly spin up new nodes, add web links, add resources and create to-do tasks.

Tool I use for this : XMind

Mind maps for test ideas

I use mind maps to document my thinking around user stories, features or capabilities that I will be testing. I will map out my ideas (no matter how crazy) and then regroup, re-factor and create associations later.

I typically do this during story chats and design meetings. I used to do this process of capturing test ideas in excel which was very effective but the re-ordering and associating process was trickier. It was also slower to brain dump.

I typically export the mind map to text and copy/paste this to excel to give me the reporting and ordering aspects that excel can offer. I know some testers continue to use mind maps for their planning, measuring and reporting too. I’ve found them less useful in this respect but my needs will differ from others.

Tool I use for this : XMind / Pen and Paper

Mind maps for Product Outlining

One of the first things I do when working on a new product is to map out the capabilities and structure of that product in a mind map. This essentially gives me a more visual representation of the entire system (and sub-systems). I can start to build a picture in my mind of how things hang together and what changes/testing might affect different parts of the product. During this mapping I will take notes and typically create another mind map with my test ideas.

I print this out and stick it on the wall for quick reference. (some mind maps don’t print well if they are large though).

Tool I use for this : XMind

Mind maps for Team “Things To Test”

Every team I have worked in has had a number of standard (or regular) areas to consider or components to check for feedback when testing. For example at NewVoiceMedia when we test we always check the “call recordings”, “audit log”, “call quality”, firebug outputs and of course the web server event logs.

We typically check these areas no matter what testing is taking place. These and many more areas give us feedback and clues as to the state of the system, even if they may appear to not be directly related to what we are testing.

These areas are documented in a mind map so that we can quickly see what areas should be consider. It’s a cheat sheet basically.

Tool I use for this : XMind

Mind maps for documenting meetings/gatherings and training/events

I use mind maps for documenting meetings and other gatherings so I can keep my notes short and concise. I have terrible handwriting so I tend to keep my writing limited to nodes in a mind map.

I use paper based mind maps for this task because I prefer the freedom a pen and paper brings when information is coming thick and fast from discussions etc. It’s also less annoying for other people in the meeting as I’m not tapping away at my laptop or mobile device.

One of my team, Simon, showed us in this weeks “tool time” meeting how he used the very excellent “The Brain” to document his teams retrospective. Another really good use of mind maps. BTW – The Brain is awesome….but pricey for the full licence.

Tool I use for this : XMind / Pen and Paper / The Brain

Mind maps for documenting presentations at conferences

Those that followed EuroSTAR 2011 might have seen a number of mind maps from me documenting the presentations I attended. Presentations aren’t typically as fast as meetings hence I get more time to type (and annoy other people with my keyboard tapping).

I find mind maps to be a good way of documenting the ideas and thoughts from the presenter (especially as some presenters may order their presentations differently to how my mind accommodates the information). XMind also has a cool feature which means I can publish the mind map straight to the web from the XMind client.

Tool I use for this : XMind / Pen and Paper

Mind maps for learning and studying

When I read a book I make notes. Loads of notes. If it’s a physical book I make notes on scrap paper or (much to many people’s horror) inside the book itself, then scan/input these notes to Evernote.

Once in Evernote I add the notes to a topical mind map.

I have a number of mind maps for different areas of learning from “communication” to “Social Media” to “Bento Tips and Tricks”. I can use hyper links and notes to add further context. Mind maps allow me to pull together learning from a number of sources around a central topic, all in one place.

For Kindle/Digital books I can use the highlight option and then pull the text from the text-file to Evernote. I then follow the same process above.

The same goes for blogs, white papers and eBooks. Everything ultimately ends up in Evernote with tags. It then makes it’s way at a later date to the mind map, at which point relationships are created and meta data added.

There is a cool feature in XMind (and probably in other tools also) that allows you to add a new sheet to a node which essentially gives another core node in the same mind map file. This technique makes it much easier to nest notes and ideas without making each mind-map sheet too big to read.

Tool I use for this : XMind

Mind maps for Checklists and To Do lists

I often stick a checklist in to a mind map format and print it out A3 size to remind me of things to do. I posted about the Phoenix Checklist mind map here.

I’ve had very little success with using mind maps for To-Do lists but I know others in the community are using them in that way.

Tool I use for this : XMind

Keeping Mind maps in Sync across devices

One question I get asked often is how to keep mind maps backed up and in-sync across devices.

My approach (and note, there are many others) is to use Dropbox to store the files. This way they sync across many devices. XMind is available on most platforms also so this isn’t a problem.

On iPad I use iThoughts which supports Xmind – although to be honest I typically use Evernote to store the information, or pen and paper then add to a mind map later.

There are a number of cool apps for Android and iPhone also. A quick search will find many. I’ve not personally used mind mapping on my mobile so cannot comment on their effectiveness.

There are a number of file sync services though to help keep your files available and up-to-date across devices. Services such as iCloud, SpiderOak, Wuala, UbuntuOne, Google Drive, SugarSync and Windows Live Mesh.

There are loads more but these are the only I have personally used. Try AlternativeTo for more.

Problems with mind maps

Mind maps are not a Silver bullet. I once spent a long time explaining an idea for a book to a someone using a mind map. They just didn’t understand the structure and the way I’d represented the format, characters, plot and sub plots.

I then exported the mind map to a .txt file and walked through the pure text and they got it right away. Not everyone leans towards visual thinking. Some people don’t find mind maps useful. Some people prefer other mechanisms and ways of organising information and data. Some people don’t find mind maps easy to read.

One of the major criticisms of mind maps (and I would agree) is that they are often poor at communicating messages (to some people). Sometimes the mind map has been crafted in a logical fashion by the author and to the author it makes sense. If that meaning (and context) is missing from the readers mind then that mind map might not make sense at all.

Mind mapping tools Example Mind maps – Software Test Related Extra Reading

It will never work here

I think one of the most fundamental mistakes many people make when talking about testing is to assume that everyone works in the same environment, thinks the same, is testing the same and is at the same stage in their experience and learning.

I remember one evening during EuroSTAR 2011 quite a few people gathered after the day to discuss the Future of Testing. The problem was, there was no point talking about the future of testing as I suspect all present had a different view of what current testing actually is. The current ideas to one person might be the future ideas to someone else.

I have an arch nemesis who occasionally turns up to events and conferences and generally argues with everything I may ever say. My Arch Nemesis works in the same industry/domain as me, yet Cloud and Agile will *never* be accepted in their company. Never.

Yet cloud and agile are at the heart of what I do everyday.

People work in different environments and each environment will have it’s limitations, differences, pros, cons and nuances.

It will never work here” is a phrase I hear all too often from Testers and mostly it is *possibly* true. And that’s cool. There’s a way of working for everyone and as great Testers we seek out these ways. We aim to maximize our strengths, whilst minimizing our weaknesses. This is what we do. Right?

Whether it is cloud, agile, continuous release, testing in live, acceptance test driven development, exploratory testing or any other activity/process you may do there will always be someone who loves the idea, but utters those words; “It will never work here”.

Instead of complaining about people who say this I’ve started challenging them as to why they think “It will never work here”. I ask them:

  • How did they come to that conclusion?
  • Who did they ask?
  • Was that the right person to ask?
  • Did they explore alternatives?
  • What makes them think that?
  • What do they honestly think?

If it is true, that it would absolutely never work for them, then that’s cool. It probably wont work. Time to explore some other avenue.

If you think “it will never work here” when you hear someone talking about an approach but aren’t sure why it wouldn’t work, then I would encourage you to dig deeper and ask further questions. Are you really sure it won’t work?

And if you think it would work but come up with resistance to change then don’t fear. This happens to almost every person who ever tried a new idea. There are always barriers; technical, people, money, time, red tape, process etc.

What can you do to make realistic changes to your testing? Note: The following ideas are based on the assumptions that changes you make are contributing to the department/companies objectives and goals – you do know what they are…right?

Instigate small changes, one at a time, and observe and measure feedback.

Find people who think the same way as you and work with them to bring about change.

Prove it – Make a change and measure the return on investment (ROI). If you can prove a good ROI for a change then you have a much stronger case. A gut feeling is good, but hard cold facts are better.

For example, let’s say you run a series of tests daily, manually. You prove that with some investment in an automation framework you can eliminate parts of the manual testing, freeing up a tester to do something more interesting and valuable instead.

Document the time saving, the costs, the expected time needed etc and present this to your manager/team. You stand more chance of getting agreement for change than simply saying “I think this would work”.

“But I have no time to make these small changes” – How about outside of work? During your lunch? Early in the morning? Are you sure you’re actually working on important things during the day? Or just “busy”? (busy does not mean you are adding value).

You can focus on your strengths and minimize your weaknesses. Let’s say you have great technical expertise in writing Selenium tests but no experience of the whole grid and infrastructure system behind it. Why not outsource this to a dedicated selenium grid cloud system and focus on what you do best? Or look to someone else in the business to get you going quicker?

Accept that nothing big will change overnight and look for small ways to improve your current situation. By accepting that things are “the way they are” you will approach your work with a different view. Tiny improvements and changes may become more obvious. Routes to new ways of working may present themselves more openly. You will notice ways to make your job easier/more fun/more challenging.

Move on. If you work in an environment where no change is possible (and you need it to change) then move on (easier said than done). When you have reached a plateau, don’t just sit there. The tech world is moving fast, and it’s important to keep your skills sharp and relevant. Move on and seek out something more in tune with your thinking.

Don’t take each approach and model as a one-size fits all solution. If you took each new idea, tried it and and it didn’t work, then try changing the approach. If it’s not right for your way of working then do something about it. All of the training and information on testing can be useful, but your intuition and experimentation are more important. And if you mash another idea to create a new one and it works…please share it with the wider community. 🙂

Are you hiring a certificate or a tester?

I read a blog post recently extolling the benefits that come from having a certificate.

The post alluded to the following:

  • With a certification you are demonstrating your skills as a Tester.
  • With a certification, as a team, you are demonstrating your proficiency to the business, and potential customers.
  • With a certification you open new markets for both yourself and the business you work for.
  • With a certification you can work on international projects or work anywhere in the world because everyone (those who have been certified) has a common understanding of testing.

It got me thinking.

With a certification you are demonstrating your skills as a Tester. I don’t think you are demonstrating your skills as a Tester simply by sitting any of the certification courses alone. I think you are showing you are “willing” and capable of learning more about testing.

I think for some people it shows a deep passion to learn all there is about testing (combined with many other sources).

Yet for many I think you are following the “norm” (which is not always a good or bad thing).

I think you are complying with stereotypes and expectations (but without knowledge of alternatives or awareness of other channels of testing knowledge, this too, is to be expected).

As a hiring manager I don’t see the certification as a “demonstration” of skills. Not at all. Especially when almost everyone has the foundation one.

I do see them as an element or components in a larger strategy of learning but on their own, they are not a demonstration of your skills.

Put another way, I don’t think not having a certification is a negative point, at least for the outlook on Testing I have. (for some companies it is mandatory though)

With a certification, as a team, you are demonstrating your proficiency to the business, and potential customers.

I don’t think you are demonstrating your proficiency as a team to the business and potential customer, however, I realise many people outside of testing may not know or appreciate this. (which is why this idea is flourishing) (isn’t it negligence for us to let this idea of certification=absolute excellence go unchallenged in the business community?)

I worked with a tester once who was fully certified at every level but he had no passion, no engagement in what he was doing and alienated all who worked with him. He dragged the team down and hated every aspect of his job. Was he a great tester? Was he a good advert for excellence?

A Test Manager a few weeks back was speaking at an event. He said he was the manager of 65 testers and was proud at how big his team of certified testers were. As you can imagine I, and a number of others, had a few choice questions for him:

Q : How much automation do you have? A : Umm. None. But I manage 65 testers.

Q : How many releases do you do a year? A : We don’t. We do one every 15 months. But I manage 65 testers.

Q : How do you coach 65 testers? A : I don’t. But I manage 65 testers.

Q : How do you do 1:1 and personal development with that many direct reports? A : I don’t. But I manage 65 testers.

Q : Have you analyzed areas of waste in your testing process? A : No. But I manage 65 testers.

Q : Have you…….(interrupted) A : I manage 65 testers. End of.

I may have exaggerated the answers but suffice to say there was no automation, no regular releases, no performance or load testing being done, no personal development, no lean thinking and waste reduction and a general lack of process improvement.

But….all 65 testers were certified and this was a great selling point. I know of teams with one or two great testers shipping software each week and working for companies that are dominating the markets they operate in. And they are a mixed bag of certified and un-certified (or should that be non-certified).

With a certification you open new markets for both yourself and the business you work for.

I think you can open new markets for yourself with or without a certification. It *might* make it easier with a certificate to be part of a company that places more emphasis on certification than personality, thinking and actual skills, but I think good testers open up new markets all of the time, with or without certifications.

With a certification you can work on international projects or work anywhere in the world because everyone will have a common understanding of testing.

I don’t believe a certification will give people the international standards and common understanding that the certification marketers would have you believe. Just get yourself to any testing conference and listen to the conversations to see that most people don’t agree on much in the Testing world.

People will approach testing (and the terminology and language used to communicate these approaches) in a way that is local to themselves and the context they find themselves in. To get a deeper understanding of how people think differently about testing ask questions to a group of testers from different industries, backgrounds and contexts at the next event you are at.

Questions like:

  • What is testing?
  • What is Quality?
  • Who should make the decisions to ship software?
  • What is integration testing?
  • What tests should you automate?
  • What is exploratory testing?
  • What is agile?
  • What artifacts should a tester produce during a project?
  • Who is in charge of your testing?
  • Should you always produce a Test Plan?
  • When are you ready to test?

I guarantee you will hear different answers from different people.

There is a world of difference between demonstration and assertion, yet certifications have somehow become an assertion of excellence to many. (and scarily a measure of a companies excellence in the wider business world)

This isn’t to say that the actual course, tutoring and learning is bad. Far from it. It could be excellent, valuable and perfect for you, but like most learning courses, some people learn deep insights and others learn just enough to pass the exam. (in fact, some may even be taught to pass the test, rather than deep insights but maybe I’m being too cynical)

As a Test Manager I like to look at it another, more simplistic way:

The certification is


the tester, and the tester is


the certification.

Are you hiring a certificate?

Or a Tester?