How to run a proof of concept

When seeking out new tools to aid your testing I strongly believe in doing a proof of concept. It needn’t be anything heavy weight.

In fact, the more concise the better.

What is a proof of concept?

I use the term “proof of concept” to describe the process of evaluating a tool (or service) for suitability against a number of criteria.

This criteria should be specific to your context, although some of the criteria would probably apply across many contexts.

Note: from this point on I will use the word tool to encompass both tools, services and other paid for systems.  Why perform a proof of concept?

I believe it’s important to run a proof of concept for a number of reasons.

Short term reviews of products and more specifically, product specs/marketing material, can lead to a snap decision that might not consider a wider view. This is even more a problem when you have a salesperson pushing at you to make a decision.

You don’t have to look far to find many companies and organizations trying to sell you a solution to each testing problem you may (or may not) have. Some of these solutions are great and right for you. Others are not.

If you don’t spend time trying the tool to see if it solves your specific problem then how can you be sure the product is right? How can you even be sure that the claims in the literature or sales meeting are even true?

Tool usage can often involve people from other departments, either directly or indirectly, so it’s only right to collaborate with them also.  (for example, a bug tracking system is not typically used just by testers).

The initial pricing descriptions or quotes may not actually be the end price you pay after adding on extensions and support contracts, etc. This is a very compelling reason to find out what the tool does and what it doesn’t do before eagerly signing on the dotted line.

The tool needs to solve a problem, but it might cause more in the long run. Snap decisions to solve one problem could leave you with a bigger problem (i.e. -training needs, hardware requirements).

Comparing a number of tools can often lead you to greater insights in to the problem you are trying to solve and you may generate creative solutions to it which might not have come about without proving the tool.

My memory isn’t what it used to be so documenting the proof of concept and having comparison data to refer back to leads me to a more accurate decision.

There are many other reasons why someone might perform a proof of concept, the above are just a few that immediately spring to mind.

When should you run a proof of concept?

In my opinion any tool adoption that impacts a wider audience than yourself should have a proof of concept process.

For example, my decision to start using Rapid Reporter rather than NotePad only really changed my working arrangements. So I didn’t feel the need to perform a proof of concept.

However, a decision to start using a new Test Management tool might impact the wider Test team so this could be a good candidate for a proof of concept.

How long should a proof of concept take?

There is no right or wrong answer. It depends on the tool, the number of people involved, the cost of the tool and the infrastructure to name a few variables.

I try to keep my proof of concepts as short and time-boxed as possible. This focuses the mind and encourages better planning in advance.

I would suggest that if your proof of concept is taking months then you have a problem. Either you are not pro-actively involving the right people and making use of the time available, or you’re trying to assess every angle, feature, component and pricing model of the tool. This *might* be a good thing when opting for the big vendor wallet splitting tools, but for most tools I’d suggest it may be overkill.

How to prepare a proof of concept Scenarios and Problems

My first step is to write down the problem(s) you are wanting to solve. Writing them on paper helps to articulate clearly the problem space. It helps you to work through the problem in your mind.

For this I typically use a mind map, but that’s my own personal preference. Use what works for you.

Once you have articulated your problem clearly you will probably find a natural set of criteria emerges that you can use to judge the tool against.

For example – you might want a tool to aid your test management. You might have a desire to report the number of bugs raised versus the number closed per sprint, or other time scale.

From this problem statement you automatically have some checks to run. (i.e – does the tool generate that report?)

If the tool does not solve your problem and meet your requirement then either move on or re-assess how important that criteria (in this example, the report) is to you.

In defining the problem and some typical uses of the tool you will start to build a handy checklist. Don’t go mad though, you could spend months assessing a tool against lists.

Budget

It would make sense to find out what your budget is. Your budget constraints will naturally lead you down certain routes.

There is no point reviewing a big vendor tool unless you have a stack of cash to throw at it. As a manager though, be cautious about blowing all of your budget on a big vendor test management tool.

Firstly, there are many tools for less (or free) that do the job just as well (if not better).

Secondly, there will always be something else to spend your money on further down the line.

Is support included?

It is always worth checking if support is included in the price. And if so, what level of support that provides.

Are there any SLA (service level agreements) against this support contract?

Do you need support?

Are upgrades and maintenance releases included in the initial cost, or charged per upgrade?

I once worked with a Test Manager who bought a giant vendor tool for many thousands of pounds but failed to consider the upgrade licenses. The tool vendor released the newer version about 4 weeks after we started using it, and the newer version obviously had more glossy features and many bug fixes…which we then had to pay extra for.

Always check the pricing structure clearly.

Are you going to need any plug ins or extensions to make the tool work?

If so, how much are they?

How easy are they to use?

Are they widely supported?

Can you try these as part of the free trial?

Do I, or the team, need extra training?

If so how much and is there a cost associated with this?

Would a book suffice?

Are there any user groups or online communities that can help?

What skill sets do we currently have?

Are there open source tools available?

You might find that there is an Open Source tool available that does 99% of what you want with no license costs.

Could you roll your own tool from the Open Source code?

There may be three or four tools that solve your problem available with open source licenses. Could you plug them together with some bespoke code to create a good solution?

Do you have the required infrastructure and hardware?

If not, how much will it cost to get it?

If yes, what is involved in getting the product running? Do you need help from others in the business?

Are other departments or teams looking to use this tool too, or are they impacted by it?

If not – are you sure?

If yes – have you spoken to them? Are they involved in the review?

How long are you planning on keeping the tool?

Could you find a creative work around if your problem is short term?

Are you 100% sure that you want to sign a contract for many many years? What about a SaaS model? Or one year contracts?

Does the tool support multiple ways of working (methodologies/approaches) or does the tool enforce a certain way of working?

It is always best to find a solution to allow for future growth and change.

Many of the old-school testing tools enforce a certain order or structure which could hinder your growth or stunt your productivity.

Be sure to assess the tool from your point of view, not the point of view communicated by the literature, sales person or wider community. The tool needs to work for you.

What is the deadline for deciding?

Do you have a deadline that needs to be met. This will give you a constraint, which will help to focus your attention.

Can I try the software?

If not – why not. Come on, it’s 2012, it should be possible to try before you buy, even products that need to be installed client side.

I would suggest you download or sign up to as many tools as possible and run each one against any acceptance/proof of concept criteria you have.

Final Thoughts

The above might help trigger some thoughts for you, or you may already have your own list, but I strongly believe that time invested in the proof of concept could be time saved (and/or cash saved) in the long run.

On a general note I would look for a wide selection of tools and try as many as you can. I wouldn’t just follow what the industry mainstream is doing. Focus on what you need and you’ll certainly find the right tool.