Evernote As a Test Case Management Tool?

Could I realistically use Evernote as a Test Management Tool?

That’s the question I asked myself the other month after finishing writing an eBook on Evernote for one of my other blogs. I’d become convinced that Evernote could be used for almost any requirement (within reason). Could it be used for test cases?

I did a quick proof of concept the other week looking at whether or not we could use Evernote as a Test Case Management tool with some interesting outcomes, and a great deal of learning along the way. I’ll share this with you here.

I floated the idea with my team after the initial spike and a few of us did a quick brainstorming session to talk through the process flow and major problems with the model. We drew it out and talked about some of the pros and cons. We concluded it was possible, but not without some potentially major work arounds (metrics, reporting, master copies etc). I’ll talk more about that in this post.

Why Evernote? Evernote, if you’ve ever used it, is a truly awesome way of capturing information.

I use it now for all of my Exploratory Testing notes.

It’s super quick to add stuff, tag it, re-categorise/shuffle it around and edit it.

There are several ways of getting information in to Evernote and once it’s in there it’s super easy to search for stuff.

It’s a perfect companion for me when exploring.

We also have a number of mobile devices which we use for testing. As Evernote is available on almost any device, it seemed like a perfect companion.

A great feature of Evernote is that you can share a notebook. This meant we could create a series of NewVoiceMedia testing notebooks and share them across the entire team. Another great feature. Any device, any tester, anywhere.

Requirements of a Test Case Management Tool

Despite being in an agile environment and heavily automating tests we will always have a number of test cases for legacy bits of the products, tests which are not valuable to automate and for compliance/reporting reasons.

We also have some very lightweight (I actually consider them checklists) test cases that we run to check over each kit, as well as regression tests.

All of the team make copious amounts of exploratory testing notes in their favourite system (rapid reporter, notepad++, Evernote) which ultimately end up as txt files in SVN too.

How Evernote might work I created notebooks which I populated with notes. As a basic guide a “note” is a test case and a notebook could be considered a container or “suite”. I tagged each note with relevant functional area and type of test (i.e. regression, call control etc) I created the following notebooks.

• MasterCopy (contained the master copy of each test case)
• ToBeRun (containing a copy of each test case needed to be run)
• Complete (containing all test cases completed – moved from the ToBeRun after it was executed.)
• Exploratory (containing all exploratory testing session notes)
• Areas to explore (a notebook containing any area we deemed was worthy of further exploration)

Creating Test Cases

I like to encourage exploration with the tester’s knowledge and insight leading the testing, so where possible we all tend to write our tests in a very checklist orientated fashion. This means we don’t have “step by step” instructions and we assume that the person running the test case has some prior knowledge of the system and a good grip of testing.

Therefore our tests are very much like checklists and we leave it to the tester (with their own skill, experience, knowledge and insight) to test against the checklist where relevant. I don’t mandate that each checklist item is tested against either. It could be that we only run 30% of the test case. That might be ok. The Tester is in charge based on their insights and local knowledge.

Evernote supports checklists in a very simple form. It needn’t be more complicated than a checkbox against each item so Evernote worked well for creating and storing test cases.

There are exceptions though. Some of the areas we test against are incredibly complex and require absolute precision to check for known outcomes. Therefore we do have some test cases that detail each step someone must take and the expected outcome. These are mainly focused around statistics, percentage reports and other numbers based outcomes that require precise timing and action. Automation helps greatly here, but sometimes a legacy code stack can hinder that automation.

These types of detailed tests can still be accommodated in Evernote easily. I used something called Kustom Note to create a template for these types of test ensuring I captured all relevant information.

The only thing Evernote clearly does not do is report on metrics of completion or coverage. I knew this going in to the experiment. That was ok when I started out.

Automatic Creation of Tests In Evernote

So far, so good. Evernote clearly does allow for basic test management where metrics are not the sole output. This suits us nicely. Our metrics are taken at an automation level which is dealt with by other systems and tools.

One of the really great uses I found when exploring Evernote was how I could trigger a new “test note” to be created when a new feature/story/workitem was created in a different system.

I did this by grabbing the RSS feed from our instance of Pivotal Tracker.

I then turned to If This Then That (a great automator of the web)(IFTTT).

I configured IFTTT to grab the RSS feed from Pivotal Tracker and automatically create a new note in the Evernote notebook.

Kapow. A new Exploratory Testing placeholder for a new story. Sweet.

I got giddy with where this could go now.

I then hooked Evernote up to the RSS output of a social media aggregation tool I use. I could therefore collect social media mentions of a product and create exploratory (or investigatory) sessions from it. Interesting.

For example, if there is a mention of something like a bug, or slow down, or any other query about how something might work we can automatically create a note in the relevant notebook for a tester to explore, or even for someone to respond to it.

But it gets more interesting still.

Let’s look at some other potential sources.

• Case/Support management tools could generate a session when a new case is raised.
• Bug tracking tools could trigger a session and include bug details when a new bug is fixed.
• Texts, emails, facebook updates, tweets…..all used to create a new session/test.
• Evernote changes could trigger further new sessions also.
• Delicious feeds could create new sessions based on new ideas/techniques or approaches that someone else has written about. Bookmark the page (and therefore idea) and create a new session based around that idea.
• Dropbox (or SkyDrive etc) updates could trigger a new session. New screenshots, files or shared resources that need exploring for example.
• If you use Chatter (or Yammer) you could automate a new session based on a post to chatter from someone in your company.

There are many uses I can think of where we would want to create a new test based on the content or changes in another system. IFTTT can help you with this. RSS feeds from other systems greatly increases the ease of which you can do this, especially if that other system is not supported by IFTTT.

Or you could step further in to the realms of beta tools and check out Zapier. It looks truly awesome.

It supports Campaign Monitor, Basecamp/Campfire, Agile Zen, Aim, Github, Jabber, HipChat, Google Drive, Google Tasks, PayPal, MailChimp, Pivotal Tracker (direct), Salesforce, UserVoice, ZenDesk….the list goes on.

The options are plenty, not just in creating sessions from other work streams and items but how about the other way around? How about on completion of sessions/tests you update another system?

Or how about stepping outside of the test case world?

How about using Evernote and IFTTT to create an amazing notebook of testing ideas and content? Each time someone in your team shares something, bookmarks something or creates some other content it could be collated in one notebook for the rest of the team.

With a little time and some deep creative thinking I suspect there is very little a test management tool can do that you couldn’t with some hacking and mashing with apps like Evernote….with the possible exception of metrics.

But why would you?

Why not? Why not use the tools and systems that are naturally good at their specific niche and make your testing process more fluid and contextual?

If you have a test case management tool that suits your needs then that’s cool. Stick with it. If you don’t, and can’t find one, then why not get creative? Most of the tools that you can hook together are free and could solve your problem.

You’ll also probably learn a lot about your own needs and requirements through the process of exploring, have a good laugh hacking systems together and probably learn about a load of cool tools that could help you in ways you’d never imagined before.

It’s good to explore the new tools on the market, even if they aren’t strictly test management tools. The tech world is moving fast and sometimes it’s worth exploring new ways of doing things. Sometimes it’s best to stay with what you’ve got also, but only you will know what the best course of action is.

After all of this exploring and trialling we are sticking with what we have right now though.

After investigating these tools I gained deep insight in to our needs and requirements and realised that we do need some metrics around our testing for compliance and for reporting coverage.

I’m not done with the exploration of Evernote as a test management tool though. I think it has massive potential. What do you think?