Exploratory Regression Testing

I was chatting through some ideas for “firing” up our regression testing process with one of my guys, Andy.

We already have a good set of the regression tests automated, but there are always some tests that need (or would benefit from) human interaction, especially when phone calls are involved.

We have a regression “pack” of standard tests but as Andy is running these each week (weekly release frequency) you can understand how it might get repetitive…and a tad boring.

So we talked about a few strategies. He and another one of the team, Simon, had decided to switch products every so often to break up the repetitiveness also.

This is a great plan. It means each of them get to see another part of the system and learn about it, it breaks up the repetitiveness and it also has an added benefit of a “fresh set of eyes”.

I won’t bore you with further details on this but one thing blurted our of my mouth whilst we were talking and at the time it made sense.

Only upon reflection later did I start to dig deeper and wonder what I was talking about.

I suggested to Andy that maybe we do some “exploratory regression testing“.

Sounds good right?

But can Exploratory Testing really be used to do regression testing?

  • Isn’t regression testing “supposed” to be about running already executed tests (that subsequently become checks)?
  • Should regression testing be about repeating already executed tests/checks? Is there value in this activity?
  • Can exploratory testing really be performed to give confidence that nothing has regressed?

The more I explored some of these thoughts the more I realised that I actually quite like the term and concept of exploratory regression testing.

No doubt there are already people using the technique and phrase, and no doubt there will be many people who don’t believe exploratory testing is regression testing, but for now I’m going to keep using it.

Here we all seemed to understand the term and idea and it’s giving us great value.

I just thought I’d get your feedback on it.

Thoughts?

10 Replies to “Exploratory Regression Testing”

  1. Maybe it depends on your definition of ‘regression’ and that might lead to what is required to test for it ?

    I gues I’ve always been doing this anyway, I never just run the same tests. And ‘it depends’ – is the new release a simple one line fix, do you have confidence in the release process, have there been problems in the past etc etc

    1. Hi Phil,

      Thanks for commenting.
      A lot does depend on context. The scary thing I see too often is people selecting previously run tests (or should they be called checks) and the whittling them down and ONLY running this subset. I’ve never really understood this approach. I’ve always done ET testing as well as a core pack of tests. I guess others have too judging by the comments and tweets.

      I think though that we are the exception. I’m not sure many testers even get to choose.

      Rob..

  2. I do believe you are onto something here and it may already be getting done.

    Similar to how the way Mr Whittaker used tours of the application, each area you are performing the regression on is split into functional areas. Use the regression’s manual procedures as your map, then via of course. Match with the automation it can be a very powerful tool in finding those pesky defects in areas of the system the automation at the time of creation or maintenance phase does not got.

    I am liking your term and your idea, some will no doubt say “it’s not Exploratory testing” yadda yadda, but the purists never pushed things forward.

    your Q: “isn’t regression testing “supposed” to be about running already executed tests (that subsequently become checks)?

    my A: No, it’s about running already ran tests to see if that area still functions as expected and nothing around it has affected that area, they are checks but not in the tick box exercise one may perceive them to be. Regression is NOT an excuse.

    your Q: “Should regression testing be about repeating already executed tests/checks? Is there value in this activity?”

    my A: Yes there is always value in it, however the risk is that we won’t find any new defects (or the count will be low) but hey, if it’s automated, just keep running as the cost is minimal. Plus if you find 1 defect it’s paid for it’self.

    your Q: “Can exploratory testing really be performed to give confidence that nothing has regressed?”

    my A: this is more tricky, I am tempted to give the tester’s default answer of “It depends”, but I won’t. In short, Yes it can, if done properly, measured and concise. I would be aware of the phrase exploratory to mean business user button bashing etc…or “off piste testing”, you just won’t find any “juicy” defects that way and will typically be more cosmetic I fear.

    If it’s giving you great value then stick with it!

    1. Hi Graeme,

      Thanks for the great comments. I’d not really considered the tours angle but no doubt see how they fits with what I was blogging about. I think the tweets and comments around this post suggest that others also vary their regression and ET combinations greatly.

      I like your point about the purists. It does seem that too many people are unwilling to bend and flex to allow innovation (or failure) to occur.

      Always nice to hear from others in the community.
      rob..

  3. I think there is plenty of room for exploratory regression testing. To me regression testing is all about ensuring the perceived code quality has not regressed, it’s less about how one actually goes about it. So using the same test cases is not necessary since there’s no certainty that the code under test is still adequately tested by those cases anyway if the code has changed.

    I’ve used this technique in my teams a lot in the past, first defining measures against which we can identify regression, then using whatever testing approach works, to test against those measures. The measures can be high level (for example, number of application resets) or lower level.

    Context driven regression testing essentially.

    1. Context driven regression testing essentially.<-- you nailed it. Would you mind sharing some other measures with me? Intrigued as to how else you (and others) measure the value of regression testing. Thanks for commenting Rob

  4. I had an open discussion about regression testing on twitter a few weeks back.

    For me there is no pre-determination about re-using a test idea or script.

    Indeed, to test for regressions you might need to take account of the code changes, the product consistency, the testing consistency & adequacy so far, discussions with developers, team and PMs, and assess risks. This might give an idea where/what to test – not how. Then the testing could be made as any other testing approach. Whether a pre-defined test script is re-used or not is another question – there may be a “fit for purpose” question about the script – does it check for “good enough” points, or does it need modification or complementing with other tests….

    Automation and scripting has been used to aid exploration before – I think Cem Kaner wrote about this, and I touched on it with the “walking in the woods” idea.

    Oh, I’m careful how I talk about regression to not include automated code change checkers – as in continuous integration suites.

    And to give (my) answers to your questions: No, No, It depends, It depends 🙂

    1. Hi Simon,

      Thanks for commenting.
      It does depend indeed and a great set of ideas on regression testing and techniques.
      The fit for purpose idea is a strong one yet I see so many Testers applying very little logic when selecting previous tests/checks. I think the same is true for any testing some people do.

      Selecting the right test for the right moment is a core skill many people ignore.

      As usual, your comments give me things to think about 🙂

      Thanks
      Rob

    1. Thanks for commenting.

      Good post and thanks for sharing your thoughts on regression testing. I think you’re right about a combination of different types of regression. It seems a nice balance is best for a number of others too.

      Cheers
      Rob

Comments are closed.