Standard. Or Not.

The international standard for Software Testing, ISO 29119, is soon to be upon us. There are people writing it and expanding it and creating it right now.

I’ve signed the petition against it because I don’t agree with some of it and it’s supposed to represent the industry I work in and I had no chance to input to it.

I wasn’t involved in shaping any of it, yet it’s soon to become a strong international standard in software testing.

My concern is not about how it will affect myself as such, but about how it will stunt the growth of the companies who sign up to it. It’s my belief that the standard will stunt the growth of the employees also.

I believe standards are useful. I believe standards can be helpful. They can turn laws in to process, or set out norms of operating that are good for business and society. This is useful.

Standards can also be restrictive though.

A company operating under this ISO 29119 standard may lose their ability to experiment and find new ways of doing good testing. Our industry and craft could suffer. The standard may hinder our ability to evolve our testing to be relevant to the market demands of the companies we work for.

I’m not overly worried just yet though. The standard is not even complete and it will take years before it starts impacting the market. The standard will likely be voluntary to adopt also. Companies that are experimenting and pushing boundaries will most likely not adopt it. At least not without a compelling reason. That means testing can still evolve.

That means there will still be a place for people like me. People who want to apply relevant test techniques and approaches. People like me who want to create new ways of testing that don’t yet exist.

And how can a standard cover something that doesn’t yet exist?

After writing this post I realised I’m describing a similar testing industry to right now. An industry where lots of companies rely on certifications. An industry where the minority are doing the innovative testing.

So will the ISO 29119 make a difference? Will it even get off the ground? Should we be trying to fight it if we don’t agree with it?

Or should we just accept it will happen and continue to do the best work we can; continue to be relevant to the companies we work for?

I suspect time will tell. As it always does.

If you don’t think the ISO 29119 is a good idea then consider signing the petition.

14 thoughts on “Standard. Or Not.”

  1. Hi Rob, thanks for joining the debate.

    I think you’re probably right that smaller, smarter companies such as yours could be unaffected. However, there are several dangers. The main one is that governments or larger companies will not only comply themselves, but insist that their suppliers do too. That could squeeze companies that do good testing out of many markets.

    Also, I am very concerned at the way the standard is being marketed as proof that testing has been responsible. I don’t want to see companies that shun the standard getting sued because of false assumptions that non-compliance is evidence of negligence if there are problems. That shouldn’t happen, but there is a clear implied threat in the way that ISO 29119 is being pushed.

    I think it’s very unlikely that we could see ISO 29119 being withdrawn, but I think it’s important and possible to get the message out to the wider public that ISO 29119 doesn’t represent a consensus on what good testing consists of. We must spread the message that it is a dated and flawed approach to testing, so that no-one feels compelled to use it.

    1. Hi James,

      Thanks for commenting.

      “The main one is that governments or larger companies will not only comply themselves, but insist that their suppliers do too. That could squeeze companies that do good testing out of many markets.” – Yep – I had that outlined in a follow up blog post as the network and ripple effect of the big boys will make it harder to operate. :)

      “Also, I am very concerned at the way the standard is being marketed as proof that testing has been responsible. I don’t want to see companies that shun the standard getting sued because of false assumptions that non-compliance is evidence of negligence if there are problems. That shouldn’t happen, but there is a clear implied threat in the way that ISO 29119 is being pushed.” – Good point indeed. It could become a replacement for good testing.

      But how do we spread that message. I see it gathering momentum on Twitter but the test community on social media reflects just a very small portion of the community. We can all get excited about how much difference we’re making but the reality is the bigger potential adopters have thousands of testers who may not have the opportunity, inclination or other reason to challenge this. Thoughts?

      1. You’re right about the limited impact we can have on Twitter. For every concerned tester that signs the petition there will be dozens working for big companies who will never see it, or would not feel that they could safely sign it.

        That’s why it’s important that AST and ISST take a stand. It’s not a numbers game, and if credibly professional bodies take a stance then that becomes harder to ignore.

        I’m also trying to get the message across to the auditing community. They’re not looking for a testing standard to help them do their job. ISO 29119 goes against the instincts of good auditors (but not those of the bad ones). There’s always the danger that auditors will fix onto a standard because it does give them something to audit against.

        I think this quote is very important, from Steven Ross, former President of ISACA (Information Systems Audit & Control Association) in ISACA Journal, vol 2, 2013.

        “I have always had difficulties with the term “best practice.” Who is to say which practice is best, which is almost as good, which is really not good enough? The members of the standards committee have been appointed (who appoints them, anyway?) to define best practice at a point in time, but as I stated previously the best sinks to “just okay” practice with the passage of time. A standard is obsolete the day it is published.

        Is a standards committee empowered to describe what they believe are generally the best methods, tools and techniques as evidenced by what most organizations are doing? If so, the committee is describing standard practice that has been overtaken by best practice, however defined. The process of standardization drives the thought process to the middle. Carried forward over time, standard practice is mediocrity.”

  2. One response to this effort to define a testing standard is to try to kill it or block it. That seems to be the predominant response now by the testing community (or at least the twitter testing community). Are folks considering other responses?

    Two thoughts on this:

    1. What is the source of the energy driving this standard effort and what are alternative ways to channel that energy? To put it another way, what are the folks behind this effort not getting that they think they would get if they had this standard and in what other ways could it be satisfied?

    2. Since you mention that standards have the potential be useful and helpful as well as restrictive (which I agree with), do you have any thoughts on what would a useful/helpful standard for software testing look like?

    1. Hi Ronald,

      Thanks for commenting.

      It’s not clear what the true motives of the standard are other than to attempt to drive in accountability and “professionalism” to our the industry by giving companies (and auditors) a clear set of guidelines and expectations to judge testing against.

      I’m a fan of standards that protect customers from illegal activity and enforce certains aspects of good practices like financial regulation, data protection etc. However, testing is an activity that is done by a great many people. There are a great many ways of building and shipping software and to attempt to standardise this is scary. By the nature of the standard it will exclude a great deal of teams and people – this is not only *not* an agreed standard but it will likely, in my opinion, limit innovation.

      Rob

      1. Thanks Rob. To continue the discussion:

        -I don’t believe a standard has a motive. It is a response of people that have motives. If, as you conjecture, those people don’t believe that the testing that they are getting is done by people who are professional and accountable, what other methods/artifacts/etc. apart from the proposed standard could give those stakeholders what they want? Can the testing community give them that?

        -When I read your response to my second thought, I “hear” you replying as if the proposed standard is the only possible one. Maybe that’s true but is it possible that there could be a (different) standard that would not exclude a great deal of teams and people and would be an agreed standard?

        Not easy questions to answer, I admit. And maybe it is important to first stop a flawed standard that would limit the industry.

        Thanks again for the post and the response.

        1. Hi Ronald,

          “-I don’t believe a standard has a motive. It is a response of people that have motives. If, as you conjecture, those people don’t believe that the testing that they are getting is done by people who are professional and accountable, what other methods/artifacts/etc. apart from the proposed standard could give those stakeholders what they want? Can the testing community give them that?”
          Absolutely – I was using the non-animate standard as a collective noun for those behind it. I don’t know who is behind the standard and may well be surprised if I ever find out. There are lots of different ways to give stakeholders what they want and that’s my primary objection. Most of these other ways don’t appear to be included and instead a standard way of planning and reporting is being proposed by the standard.

          “-When I read your response to my second thought, I “hear” you replying as if the proposed standard is the only possible one. Maybe that’s true but is it possible that there could be a (different) standard that would not exclude a great deal of teams and people and would be an agreed standard?”
          — Yeah – no doubt there are more than one proposed standards but really this is the one that appears to be the biggest and baddest and the one with the most support (and objection). The problem with standards is that they are out of date too quickly. It takes so long to create them and document them and communicate them that they’re often too late. So whether there are other standards or not I believe is irrelevant – the biggest question to ask is why are people feeling the need to standardise what is actually quite a creative and thoughtful activity – testing.

          Thanks again for commenting. I’m enjoying the discussion.

          Rob

  3. How does the proposed standard, or having a standard at all, stop “People who want to apply relevant test techniques and approaches”? Seems to me that a long list of relevant techniques are (according to the publicly available outline) in the standard.

    Why does a standard stop “People … who want to create new ways of testing that don’t yet exist”? Seems to me that it’s a handy list of what *does* exist, so that people can judge if claimed “new ways of testing” really are new.

    “And how can a standard cover something that doesn’t yet exist?”

    Of course he current version never can, but why should it? The *next* version should. Good standards undergo periodic updates to keep up with industry progress, don’t they?

    I have not understood the objection here.

    (I do admit that I was unimpressed, though, to have a typo jump out at me from the home page of a *testing* site — somebody should be testing the site …)

    1. Hi Tom,

      Thanks for taking the time to comment.
      “How does the proposed standard, or having a standard at all, stop “People who want to apply relevant test techniques and approaches”? Seems to me that a long list of relevant techniques are (according to the publicly available outline) in the standard.”
      — If the technique that is relevant at that time is not part of the standard then it will be hard to apply it – even though it is the best one at that point in time. Non-conformance to standards is often seen as negative so people may be pushed in to complying with standards rather than trying something new, or adopting something more relevant.

      “Why does a standard stop “People … who want to create new ways of testing that don’t yet exist”? Seems to me that it’s a handy list of what *does* exist, so that people can judge if claimed “new ways of testing” really are new.”
      — If you cannot do testing that is not part of the standard, and this is how some companies will comply with the standard, then why will you be motivated to try anything new? And how could you try something new if you know it will cause a non-compliance?

      ““And how can a standard cover something that doesn’t yet exist?”

      Of course he current version never can, but why should it? The *next* version should. Good standards undergo periodic updates to keep up with industry progress, don’t they?”
      — Totally agree. But. Standards often lag behind industry. In order to define, quantify and articulate something to be standardised you need to have understood it deeply and hopefully, measured it’s effectiveness. This takes time, effort and energy. And will most likely only be done for techniques/process/etc that is being used by a significant number of people.

      Rob

  4. Rob, I’m still feeling that it’s not standards that would hinder innovation, but how companies use the standard. We’re not talking about a hardware standard that says, for example, the distance between the two prongs of an electric plug must be a certain number of millimeters with a certain tolerance. No room for interpretation there. But a *practices* standard. A professional (as opposed to an empty, legalistic) response by companies to that is “we will do *at least* what is in the standard. If we are going to not comply with some item, we will have a good, documented reason. And we will continue to *go beyond* the standard however we see fit.” That keeps the door open for innovation that *leads* industry practice and, if the method and company is successful, gets reflected in future versions of the standard.

    Since the standard promotes, “a risk-based approach to testing [that] should be used to determine the set of techniques that are applicable in specific situations,” I see no barrier, only encouragement, for companies to review, choose within, and go beyond the standard.

    James, again, I glanced at the outline for the testing techniques and, while it may be missing a few things, I can’t see why it represents a “dated and flawed approach to testing”?

    I agree with Ronald that there are positive, rather than just blocking, responses to a practice standard. I think the response of the professional testing community should be (a) to highlight anything current that is glaringly missing from the standard: whether or not it gets included right now, at least practitioners will hear about the missing issues, and (b) to remind industry how to *use* a practices standard to best effect.

    1. Hi Tom.

      I agree – it will be about how companies use the standard but as James points out they may just blindly adopt it and use the standard as a marker of good testing.

      I have read through all of the sections in the standard and I’ve yet to really find anything that sits nicely with how we work. And that’s fine but it looks like the standard is being worked on by people who treat testing as a separate and distinct activity and phase. Are they talking with anyone who works in a different way? Maybe they will and include something but right now I’m concerned at what I see. Some is good – and basically common sense. Some is old fashioned and not relevant, and some is frankly quite odd.

      At present, if we were to adopt this standard as it stands (and I know it’s far from complete) then I’d either have to add a huge overhead to our development process or raise a significant number of non-compliances.

      There are positives and they should be celebrated but the overall approach is worrying in my opinion and I predict many companies blindly adopting it and then saying they do good testing.

      I will indeed be blogging about some of the things missing and some of the things that wont work, but that seems too late. Why are more people from differing points of view not included in the creation of it, or at least consulted on it?

      Rob

Comments are closed.