Cannot Reproduce

Image from : viZZZual

Call me old fashioned, but in all of my years working nothing has beaten a good old face to face conversation with a developer about an issue or bug in the software. Using a bug tracking system as the primary means of communication is a “no no” for me.

Yet, last week I was chatting with a seasoned test manager who was extolling the virtues of bug reports. Now, I’m a fan of bug reports (in moderation) and I’m a fan of tracking issues (though not necessarily in a bug report) and I’m also a major fan of communicating clearly, but I also like to get things done with the least amount of friction. But this test manager was suggesting that bug reports are the single best form of communication between developers and testers and therefore every single bug should be logged in a defect tracking system mainly for no other purpose than communication. Communicating through bug reports is “fundamental” apparently. 

 

In my view, communicating through bug reports is a sign of some deeper issue within that organisation. If you are using a lifeless remote system to communicate between two humans as your primary medium of communication then you are destined to have lost meaning, wasted time and rework. You have a problem.

 

When we write things down we often (not always) lose information and more often than not, in the case of defects, waste lost of time bouncing bugs backwards and forwards when a simple conversation might have cleared the whole thing up. It often means the bug reports themselves are more thorough and complete when the developer and tester have chatted about the issue in advance. Now I’m not adovcating badgering a dev each time we find something, but I am suggesting we stop and think about how best to communicate the issue. Think about The Purpose, The Audience and The Context.

Would it be easier and quicker to show the dev the issue and talk it through before raising a ticket for it?

Or would it be easier to raise the ticket and then talk it through?

Or would the ticket clearly explain the issue without talking it through?

Or should I raise the ticket and then sit back and wait for the inevitable bounce back with “cannot reproduce” or “as designed”?

Or do we even need a ticket in the first place?

 

Here’s an over the top example, but scarily it’s not far from the reality I’ve seen in the past,.

Defect Report

 

Title:

When I enter some text in to the name field I get an error

 

Steps to reproduce:

Enter text in to the name field and hit save

 

Attachments:

None

 

Defect Change History:

  • changed to “cannot reproduce” by Dev 1 – It works fine for me

 

  • changed to “open” by Tester1 – I can reproduce this all the time

 

  • changed to “cannot reproduce” by Dev 1 – Works on my machine. Still cannot reproduce.

 

  • changed to “open” by Tester1 – Doesn’t work here still. Might be worth you spending longer trying to reproduce it 🙂 

 

  • re-assigned to Tester1 – What error message do you get?

 

  • re-assigned to Dev1 – I get a general server error

 

  • re-assigned to Tester1 – how about attaching a log file….?

 

  • re-assigned to Dev1 – I had to reproduce the issue to get the error, took ages, too busy to keep doing this. Error is “General Server Error”

 

  • re-assigned to Tester1 — any more clues than that? What about the log file from the server? 

 

  • re-assigned to Dev1  – Had to reproduce this again at great cost to my time schedule.. Attached the log file from the server.

 

  • re-assigned to Tester1 – You attached the database log file. I need the webserver log file

 

  • re-assigned to Dev1 – You didn’t specify which one. Attached the web server, database, client, domain controller, event log….enough?

 

  • re-assigned to Tester1 – Having had a look through the log file it looks like it’s some sort of issue to do with multiple record updating. Were you doing anything else whilst saving the record?

 

  • re-assigned to Dev1 – Yes. I was updating the customer details whilst accessing the user records in another browser tab.

 

  • re-assigned to Tester1 – Any reason you didn’t explain that in the first place?

 

  • re-assigned the Dev1 – Any reason why you coded this bug in the first place?

 

Don’t let your conversations with the people be through a static system. Even with remote teams there’s still Instant Messenger Systems, Video Calling or the good old telephone. Technology is making communication easier; there’s desktop sharing, live desktop and video meeting  systems, webinar systems and a whole host of other cool stuff designed to get us all talking and sharing. Don’t let your written words in a defect tracking system be the only route through the wall that sits between dev and test. Break the wall down and say hi to your friendly neighbourhood developer. 

 

34 Replies to “Cannot Reproduce”

  1. good example. I have seen this happen.. though it was onshore and offsite issue. They had to use the tool as a medium of communication. We solved this by having an hour of overlapping time where developers and testers could talk things through. Helped solve lot of issues and conversations like the one from your example.

  2. Raise the ticket AND talk to the developer :)The ticket is to account for my time – sometimes the boss likes to know! – but the detail usually requires face to face / IM etc. I know my manager would argue that it also allows him to allocate/plan resources more efficiently – whether it’s more efficient for me is another matter!

  3. Rob, very true & what it perceived to be common sense by many is actually a very real and major problem in many businesses.Its incredible how much we can talk about good communications & how it is always present there flashing in the back of our minds. Yet so often we fail even at the simplest things, such as this.I like you image of the brick wall, you’ve summed up poor communications brilliantly there! Good post! Thanks for sharing.

  4. Hi Shilpa,Thanks for commenting. I’ve seen the same thing where companies have formal or informal meetings to air things and talk things through. It’s not the same though as having access to people but I guess this isn’t possible for many offshore orgs,ThanksRob..

  5. Hi Phil,Seems this is working well for you but I would agree with you that I’m not sure it is efficient for you. Interesting how your boss needs these tickets to account for your time…I have a post about this type of measurement in draft :)Thanks for commentingRob..

  6. Hi Darren,Thanks for commenting. Communication is an interesting topic to me because very few educational systems teach communication as an everyday lesson. So people leave schools and universities with often poor communication skills.In my experience, almost every single problem in business is caused by (or affected by) poor communication. Communication is key to everything we do, yet we don’t teach people how to communicate. I like how you mention that people think about it, but I’m not so sure they do. To think about communication (even if we don’t apply it) means that we are aware of it. I’ve seen too many people who aren’t aware of their communication or the impact it has at work. But it is teachable, can be learned and for me, is the most important trait I look for in a tester…but being aware of how it impact our work I think is often the highest hurdle to overcome.CheersRob..

  7. My entirely unscientific and unresearched hypothesis, completely lacking in any objective evidence is as follows.Too often people have their head down and are focussed only on their particular job and don’t think about the big picture, the reason for their corporate existence or why they get paid something more than the minimum wage.Their purpose is not to provide information about the product under test, or … (complete with whatever reasonable definition of testing you prefer). Their purpose is to produce things. It doesn’t matter much what you produce. Everyone who does anything useful produces things. Conversations aren’t useful. They’re just chit chat. There’s nothing to show for it. You’re just wasting time you could be using to churn out the bug reports.Bug reports definitely count as output. Like test cases, you can pile them up and point to them. They’re also handing for referring back when it comes to the project post mortem, or performance appraisal time. Just look how thick that pile is!As I say, this hypothesis is entirely subjective and based on nothing more than a mixture of bias and experience. It happens to suit my prejudices, so I’m happy to go along with that for the time being.

  8. adding to what James said.. doesnt help that some contracts are based on these outputs … number of issues found. Another can of worms I am going to stay away from. Shhhuuuu

  9. Hi James,Thanks for commenting. Couldn’t agree more. It’s a culture within the software development world that is grinding this industry down, but there is hope. There are teams valuing people over process and end results over artefacts produced. Heck, I even know of a company who don’t care how many hours you work or where you work, as long as features get shipped. And you know what…everyone who works their LOVES it. But you are absolutely right. I tend to call this CYA (cover your ass). The more artefacts you have to point at in times of crisis, the more likely you are to not get blamed or to win that particular argument. The problem is, this breeds even more of the same CYA from everyone else until every person in the business is taking 8 times as long to complete anything because they are producing mountains or terrabytes of proof. Cheers for commenting JamesRob..

  10. I used to fall victim to this, then I started getting to know how my devs thought, how the products worked, and once I knew their headspace, it stopped happening. While I can’t always say it was my failing that lead to the problem, I can’t say that it was more often my failing than Dev’s. My role is to be an accurate observer to the bugs I witness, and failing to provide the data needed to corroborate the story I tell is me being a poor witness. If you can’t establish proof, then it didn’t happen.

  11. Hi Jessedyer,Many thanks for your comment. I think the communication breakdown often happens both ways but as long as one party acknowledges it and makes an effort to amend the situation then I think the problems tend to go away.It sounds great what you did and you’ve seen the results pay back tenfold. Great stuff.ThanksRob..

  12. I’ve not seen minimum targets for bugs to be detected, but I’ve seen maximum figures for each defect severity level. In practice this can be just as bad because the incentive is for user testers to try and get up to these figures to put one over on the supplier, while the supplier has the incentive to suppress figures.Even if neither party does that, the suspicion is always there. That’s highly damaging and corrupts the relationship between user and supplier.When bugs are being discussed and classed, it’s not a simple matter of analysing the bug and its impact. It becomes a highly political and contentious process in which both sides try to work out whether the other is being straight, or has ulterior motives.The rational response to these pressures and incentives can then be to distort the figures in anticipation of the other party trying to do the same. The whole thing deteriorates into a depressing exercise in game theory, rather than just doing the job professionally.

  13. Rob, I agree to this but it depends on context .Most of time bugs get re open because tester could not investigate in best possible way .If he talk to developer then also he need some time to investigate or look in to the bug until he can say anything about the bug.One thing for better bug report is we need better bug investigation skills.From communication point of view, I will say we should always ask developer that how we is going to debug the issue when he first see the any issue in product. We can also investigate in similar fashion next time and get enough data so it is easy for developer. I usually report almost every bug and if it gets re open due to local issues then I prefer talking/IM before updating next comment to speedup the issue.

  14. The best way to resolve this issue is for the tester and developer to go out for a drink and talk about the pain that one is causing the other. Once they start talking to each other and actually understand what makes life for the other person more difficult and what information helps them this bug tracking ping pong will be a thing of the past.Of course, one person has to make the first step and want to improve the communication in the first place, that’s where it stops in many places.

  15. Hi Thomas,thanks for commenting.Absolutely, The first step is acknowledging the issue then looking for ways to solve it. Often solves very easily indeed.Rob..

  16. I agree Rob. When I see a mode of communicating is not working in an instance, or in a general situation, I change the way I communicate with the person or people. Most often, I make it more personal. The exception to that is when I realize that the personal communication is the problem. I also subscribe to the camp that says the cost of bug fixing includes the overhead. If the overhead is high, in formal systems for instance, then the cost savings of finding defects early is lost at least in part. There could be reasons for tracking very tightly, such as situations that tests themselves need to be continuously evaluated for worth. But if you aren’t making pacemakers and bomb guidance systems, you could probably just point to the screen and tell a developer “you better fix that”. [Note: I give credit to Janet Gregory for “show me” defect reporting method.]Dave

  17. Dave McNulla do you have link to Janet Gregory work. I would love to read it.

  18. Thaks Rob for your comment .I misunderstood 4th paragraph in the post and thought you are saying not to report bugs in first place and just directly have talk with developers.Thanks for the link too.Nitin

  19. Written communication is always tricky – usually because we don’t write or attach “interpretation notes”! Communication – whether written, verbal or non-verbal (body language) actually needs a synchronisation to happen (from Tor Norretranders) – before real information flow (communication) happens. This is very difficult in the written form – so if that’s the only means used someone is missing a trick.How do I know any of what I’ve written is going to be understood as I intended it? I don’t – so if I’m really interested in communicating I should follow that up in some way…

  20. Hi Simon,Thanks for commenting. Excellent point you make about following up on the written communication. CheersRob..

  21. I agree communication is very important between dev and tester and you can’t go very far without it. We are not yet fully robots even with the modern technology. But now, how we communicate in particular depends on situation and context. If the dev is busy or issue is not critical, entering the bug looks the best approach. About adding detail I have to be honest and I recognize I am sometimes lazy. But when I try to add all the logs and Steps-To-Reproduce I find myself the real and more clearer issue (often misconfiguration) Entering bug can have other advantages: proof of work, proof of efficient work, tracking for new testers, etc.

  22. Hi Rob, nice post.I agree totally on maintaining face to face communication whenever possible. I’ve been bitten with bug tennis in previous organisations, it can become a particular problem when combined with the ‘wizards towers’ issue of having two developers owning separate parts of a system, refusing to accept an issue, and the hapless tester stuck with a bug ticket that is just being passed back and forth between them.I think that bugs can be boiled down to 3 essential roles:-1. A reminder to do a piece of work on a prioritised basis2. A packaged demonstration of a problem3. A token under which progress on an issue can be documentedIf we examine each in turn.1. A reminder to do some work is only necessary if we are not going to tackle that work immediately. If you are going to tackle the problem straight away, no reminder is needed, I try to take this approach on issues found in active development work where I try to limit raising a issue to when it cannot be addressed immediately and there is a risk that the issue might fall out of the iteration. 2. Again, if the problem is going to be tackled immediately and can be demonstrated, why the need for a packaged demonstration kit? If the work is to be deferred then a bug may be required to document the problem. However we need to bear in mind the information that can be lost between having the bug in front of you and trying to recreate it from a set of instructions. I’ve seen bug tickets before marked as ‘seems to have gone away’ only for that problem to re-appear later, as the full problem was not documented in the bug ticket.3. If there is not a simple fix, or the fix involved is high risk, then it may be that we want to track the decisions made on that issue. It is sometimes a useful approach to raise a bug to tackle an issue where the first attempt at a resolution failed as that implies a deeper problem or misunderstanding somewhere, however in this case the primary means of communication remains face-to-face, with the bug acting as a decision repository. On this last point, when you say “I’m a fan of tracking issues (though not necessarily in a bug report) ” – I’d be interested to know what other methods you have found useful to track issues.Cheers,Adam.

  23. Hi Eusebiu,Thanks for commenting. You are spot on, Context is King for everything we do. I do tend to raise most issues as bugs, but the more towards agile/lean/iterative a company moves to more we need to drop our need to raise bugs as proof of work. We need to instead concentrate on getting things done with the least friction..but again, context comes in to play.Thanks for your comment :)Rob..

  24. Hi Adam.Great comment. I love the three categories you’ve broken bugs down in to. Excellent stuff. Might have to use these three categories myself now :)I like to use agile planning tools to manage work. A bug is either some feature that is missing that shouldn’t be or some feature/capability there that shouldn’t be (or isn’t working). So in essence I think bugs belong as stories on a backlog. They belong with the other stories the team are trying to get done. They should be prioritised and scheduled (those that aren’t immediate fixes) just like a story.The business should make a decision on whether to fix the bug or do a new story. There are obvious stoppers which I like to see fixed immediately or bugs that break current stories, which again, should be fixed immediately, but everything else should live with the stories/features/requirements. That way the business can make a value based decision on whether to live with bugs and have new features/capabilities or get bugs fixed. This is easy to say when working on new stuff, but legacy code/projects always represents a different problem 🙂 One which could mean harsh realities like deleting bugs that are a year old……why has no-one fixed them yet? But it’s all context and skills and business based, so I doubt my way of working suites many people :)Thanks for the excellent comment. Rob..

  25. Could not agree more. The amount of times I’ve been sat infront of a bug report thinking “I wish I could just sit down with the tester and work this out”!

  26. Hi Rob, Interesting..Whoever started that bug report needs sacking. Who could reproduce it with all the missing information that was later added? Communication is vital, but it DOES need to be included and tracked in the report. Teams seem to be becoming more and more distributed. I have developers in Helsinki, Poland and Tampere, face to face is not an option, detailed reporting is a must. Not sure why Agile means we should raise less bugs, a bug is a bug and without a report can be missed? Agile offers regular (daily) team collaboration sessions when bugs can be discussed to avoid ping pong.

  27. Hi Charlie,Thanks for commenting. Couldn’t agree more. Communication is a two way thing and the bugs need to be thorough in the first place for anything to really get fixed.In a sense, you need less bug reports in agile because the whole team should be working on the same feature at the same time. In my experience this is great because fixing a bug is more about leaning over the desk and showing it to the devs (who should be working on the same story) so it’s normally just a case of fixing it straight away. As the stories are done in priority order then the story you have found the bug in is the most important (assuming you are working on them in the right order), so that story is not complete. In a totally agile world this would mean dropping everything to get this top priority story done, but in reality this is not always the case.The time to fix the bug is dramatically increased when a bug report is used versus fixing it directly in the code under the watchful eye of both dev and test, but unfortunately that means we lose the metrics many people like…….Good to hear from you again Charlie 🙂 We never did build that veg oil petrol station or pursuade Top Gear to start a football/car league. In fact, I think that idea is still on my backlog..Rob..

  28. I think this article is an overreaction to sensible QA audit approach. Issue tracking systems are is not intended as replacement for communication, but I have lost count of the number of times when an issue is regressed, you get blank looks as to how it was fixed last time. And why? Because no-one recorded the information anywhere. Bothering developers every time you find a bug is not sensible – it is only of value if you have genuine query. Whether you have a conversation or not, results should be recorded, even if it seems irrelevant at that time. To do otherwise belies an inability to think forward, only in the now.Keeping a good issue management system should also be backed by good workflow principles. Of course, sounding out developers prior to issue raising is good idea, but the assumption is that the tester is competent enough to raise good clear issues. Your article seems based on assumption that the tester could be anyone, not someone with testing skills. Whether or not an issue can be reproduced is neither here no there – there is value recording it and following process. If these “unreproducible” issues surface again, and you haven’t bothered recording the detail, then there is no history to help you or the developer. This is the old mistake of thinking that ALL documentation is useless – Agile principle is to minimise unnecessary documentation, but issue history should not be included in that. This article makes little pragmatic sense, as issue management and tracking is sensible approach to recording project history. Maybe the issue system you have been using isn’t very good(?). But I think resistance to formal issue management is largely down to peoples resistance to doing “boring” documentation. As someone who has worked 7 years in Agile/SCRUM environments (do varying degrees), it is clear some people use the methodology to hide behind when justifying lack of documentation. An issue management system is very much part of the daily SCRUM process, and the more information it has, the more people refer to it. Another good justification of thorough issue management is the poor testing sap that follows behind you when you leave! Give them a break, heh? 😉

  29. I have to add I would be sacking that tester who wrote that report – abysmal! Testing is a skill, not something anyone can just do – surprisingly prevalent attitude

  30. Firstly in the example you’ve given after the first bounce back it would be easier to converse with the developer than to have an ‘E-guement (emaill argument) with the developers. This shows the reliance on a the defect tracking system as a the single vehicle to communicate. One of the most valuable qualities of a tester is to identify issues and manage around them. Not just in software. But in conflict, politics too.

  31. That is just meaningless rhetoric. I am not suggesting no communication – I take that as a given. Its only in modern web projects that communication between developer/tester has become an issue. Largely because there are too many “frightened mice” who got into IT the last 5/6 years, without really thinking about why they did. The nathan barleys, the chancers, etc.. And you can “over-bother” developers – at least you should have to ensure you have a whole picture of an issue before bugging them! And developers like clear bug reports with all the information they need. It is not always convenient to ask questions at any time.There is is flip-side of too much conversation – not enough recording of the detail of conversations. Is that clear enough? You would develop without source control – so why shouldnt that centralisation of information principle apply to any project information. To do it any other way is plain lazy.Just to clarify, I have been a test manager for over 7 years, and a tester even longer – on over 35 projects in total at 28 companies (large and small). So I think I have a very good view on weaknesses – and recording information, and ensuring information is up to date is a very big weak spot on projects.

  32. Hi Paul,I may have lost a comment somewhere along the line as your final comment seems aimed at something someone said, but it doesn’t flow well.It’s obvious you are passionate about the topics you talk about and they are all very valid points, but I can’t help but feel you missed the point of the post.The post is not about replacing the bug tracking system with face to face communication, although many teams have very successfully done this, but more about using person to person communication to talk through the defects, rather than a defect tracking system. The defect tracking system, in my opinion, is about storing the decisions, the conversations, outcomes and actions. It’s not about having a conversation.If you are using the bug tracking tool to communicate (i.e. converse) to each other than you will have ambiguity. Many people struggle to express in writing what they can explain verbally and with a real demonstration of the bug.At no point did I suggest you never use a defect tracking tool, but use it with caution.Every context is different. One project I worked on would have meany I flood the system with defect reports if I had logged them all. This meant we wouldn’t be able to see the wood for the trees. Other projects have meant that we could log each one, mainly because we had to; the team was remote.On more recent projects I’ve not felt the need to record each and every defect because we were still working on the story. The story isn’t done until it meets its acceptance criteria. If it doesn’t is there value in raising a defect? Maybe? Maybe not? Some teams may want to, some teams may not. But to raise a defect to converse with a developer, particularly in an agile environment, is insane. You are right, there are times when we could be bothering developers too much, but in an agile team this becomes less of an issue, in my experience, because you are all working on the same story(ies). In my experience, developers love to have defects reported verbally, reproduced on their environments, fixed locally with you watching, checked in and immediately spun to the live environment. I love this. It is SO much faster. But it might not be right for all environments and all teams. I wouldn’t have sacked the person who wrote that report or the developer who continued it. I would work with them to improve their communication, to understand why it was wrong and to work at ways to improve their communication moving forward.I’m not sure where you got the impression I was for NO documentation, NO recording of defects or NO tracking of process. I’m all for that, where necessary, but that has nothing to do with communicating defects with yourself and the developer. I also tend to think in terms of a team, rather than them and us, which, in my experience, aids in the breaking down of barriers. This blog post was just my own thoughts on where I am in my career, my experience and my workplace. If it’s not right for you, that’s cool and thanks for commenting.Rob..

Comments are closed.