I may well be opening a can of worms with this blog but I am genuinely interested to see what people think about crowdsourcing and in particular services like uTest.
I’ve read some glowing reports about uTest and heard nothing negative – which is unusual in our community. We often see past the gloss to reveal the real story. But maybe uTest and crowdsourcing really is the future.
So I’m genuinely interested to hear from people about why crowdsourcing is becoming increasingly popular amongst the testing community.
I have my own opinions and views on crowdsourcing but they are formed from the little experience I have with uTest. Let’s just say I found the payment, project allocation and script assignment somewhat confusing and unfair – but that was right back in the early days, things have changed since then.
I also have a concern that lots of testers are working for nothing. No bugs, no pay. And the stats seem to show that too (on the assumption that everyone registered is testing – which I know is not right). I know it is their decision though.
For example.
In the UK there are 778 registered testers. They have had 270 test cycles between those that take part. And they have found 1159 defects. Not bad. But it works out about 1.5 bugs each (assuming all take part – which we know not to be true).
It fairs slightly worse in India. 5096 testers, 430 cycles and just 4242 defects. On basic and simple maths that’s less than 1 defect each.
Spain has 142 testers, 32 cycles and just 78 defects. Half a defect each. Not brilliant.
America fairs a bit better but I know these numbers don’t tell the whole truth. It could be one tester raising hundreds of bugs. But why are so many registered and not taking part?
(numbers taken from http://www.utest.com/meet-testers – assuming this is up to date)
So please do leave comments and let me know what the benefits are of crowdsourcing services.
I am genuinely interested to see how the crowdsourcing model works for those doing the testing.
- Are you making money? (note: please don’t disclose how much you are making – a simple yes/no.)
- Are you learning more about testing?
- Is the 20,000 + community really all experienced testers?
- How long do you test on average for before you find something?
- How often do your bugs get bounced back to you?
- Are you simply running basic scripts or being utilised for the creative, analytical and questioning minds that you have?
- Is the paid per bug scheme based on luck (i.e. which test scripts you get?) or is it based on skill (i.e. being picked for your ability?)
- Should more companies be getting involved in crowdsourcing?
- Does it solve the testing problem? Or simply solve the outsourcing problem?
There’s some suggestions to get you started.
I look forward to finding out more.
Rob..
Rob,I’m not confused. I’m just shaking my head in disgust. Nuff said.
I understand that there is lot of skepticism around utest and what it is doing to the community. I had similar questions as you but then participated in a bug battle to find out the experience.I see it this way:For testers1. It is a practice ground – irrespective of their bugs being accepted or not. 2. It is a learning ground – by looking at bug reports that are good, my reporting skills can improve.3. It is an additional pocket money – Most testers registered with utest are employed elsewhere and already earning enough to feed themselves and their families and hence the money they earn there.4. It is a networking ground – I could meet a lot of testers from various countries and organizations and backgrounds and skills and blah and blah and blah5. Solace: You might or might not know how many testers I meet here in India who do not get an opportunity or freedom to perform exploratory testing and have to show their productivity on a per hour basis. They could be potential excellent testers being asked to follow scripts and this can be a solace for them because they are their own manager. ( Of course there are some test case writing assignments in utest too )Now coming to those numbers:// It fairs slightly worse in India. 5096 testers, 430 cycles and just 4242 defects. On basic and simple maths that’s less than 1 defect each. //That’s 5096 registered testers out of which hardly 500-800 might have participated so far. That’s the case with any country I guess. Plus, you know – reported bugs < found bugs.Coming to my utest bug battleI just wanted to see if my bug reporting skills are decent enough and it turned out to be true. So, I won the top bug award from them. The money I made is public – $400. Now, I reported 4 bugs in bug battle: and I won $400 in one hour – that’s like $100 a bug 🙂 / Highest paid tester / ( haha, just for the sake )Well, that can happen only during bug battles. In releases I have to be careful of which bug I am reporting. There is a mechanism to challenge the decision if my bug is rejected but I need a lot of time to do that.However, how much do I care about the product I am testing – not much. Pay per bug model, in my opinion has its own pros and cons. I think we testers and thinkers should work with them to focus on its pros and try to challenge ourselves to solve those cons.I have challenged the active participation of the 20K member community in their forums and that also shows I am not biased about my opinion and am neutral.What testing problem does it solve?I have met 3 startups who said “we need more hands but cant afford it”. Well, I would prefer they say, “We need more brains but cant afford it” – I still understand they meant more testers.The cost of hiring more testers for a major release is painful but the hiring cost can be invested on utest, lets say $2500 and they can have a coverage of platforms and country and usability dimensions of various testers. So, there could be a quick feedback but it could be huge in some cases that the management fails to understand how to tackle it.So, it can solve one problem and create another.The skepticism that I have:I report a 100 bugs for a product. Only 20 of them approved and rest not approved. So, I get paid for 20. However, what is the guarantee that the organization is not using my information from the other 80 bug reports?To discover all these: I have dived in to swim with it for a while to have a better understanding and also be of help to utest if possible.need more from me?
Hi Jim,Thanks for the comment. I think a fair few people feel the same wayThanksRob..
Hi Pradeep,Many thanks for the lengthy and detailed answer.I like your honest and forthright style.I agree the numbers should not be taken at face value. Like I say many of the registered users don’t actively take part so the bug find ratio is probably higher per tester.I also like the sound of a start up being able to throw more ‘hands’ at the product than they would have reasonably been able to afford. I will hold reservations on whether this is solving their testing problem though. More hands doesn’t always improve the quality. It can often have the opposite effect.And congratulations on winning the bug battle. I too was wondering what sort of honesty there was with the companies reviewing your bugs. There appears to be nothing documented on how uTest stop them rejecting a bug and then going away and fixing it. I guess the assumption that many will not follow up is also a high factor in potentially rejecting bugs. Would be interesting to see the transparancy here.I too believe most testers are employed already and so it brings about a little bit of training, practice and ‘fun’ in their spare time. But this is where I fail to understand the benefit from the customers point of view. A dedicated, passionate and enthused test team would be more beneficial. maybe?I’m also a firm believer in testers understanding the culture, company ethos, product, customer base, domain and any other piece of information that makes our testing roles more effective to our clients and I simply don’t see this happening with these crowdsourcing models.Sure, we can test anything at any time. But when we know the background, the person who matters, the context that the software is used in, the audience, the purpose and the commercial angles we can offer much more rounded, valuable and targeted testing. I also believe testing isn’t something that should be done after the code is dropped. Testing is about being integrated as a team, using static testing techniques, forming relationships with the programmers, understanding the whole development team mentality, offering help and advice throughout the lifecycle, getting involved in the requirements and not just a push it out, test it, push it back process. This breeds long and drawn out communication cycles (like you mention) and the inabilty to create a dynamic and rapid delivery of software. But then I am making the assumption that the uTest customer doesn’t have an interanl test team – which could well be wrong.I can really see the benefits. I absolutely can. And thanks Pradeep for your candid response. Always nice to read your views on things. It’s helping me form a more solid idea. But I still have my doubts about the real benefits for the customer. I can clearly see some benefits for the tester. But like you say “… how much do I care about the product I am testing – not much” And I thought the same when I tried it. And for me, the care, the passion and the buy in are what drives me to do my best testing. And if I don’t have that with a customer then they aren’t really getting the true value they could be.Rob..
Hi Rob,Good post and Pradeep’s input is very helpful.I posted a similar challenge on my blog in August here: http://uktmf.com/index.php?q=node/198 – check it out. (It’s been read c. 350 times but no one has responded to me).I took a slightly different tack but like you, I’m somewhat sceptical on how a crowd can help. I think it can, but it’s a small niche market. Nothing like the over-hyped publicity would suggest. Utest are backed by VC aren’t they? Do they need the hype to get funding perhaps?Anyway, since I wrote the blog I’ve thought a little more about this and the benefits seem to accrue to two kinds of clients:- developers of tiny, toy products – e.g. $2.99 apps for the iPhone – things like this might be fun to test if you’re an enthusiast- clients who really do not know what testing is or can do for them – e.g. startups with very little money and less sense.There are probably other niches – but where the crowd could help, is also where I see the difficulty in managing it. If you have a more substantial product and it is business critical (1st products from startups usually are), would *you* rely on a crowd or test it thoroughly and use the crowd as a mopping-up approach?From the point of view of the tester I can’t see how the potential financial benefit is worth the effort to employed testers. Winning $400 for an hour’s (fun) work sounds fantastic. But how many people took part in the Bug Battle and didn’t win – i.e. lost their hour? If you back yourself to be truly exceptional and competitive, or you are a true and generous enthusiast, or you really have no idea what you are doing, you might go for it. But would a professional tester compete with 10, 100 or 1000 others with such a low probability of winning?I think you should save your hour. Betting five quid/dollars on a horse could be more fun and more likely to be profitable (and certainly more transparent).I’ve read about 70 pages of “The Wisdom of Crowds” and am quite liking it. Maybe crowdsourced testing is making some use of this wisdom. I haven’t read enough of the book to judge really, but I very much doubt it.Paul.
Hi Paul,Thanks for your detailed response. It seems we share similar views on the topic. I like the way you refer to small apps like mobile phones etc. This is exactly the same market I see benefiting from crowd sourcing. I also agree that crowdsourcing is a good way of mopping up. Awesome way of putting it.I’m a firm believer in good testing being more than just testing the app at the end of the cycle. Something I can’t see crowdsourcing addressing.The Wisdom of Crowds is a superb book and I totally subscribe to the theory. But that only really works when you let the crowd do the decision making which I don’t see happening on software testing crowdsourcing.I’d be interested in chatting some more about this. I find it a fascinating topic. I’ll see if I can make it along to one of your test management meetings.Thanks for spending the time to write a lengthy comment with insightful comments. Rob..
Hi Rob,I also put this comment on Paul’s blogI’ve been thinking about testing with crowds (crowdtesting) since April this year (see my blog http://www.testingthefuture.net/2009/04/testing-and-crowdsourcing/). I think there are some more benefits from using a crowd.Like Pradeep said in having more ‘hands’ when needed. Like you said by testing more configurations. But for clients they have benefit from no long term contracts to testing experts and also give the power to your user. Your user will test for you and tell you what he thinks.We did a pilot in Holland with a game (I know interesting to many) and the client was very enthousiastic about the response he had. We didn’t yest have a payment model behind it so I agree that is something to work on. Pay-per-defect seems nice but more is needed. Something like a rating on how good a tester is perhaps. A tester can showoff how good he is at funding bugs instead of saying he’s a certified tester and hasn’t tested in years.Also one thimg I think is good tested with crowdtesting is usability.Crowdtesting also should e focussed more on accepting then on finding bugs (from viewpoint) of the client. Testers should see it as fun, practice and extra money. Not do it as a full time job.I think there a more things to crowdtesting then you state, and also more benefits. My biggest question is getting the correct crowd. When you only have novice testers in the crowd what bugs will they find? Sure some are good and some are ‘dumb’ enough to find that mistake that is needed. But still. I think the biggest flaw of the crowd lies in the buildup.-Ewald
Hi Ewald,Thanks for the comment. You make some really good points. But I’m always skeptical of throwing more testers at a problem, especially if you don’t know how good that tester is.If the testing is simply treated as a checking exercise then throwing more testers at the testing might work, but so would a well planned automation strategy or better planning for the testing.My real issue though is that testing in a crowdsourcing arena is generally treated as something that is done as a phase, at a certain point or at the very end.This undermines the value of what a tester does. It takes away any value they may add during the project. It treats testing a separate entity rather than a principle that exists throughout the project.I believe there is a lot more to crowdsourcing than I’ve stated in the blog post. Far more to it. But I’ve not yet found it and all I’ve seen so far is the glossy marketing speak and nothing but positive stories. Maybe it really is all rosy and good, but I somehow suspect that it’s not.I think the contract point is a really good one. Maybe not being in contract with a tester is a benefit to some people. For me though, having a dedicated contractor who becomes part of the team, learns how things work, how the team work, how the business works and brings value throughout all stages would be far more cost effective. Maybe..Ewald – please keep me informed of any further experiences you have. I genuinely am interested in finding out some more about crowdsourcing. The more information I have the more I can formulate a view of it.Thanks for commenting.Rob..
The issues raised here can (and will be) addressed. Building better crowdsourcing solutions is not rocket science. I agree that this option will remain an additional ‘arm’ of in-house testing group and will not replace them. However the possible time saving, cost saving benefits are too high to be over looked.We have plans to try and chalenge the issues raised- how do you know who is the tester and how good is he? how can you control the process? how do you monitor the tester’s work? how can the tester be sure he will get his work’s worth paid? and others. The goal is to make crowdsourcing testing more similar to your “regular” testing, enabling you to add and remove resources at will, getting testing results in days rather than weeks.
Hi Gil,I’m sure there will be possible answers to many of the tricky questions and I will be interested to see how these are addressed and whether or not it truly is a viable solution to outsourcing or any other problem.As a customer of Testuff I just hope you don’t spoil the good test management solution you have in place. Thanks for commenting.Rob..
Rob – no intentions to spoil i, on the contrary :-)Keep on writing, we’re enjoying it here. thanks.
Hi Rob,As you can see i’m with it & down with the kids & its only taken me 2 years to find this post & respond to it.We touched upon the crowdsourcing/uTest topic at the Manchester meetup so I thought I’d elaborate my position a bit more.The reason why I signed up to uTest was purely mercenary – I use the money earned for things like diesel so I can carry on doing my hobbies – namely kayaking, which involves long drives into deepest, darkest Wales looking for that adrenalin rush I crave.I don’t think I’m using the business model correctly – It appears I spend far too long crafting a useful, informative bug as I’m definitely earning less than minimum wage!In my corporate role, I like to discuss my observations before raising bugs. Collaboration is quite tricky, especially through the particularly poor uTest interface.I’ve put in a couple of feature requests, one being Testers are allowed to add comments/messages to other Testers bugs as some of them are shocking. I’ve been burned a couple of times for raising a duplicate bug. When I go to the original bug it holds no relevance to the bug I’ve raised.Which leads me onto the other bugbear – you are completely beholden to the triage team – they are the gatekeeper *shivers*. Whatever they say goes. You can dispute their decision, but if it goes against you it impacts your rating, so is it even worth the risk?On the plus side (maybe cos its close to Christmas & the client is feeling generous) I’ve just earned $85 in one evening for watching a couple of movies on a couple of devices – very handy at the moment.So in summary (finally, rant over) I agree with several of Pradeeps points. Its a useful training ground, & it earns some pocket money. Yeah it has some frustrations but I can get the wife to help & when my girls are a bit older I’ll be giving them some apps to test as well (they all have to earn their keep!), with me giving the guidance on all the other roles a Tester can fill other than bug basher.
I’ve personally been trying utest and some of the other options to learn more about it. As excited as I am about the potential, I’ve got major concerns about the negatives for new testers. uTest has awesome training areas and I like much of the platform. The pay for bug model has a few significant flaws that are a problem.
I see great opportunity for crowdsourcing to improve. Right now there is one way feedback and poor communication hampering the usefulness. Not every project is run fairly, and it depends who is running the project what the experience is. Some projects were amazing! Another one literally made me cry it was so negative. I’d never want a new tester to go through what I did. After asking flat out “Is X area in scope? I’m not sure from the notes which specify that only some forms are in scope.” They assured me it was in scope in the chat area. I wrote a detailed bug. They then rejected it–each rejection lowers your score so you are penalized if they don’t agree it is a bug. They also said me entering characters into a comment field was “abusive”. What a crazy hacker! Me typing. To the boundary. With plain characters. Certainly worth a punishment in that particular uTest project. I feel the current model encourages new testers NOT to report bugs in some cases, and tries to minimize communication. A preferred solution would be a “no pay” option for if a bug isn’t useful, but still follows the rules. Some of the test project leads are just lazy, so they reject bugs left and right. I mean well over half of the bugs on every project is rejected. Now that I’ve seen this game, I don’t agree to any projects with a list of hundreds of known bugs that aren’t minor. Obviously, they don’t care. They are just trying to cover their butt that the “did testing” without taking quality seriously. I’m not in support of their antics.
Other projects are clear about what they want. They are responsive, consistent. Some even pay for you doing something with or without finding bugs.
It isn’t enough money to be worth the time for Americans. Really. You’d earn more money ebaying stuff from your garage. It is worth doing to gain experience with something new though. I appreciate learning more mobile app testing. I appreciate the projects that were positive, and I know the signs to avoid the projects that aren’t run in a way that I feel positive about contributing to. The choice testers have is do you accept the project or not. That’s about the limit.
Wow – thanks for the insightful and interesting comment Lanette. It seems you’ve had both good and bad experiences with the UTest model. I think it does well for those that are “covering their butt” – it shows testing was done, it just doesn’t show the quality of that testing. As a proving ground and practice ground I think UTest does work well and some websites do benefit from the crowdsourced model for sure. I reckon if they nailed the payment and the “quality” of tester guarantee for customers then it’s certainly the model of the future.
Thanks again for the great comment.
Rob