Crowdsource Testing your Application

[Note: This is another in a series of posts by vendors I’ve invited to write a guest post here because they have something to offer of value to microISVs. If you have a product or service that helps microISVs succeed and would like to do a guest post here, please email me at bob.walsh@47hats.com.]

By Stanton Champion
Marketing Manager
uTest

Developing a new application, whether it’s online or offline, is always exciting. Who doesn’t love speccing out a new tool, writing fresh code, and watching ideas turn into reality. Yet when it comes to testing, we so often come up short. Doubt sets in, and we’re never quite sure if our code will really work.

  • Will those text labels really work with reverse fonts like Hebrew?
  • Am I sure this HTML will render correctly in every browser?
  • Does my application function correctly with all the latest operating system updates?
  • Do I really know how this my product works with every antivirus and firewall combination out there?

The reality is that truly robust testing is time consuming, and making sure that every possible combination of features, localizations, and platforms works the way you expect is nearly impossible. As developers, it’s no wonder we approach testing like a chore.

All too often we give our customers buggy code and hope for the best. If our SaaS application doesn’t render in IE6 or our killer business tool won’t work in Windows 2000 with Cyrillic localizations, we just rationalize it away and promise solutions in a later release.

Our fear of testing always produces the same outcome: The Customer Loses!

What is Crowdsourcing?

Thousands of people want to help you make your product better, and they’re in places around the world both near and far. Some of them may live down the street and some of them in places on the other side of the planet. They’re all part of a global crowd, ready to pitch in and contribute their knowledge and skills. The Internet makes it possible for you to connect with them, share your product, and learn from their experiences.

While members of a crowd devoted to quality assurance will all share a love of testing, they’re otherwise all pretty unique. For you, that’s a win! For example, it could be the tester in Hong Kong running Windows Server 2003 who discovers that your application crashes when it tries to read files containing unicode written Cantonese characters. Or it could be the tester in Brazil running Red Hat Enterprise Linux 3 who discovers that your application relies on features in glibc only found in version 4 or later.

Did you think of all these scenarios? If you’re like most developers, probably not all of them. So what does crowdsourcing offer that you can’t find anywhere else:

  • Access to diverse platforms, languages, and people
  • Real insights from the real world, not just made up test case results
  • Testing done by hundreds of people at the same time
  • Rapid feedback right away

Not like Outsourcing

Crowdsourcing isn’t like outsourcing. With outsourcing, you give your code to another company and expect them to do the heavy lifting. They hire people to do the testing, attempting to replicate as many real world conditions as possible. What makes crowdsourcing valuable is the variability that outsourcing just can’t replicate. Your application gets the same kind of analysis that only comes from having hundreds real people use it on hundreds of real platforms.

Good fit for Agile

If you’re using Agile methodologies, you’re in luck because crowdsourcing is a great fit for you. As you develop your application, testing has become a continuous process. While you may be using Test-Driven Development (TDD) techniques in your project, it’s still critical to make sure your project really works outside of automated test cases. Integrating crowdsourcing techniques into your project along with other Agile testing principles can provide new insights you may have missed before. Because crowdsourcing emphasizes rapid responses, you can incorporate feedback into your next iteration just-in-time.

uTest Leads Crowdsourced Software Testing

uTest is the global leader in crowdsourced software testing. Based in Ashland, MA, we provide access to over 7,000 testers from around the world with hundreds of languages and platforms. Our testers are compensated for finding bugs that are actually valuable and important to you, with different prices for different kinds of bugs. We call it “pay for performance,” and it’s our way or making sure that you get the most out of our testing platform. We offer both an on-demand model tailored for companies with periodic test cycles and a subscription model tailored for Agile companies with continuous test cycles.

If you are interested in learning more about uTest and crowdsourced software testing, visit our website at http://www.utest.com or download our whitepaper,
Community-Based Testing For Consumer Software. For more information or to see if uTest would be a good fit for your testing strategy, contact our sales team by emailing sales@utest.com.

Comments

  1. So I went to their site, because this is something that would be very useful for my MicroISV… *NO* price information anywhere, not even range or a any hint. My reaction was exactly as Bob tells us in his books or on this blog: instant fail. I’m not going to BEG them for their service. If they can’t tell me the price up front, their loss.

    BTW, I find the claim that their testers share “love of testing” absurd to say the least. More like deceptive, nobody *loves* tedious, repetitive, boring, thorough testing (you know, the only kind that is actually helpful).

  2. Hi Peter,

    You’re right, we don’t have pricing information on our website. We’re discussing this now and trying to decide if we want to have more details online, especially given our variable pay-per-bug model where prices reflect the number of bugs actually discovered and your budget. However, I really do appreciate your feedback. This is definitely an open issue for us.

    As for our tester community, I have to respectfully disagree. A large fraction of our testers are professionals with regular testing jobs, and while not everyone enjoys the work they do, our community is filled with people who have chosen software testing as a career. We’re very proud of our community and their involvement with uTest.

  3. Abhijeet says:

    What if developers purposely leave some bugs in the application (that is going to deploy on uTest) and then create a bug through uTest community and get paid for the same ?