Observing to help our Testing

Whilst on the train a few months back I spent some time observing how people were using technology.

Some were using the tech as I assume it was intended, some were “street hacking” the products, whilst others had adopted unique ways of utilising technology (and other devices) to fit the context they found themselves in.

I often wonder how effective testing can be (and overall design and development) when we have little insight or understanding of how our applications/systems are used in everyday life.

 

We can make sure it’s “functionally complete” or “performant” or “meeting requirements” but that doesn’t mean we’ve helped to create a great product right for the audience it’s aimed at.

I think Testers can help greatly in this respect, either by simply asking questions at all stages of the design and build process but also by getting out and seeing end users in situ (if you can).

 

Many of us assume, predict or speculate how our end users are using the software, it’s often all we can do, but I wonder how much of this speculation is accurate. I often hear it said that Testers “pretend” to be the end user, which is fine if you know who your end user is, and in what context they will use your product but this approach should never be used as an absolute point of view.

No-one will ever be able to use your applications in exactly the same way as your real end users, but we can certainly work hard to get close to it.

 

Here are some of the things I observed:

• Several people were using multiple mobile phone handsets. Work and Play? Feature versus Feature? Or simply taking advantage of the best tariff? 
  • What about data on each phone? Did it need to be synched?
• One person was swapping sim-cards in and out of one handset to make use of the best tariffs.
• Two people were using a technique called “flashing” or “beeping” to communicate with someone else via mobile phone. (flashing is where you ring a phone and then hang up before they answer. There are all sorts of social norms growing around this practice – good article here: http://mobileactive.org/mobile-phones-beeping-call-me)
• One person was trying desperately to get a good photo out of the window on his mobile phone whilst the train was moving. I’m not sure he was happy with any of them. Misplaced expectations?
• One lady was using a USB expansion device blue-tacked to her laptop lid. I “assume” she was doing this to expand the capacity and/or maybe to reduce the overall width of the machine (the two USBs she was extending were bigger than the expansion USB itself). Street hack? Do we need to test any usages like this (i.e. overloading original intention)
• The person next to me complained that his phone was running out of power too quickly. He’d been playing Angry Birds for the entire journey whilst using the phone to play music.
  • Expectations of battery life needing to be longer to support modern multi-usage?
  • How did both apps perform?
  • Which one used the most power?
  • Should we be thinking about how much power our apps consume on mobile devices?
  • Can we realistically even measure it?
  • Should we rely on the underlying platform to manage such usages?
• Many of us live in an age of “smart phones”. How does this “one for all” device fit with other electronic devices like eBook readers, Tablets, Laptops etc
  • Can we read something on a Kindle and start reading where we left off on an iPad at home or Laptop at work? 
  • Can we seamlessly share information between devices?
  • How is our information distributed, stored, secured, managed, protected? Do we care?
• Another lady caused minor fits of giggles as she took out her “early designed” portable DVD player. It was pretty huge by modern standards. She proceeded to play episodes of Friends on it as she slept for the journey. Old tech and usages is a major challenge for many Testers.
  • At what point do we stop supporting old tech or old versions of our software?
  • Do we need to test all versions?
  • Are there techniques for testing many platforms/versions at once?
  • At what point does this old tech stop being “acceptable” in society….which society?
  • How do we know what versions our customers use?
• One man was reading his Kindle on the train. He had created a hook for his Kindle. The hook was made from a coat hanger. This hook was suspended around the ceiling hand rail so he could essentially suspend the Kindle at eye height. At the bottom of this bizarre Kindle concoction was a ribbon so he could stop the Kindle swinging around with one of his hands. Street Hack?
  • Could we have ever envisaged it being used this way?
  • Would it have changed the way we tested or designed or implemented?
• Whilst on the train it is inevitable that the phone signal will come and go. With bridges, buildings and shady coverage it would be ambitious to assume total connection for extended periods. Yet I observed a number of people “hating” (i.e. swearing and getting annoyed with) their devices when they lost connectivity.
  • Two people physically threw their phones on the tray table when they lost a voice signal. Others uttered offensive and angry rants at their devices.
  • Yet the problem existed outside of the device or software in use; it existed in the infrastructure. Are expectations outgrowing reality?
  • The infrastructure no longer supporting the norm and expectations of device usage. This is an interesting challenge for those creating devices relying on technology infrastructure like broadband and mobile networking.
  • What’s the minimum connectivity? What’s the risk of it going down? What happens if it does go down? Does it recover? Do we lose data? Does it matter?
• One person was drinking heavily whilst attempting to use a Smart Phone keyboard. I won’t repeat what he said, but he struggled to type his SMS. One interesting point he made was that he used to be able to type whilst drunk on his old phone. Loss of capability? Extreme contexts? Trustworthy observation?
• Two people were “chatting” to each other via their mobile phones (maybe Skype or other chat system). They were sat opposite each other.
  • Are cultural changes in communication reflected in your products?
• I was sat in a quiet carriage. As usual there were a few complaints and confrontations about noise. One of which was about the noise being made by the keyboard of someone’s laptop.
  • At what point do we pass social thresholds of acceptability? Do we need to do more Inclusive Design?
  • Should we consider more extreme contexts when testing or write these off as edge cases?
  • Could we ever imagine all of the contexts our products may be used in?
• I kept turning on my wireless hotspot on my phone to sync my field notes (Evernote) as the train stopped at each station. This raises questions of synchronisation, offline working issues, data storage on the cloud and a whole host of privacy issues. There are some interesting examples of how people are using tools like Evernote in the wild.
  • Do you have stories of how people are using your applications?
  • Do you actively seek stories from end users?
  • Do you use this data to build user profiles and system expectations?

Does this really have anything to do with Testing?

There are a few people starting to talk about social sciences in Testing and I believe the application of many areas of social science are going to grow and diversify as Testers seek to find out more about people and technology and how our products can find the right audience(s).

Social sciences can give us a deep insight to ourselves, culture, mass media, communication, research, language and a whole lot more.

Observing people is just one element of research. Research will give you insights, clues and information about the things you research.

Observations and research will help you to make decisions on what to focus on, and what to overlook.

Testers are also natural skeptics; we should help to experiment, prototype and challenge simple assumptions and social categorisation (i.e. all young people use Twitter, wear hoodies and listen to hip-hop) *

Observing people in every-day life is often the biggest eye opener for any Tester wanting to learn more about people and tech.

 

Why not have a go? Focus on the world around you. I bet you’ll see stuff you never saw before. I bet you’ll see people using tech in ways you’d never seen before. I bet you’ll learn something new.

The challenge is how you can bring what you learn to your Testing.

 

As usual, if you have a go, let me know what you think.

 

* This is a real generalisation I heard someone say at a technology, culture and local council meeting!