Who is most qualified to weigh in on the quality of your mobile app’s experience? Hint: It’s no one on your dev team. It might not even be anyone at your company. The best judge and jury of your app’s user experience are real users. They’ll rate their experience later, when they finally get their hands on it. But, later isn’t the ideal time for you to find out you’re not delivering the goods.
The biggest and best tech companies bring real users into every stage of their research process, from concept to release. Yet most companies have room to improve their customer experience, probably due to a few remaining myths surrounding how QA is done. Having built a career as a software tester and founded my own testing company, I’ve discovered that it pays to think differently when it comes to perfecting the customer experience. Here are a few of the myths I’ve debunked over the years:
Myth 1: User stories deliver the best customer experience.
Engineering and development teams know how their product works best. They made it, after all! In the testing stage, though, that kind of familiarity is like looking in a mirror. You can only judge your own image according to your own expectation of what you’ll see. What you need to know, however, is what other people see when they look at you.
User stories tell how a feature should work in a given scenario. Internal testers can validate it. External testers, however, tell us if a feature worked in the scenario that matters most: theirs. User stories, after all, are speculation. Without external testers, you don’t know how users truly experience or fix your product. We’ve helped CBS Interactive, for example, to reproduce bugs that they may be aware of but don’t immediately know how to fix. Without an outsider perspective companies waste countless hours trying to reproduce bugs instead of actually fixing them, but it doesn’t have to be that way.
User experience is inherently personal. Think about who really knows how you, personally, use each room in your home — whether you like to read in the sunroom, or do most of your socializing at the kitchen counter. It’s not the architect, nor the contractor — it’s you. Similarly, the people who know the real paths they take in your product’s workflow may not your dev team. It’s the people you hope will take the journey.
Myth 2: Beta testers make great QA testers.
Your beta testers share something very special with your developers and QA team: experience and understanding of your app. That experience is a great thing until you’re looking for a fresh perspective. Knowing how you have used an app in the past will influence how you use it today. Another analogy: The first time a child sees a gate, she doesn’t know that the latch opens it, much less how. In fact, she might not even want to go through the gate at all. She might even try to use it for something else, like a climbing ladder.
Your app’s beta testers are no different. They have expectations and familiar routes. They are already conditioned to working around what might have once been less intuitive. A lot of the time they don’t know how to reproduce issues or don’t know how to find the key technical causes of a bug. Since beta testers tend to look at the app from their point of view — which may be different than most of your customers — they don’t always make a difference in QA.
New users are a blank slate. They can tell you that your gate’s lock is not intuitive. They are much more aware when anything isn’t the best it can be — or when what they needed was a ladder, but you gave them a gate.
Myth 3: QA teams can perfectly envision the customer’s perspective.
The best customer experience improvements come from a combination of internal resources and real-life users. You want quality pros at the helm whenever possible. They provide structure, process, consistency and guidance. But, recruiting a fresh perspective for feedback can help broaden the narrow view of internal testers. Whether they’re validating app functionality or providing feedback about your implementation, real users across a range of demographics, devices and experience levels give great insight into your app by simply executing their tests in the wild, as only they can.
Apart from the service we provide, our clients find value in our weekly bug reports and regular check-ins so they can understand their strengths and weaknesses in the development cycle, not simply check another box on the way to a build release. Teams like Outlook Mobile and Salesforce know that a single test cycle could reveal a bug that capable of upending development priorities or even causing teams to recognize and fix it. Reporting doesn’t just save time — it saves money, too.
Myth 4: Enhancing customer experience isn’t a development priority.
The inherent speed of agile methodologies doesn’t negate the need for a killer customer experience — just as having timely QA doesn’t negate the need for testers to crush hidden bugs. While some organizations save QA for the end of development, it deserves the budget and resources of the whole engineering team. Consider this: Most companies spend 31 percent or more of their IT budget on QA and testing. If you’re not there yet, you definitely have the opportunity to enhance QA’s impact on the customer experience.
Companies like Lyft are now competing exclusively on the customer experience. They realize that quality is so essential to their brand that they can’t afford for testing to be a "nice to have." If your QA team is overwhelmed with projects or your app rating has plateaued, ask yourself: Which customer experience myth are you buying into?