We’ve come a long way from when product changes were driven by the most opinionated and vocal team members to a place where unbiased data and metrics are used. Now, designers and product managers make decisions based on real-world information. But there is always pressure to build the best product in the fastest time possible.
That’s why we often tend to build products first and think of optimizing post-launch via A/B testing. When time and money runs out, usability testing is sometimes one of the first things to get shelved. But this approach is problematic because of two core issues:
Recently, we were speaking with a startup that shared that in their last release, they did A/B testing for a new feature launch. So, version A had the new feature while version B didn't.
After the test, the data indicated that 63% of the audience preferred Version B, so the new feature should not be launched. However, they did realize that the test didn't tell them why the feature didn't work and what they could do to improve it.
If they had done usability testing first, it might have revealed that the 63% of users who did not use the feature simply did not see it or did not know its purpose. This would have given them the insight to add better onboarding, improve the placement, or provide more context to the user — everything that could have made the launch a success or told them why the feature is not worth launching. Quantitative data can not cover up for qualitative insights.
A/B testing seems to be faster and cheaper in the short term. You can check off the task that you shipped a feature or product. But you won't be able to prove with certainty if it will add business value or if people will use it the way you intend them to.
Divya, a seasoned UX researcher and leader, recently shared with us this core insight: “A/B testing will tell you much later if the feature worked or not while usability testing will provide you with instant, near-term feedback. And it will help you deliver the best design.”
That's why usability testing is a must.
With usability testing, you can let the user guide you towards the experience they will love. You don't have to wait for a finished product to take it to users for feedback. The best part is — usability testing can be performed at any stage of development, whether you have simple wireframes, very detailed working mockups, or actual products, so long as you effectively set expectations at the outset of a session.
One of Betafi’s early customers Rishabh, CEO of Data Narratives, shared that when he started out he was extremely data-driven, but he soon ran into the limits of quantitative data for making core product decisions. Post-launch, as people inevitably ran up against rough edges with the early, scrappy MVP, he first relied on the performance metrics from analytics software. This information indicated that something was wrong with some core product flows, but it did not tell him the root cause of the problems early users encountered.
That’s when he switched gears and started conducting usability tests on the Betafi platform. Speaking with users made him realize that parts of the user experience were unintuitive and small UX issues compounded throughout the tool, making it unnecessarily difficult to accomplish key tasks in the product.
With the help of such usability feedback, they launched a new version. The results this time were super impressive — their average session duration was up 12x, and they got 8 large enterprise customers across 3 countries since then.
The turnaround in terms of traction helped Rishabh realize the true value of qualitative research even if it meant adding a little more time up-front to the product development process. As a result, usability testing is now a standard practice at Data Narratives. So, now once the wireframes are ready, they speak to their users, gather usability feedback, and share it back with their team. And this feedback loop ensures that they continue to delight their users with every incremental launch.
There are certainly merits of doing A/B testing, as you can quickly identify what’s working or not with a significantly large user base. But usability testing can uncover impactful improvements that can make or break your product. Without usability, A/B testing can be a risky, reactive approach as, by the time you try to retract your steps, you may have already lost the game.
We would love to hear your perspective on which methods you use to ensure you build fast but build the right product for your users. Write back to us at hello@betafi.co to share your take.
