Spend a Little, Save a Lot | The Compelling Economics of Research-Driven Design

Imagine this. You’re a product manager whose new app launched a month ago. The results are in, and they’re mostly bad. Low rates of adoption, a lot of negative feedback about the user experience, and a few missing features. Two hours from now you and your boss will meet with your VP of Product to request additional budget to fix these issues. Will you get it, or will it go to another team with a better track record? Does your VP still have confidence in you?

This won’t be a fun conversation. But what if I said you could have greatly increased the odds of your success, with little impact on the timeline or budget? Sound too good to be true? Read on.

Validation will happen

Users will validate, or evaluate, your product. We all do it. We navigate to an app or download one from the App Store, and we check it out. For the product, this is the moment of truth. Do people find it useful? Easy to figure out? Does it have the features they need, without being loaded up with features they don’t want?

This decision point is inescapable. But you as a product manager have a great deal of control over when and where it happens, and how to react to it.

Two approaches

One approach to product validation is to wait to get feedback until after the product launches. We’ll call this the high-risk approach. It saves you some money up front — although probably not as much as you think — but at a high price in terms of risk, as you won’t know if you hit the mark until you launch. And at that point the damage, and the cost to repair that damage, will be very high.

Approach number two is to practice research-driven design. Let’s talk about what this consists of and what it gets you:

Typically it comprises three rounds of research: 1) interviews; 2) design concepts; 3) mid-fidelity designs.

Each round involves (only) 6-8 people, i.e., you don’t have to interview hundreds of people. You start out by asking a lot of questions about the tasks people want to accomplish and look for patterns, priorities, and pain points.

You then use these learnings to generate design concepts. Speed, not beauty, is the word here. Pen and paper sketches are my first choice or you can use very simple digital designs. Your goal is to quickly validate the overall design direction and make sure you’re focusing on the right feature set.

Round 3 takes things up a notch in terms of the visual presentation, for example, color, typography, and fine tuning the layout, to get the design closer to what the final product will look like. You might make the prototype clickable so users can evaluate the navigation as well.

What would that get you?

At the end of this, you as a product manager will have 18+ hours of video content of real users telling you what they are looking for in this product and reacting to the initial designs, with the findings distilled and prioritized by your researchers. Not only have you significantly upped your odds of success, you’ve done it in the cheapest way possible, because your development team hasn’t written a single line of code yet.

How much time does it take?

Not much. At 3Pillar we typically complete three rounds of research and report out the findings within six to eight weeks. In the case of an iPhone app for a global apparel brand, where the entire engagement lasted only thirteen weeks from kickoff to launch, we condensed that timeline further. But even though the app really only did five things, our research uncovered the five most important features to build and the app was an immediate success.

Other research efforts might go longer, depending on the complexity of the product and the length of the engagement. On the other hand, if reducing the timeline is important, adding an additional researcher will make a significant impact on the team’s velocity.

How big a difference can the research make in terms of risk and cost?

In 2017 we conducted research for an integrated web/smartphone app that had to deal with a significant amount of content variability across regions. Going into round three, we felt like we hadn’t nailed the content structure. We had all the right pieces and parts but they weren’t in the right place yet.

The first person we interviewed (in round three) told us how he processed the information, in “stacks” as he phrased it, and it was truly a lightbulb moment for us. We made the call to immediately revise our designs based on his feedback, and tested these over the next two days, with excellent results. We had our answer.

Now what would have happened if this lightbulb moment had never occurred? It’s highly likely the earlier content structure would have fared poorly, as it separated things that (as we learned) users wanted to see together. This was pretty fundamental to the app, so ripping out that code and rewriting it would probably have taken two sprints (four weeks), if not more.

Let’s compare these scenarios:

Research-driven design

Cost for two designers to redesign 10 pages within 24 hours: $2,000

No-research approach

Cost for the development team (say 5 developers) to rewrite major parts of the code base over 4 weeks: $50,000

Compelling economics

As you can see, the difference in cost is sky high, since the research-driven scenario doesn’t involve rewriting any code. It’s also worth mentioning that these figures don’t include the reputational damage from a failed product launch. For these reasons, our approach to product development is not “Can we afford to do some upfront user research,” but rather, “Can we afford not to?”

About the author:

Michael Rabjohns is 3Pillar’s UX Practice Leader in the US. He has twenty years’ experience in the digital product space, and has worked as a User Researcher, Interaction Designer, Content Strategist, and Product Manager. He holds MBA and MSFS degrees from Georgetown University.