The Discovery Sandwich: Data-Driven Product Discovery Research
Here are two consensus statements:
1. The best products are built by startups that really understand the people they’re selling to.
2. The best teams use data to make informed rational decisions.
Though you likely agree with both, these two statements are often in direct conflict. Product teams are constantly forced to decide between what they know from talking to their customers and what their data tells them. And there’s no correct answer or objective truth that shows which side to trust — you’ve just got to put your chips on either red or black and hope for the best.
Over the past 7 years, I’ve worked alongside hundreds of product builders (my own startup included) who have faced this conundrum when deciding what to build. What I’ve seen is that the best products are informed by qualitative insights, but at the same time the best teams ALSO tend to make big decisions based on quantitative data.
We need a way to bridge this gap between qualitative insights and quantitative data so that both can be leveraged to inform the same decision. That’s where the Discovery Sandwich comes in 🥪
Allow me to explain…
How I f*cked up my customer discovery the first time around
Back in October 2020, I launched a startup to build a new kind of online research tool. We had a clever solution that was like a normal survey but could adapt and change as more participants joined as if it was a live conversation.
I spent a year raising investment, building a team, and launching our first prototype. But our startup was a solution in search of a problem. In the first 6 months, we only got one paying customer and they churned within a few weeks.
Rather than giving up, we rolled up our sleeves and found 100+ people to interview to figure out who nobody cared. We developed a very broad understanding of what research means to every type of team — from government policy teams and research agencies to HR functions and product teams in companies of every size.
But no amount of tweaks and iterations seemed to work. By this stage, we were so deep in insights that we didn’t even know what we were building anymore. We had become lost in the details.
As a last-ditch effort in March 2021, we decided to run one final experiment.
We wrote a list of the 45 most common problems that we had heard during all these interviews. Then one night we messaged hundreds of UX Researchers online and asked them to rank all these problems for us. Within 2 hours, we realized why nobody cared — the problems we thought were most important to our target customers were ranking at the VERY BOTTOM of this stack ranked list.
We repeated the same experiment again the next day with Product Managers and got the exact same result.
It might seem like this experiment would be the straw to break the camel’s back. Instead, it was the final ingredient we needed to make our startup work. Yes, stack ranking these statements showed us that we were chasing the wrong problem. But at the same time, it also showed us which problems our target customers cared the MOST about solving — and we had the hard data to prove it.
So rather than continuing to interview such a broad mix of different people and asking about all their problems, we could focus solely on Product Managers and learn everything we could about their most important problems. That’s when things finally started to work.
People started paying attention to us. Some even started paying us too. Over the following 12 months, we rebuilt our entire product from the ground up. Today, thousands of teams from some of the most innovative companies in the world are using our product every single week.
This experience fundamentally changed how I think about building products.
I no longer believe that user interviews and qualitative research alone can lead most startups to success.
I no longer believe that quantitative data and product analytics alone can show teams the right thing to prioritize.
Instead, you need both.
By combining qualitative insights with quantitative data, product teams can navigate their way towards a deeper understanding of the right problem to solve for the right customer. In other words, the most important part of product discovery is its layers.
The Discovery Sandwich: The Mixed Methods Framework for Product and Customer Discovery Research
The Discovery Sandwich is a really simple three-step framework — first learn as much as possible about your customer’s problems, then use data-driven research to figure out which of these problems matter most, and then go deep into learning all the context surrounding that problem.
Here’s how I tackle those three layers and the tools I use to get the best possible insights at each stage of the process.
Layer 1: Qualitative (The Bottom Slice of Bread)
When we started interviewing 100+ people in late 2020, we thought what we were doing was the entire customer discovery process — that you discovered their biggest unmet needs, pains and desires by just sitting people down and asking them questions like in the book ’The Mom Test’.
What we didn’t realize is that this is just the first step of discovery — breadth interviews. The first step is to understand our target customers and the various problems they face. What objectives are they trying to accomplish? What do the processes look like? Who else is involved? What solutions are they currently using? What are their most common and highest impact frustrations? Like a good slice of bread, your job is to soak all of this up.
This is why The Mom Test principles are so important — you’re not actually trying to validate anything at this stage. You simply want to learn what your interviewee’s life looks like with an open mind.
This is where we went wrong the first time around. We treated these interviews like an exam to pass/fail. If our interviewees didn’t say the right keywords or didn’t complain about our shortlisted problems, we saw it as failed validation. So we would desperately steer the conversations toward our topics of interest to try and get them to say what we wanted to hear. That’s how we ended up chasing their 45th most painful problem...
Layer 2: Quantitative (The Fillings)
Once you have a good understanding of your customer’s various unmet needs, pains and desires, you want to use data to figure out which they find most impactful.
Most impactful can mean most frequent (happens 5x every day), most frustrating (creates 3 hours of extra work per week), most costly (loses 10% of the value of every customer deal), or ideally some combination of multiple factors. If we solve a problem that customers feel is very impactful, then it’s likely that those customers are already actively searching for a solution. This is the ideal scenario for building a product.
This would all be very easy if each problem already had a data point for cost or frequency, but things are very rarely that easy. So we usually have to rely on other methods to turn these qualitative insights into real quantitative data.
One way we can do that is with a research method called Customer Problem Stack Ranking (CPSR). First coined by a Stripe product leader called Shreyas Doshi in May 2020, CPSR asks our customers to compare and stack rank a set of problems to measure which are most impactful. If you’re looking to learn more about CPSR, try The Beginner’s Guide to Stack Ranking Research.
One thing to note: every product should be built to solve an impactful problem for a specific segment of customers. WeatherBill’s $930M Pivot is the perfect example of why picking one target customer segment is the most important first ingredient for building a scalable startup.
If you’ve already completed many breadth interviews while juggling multiple different customer segments, you can use Needs-Based Segmentation to run one stack ranking experiment and then split the results into each segment to see which problems each subgroup cares most about.
OpinionX is a free research tool built specifically for stack ranking projects and it also comes with needs-based segmentation already built in. You can take your list of problem statements and have your first stack ranking survey ready to go within 5 minutes.
By the end of this step, regardless of the tool you choose to use, you should have real quantitative data that shows you what the highest impact problem is for a specific customer segment.
Layer 3: Back to Qualitative (The Top Slice of Bread)
The quant layer of The Discovery Sandwich gives us the validation we need to prioritize one key problem. This is not necessarily validation in the traditional startup sense of ‘proving that your idea has potential’ — instead, this validation gives us permission to ask customers directly about the problem without fear of biasing interviews.
Here is where we bring back The Mom Test’s principles once again like not pitching, disregarding compliments, and digging for specifics. But this time, we don’t need to dance around the problem. We want to focus very specifically on it instead. We’re doing in-depth interviews now rather than breadth interviews.
If you use a purpose-built stack ranking tool like OpinionX for step 2, you can spotlight which participants personally ranked your key problem in their top 3 and then ask them to do follow-up interviews. This cuts out a huge amount of guesswork. Not only do you have the data to prove that ‘Problem X’ is of high importance to your target customer, you even have a handful of identifiable people that you know are personally struggling to solve that exact problem.
Top Takeaway → The order of layers matters as much as the layers themselves
^ shitty sandwich
If you make a sandwich with two slices of ham on the outside and one layer of bread in the middle, that’s a pretty terrible sandwich. If you try to validate the problem your product solves before you know anything about your customers’ lives and the various problems they deal with, you’re probably going to make a shitty product. Choose the order of steps in your discovery process carefully.
The Discovery Sandwich is a lightweight framework for identifying the set of possible options, prioritizing which to focus on, and gathering rich contextual insights to inform the right decision. It’s easy to implement independent of the method you choose like Customer Problem Stack Ranking. It can be used for processes outside of Product Discovery such as customer-led roadmap prioritization. And it can be incorporated into existing frameworks like Opportunity Solution Trees for Continuous Discovery.
If you’ve already finished the breadth interviews in step one of the Discovery Sandwich and have a bunch of insights gleaned from your interviews, jump straight into the quant layer with OpinionX — the free research tool for stack ranking people’s priorities. You can create your first Customer Problem Stack Ranking survey in minutes (with no credit card required) at app.opinionx.co.