Sachin Rekhi

A decade ago Steve Blank authored the book The Four Steps to the Epiphany: Successful Strategies for Products that Win to share with the world his philosophy for building winning products: customer development. He espoused that the reason so many startups failed was they were too focused on product development without an equal focus on customer development. He suggested we all needed to "get out of the building" and speak directly with our potential customers to truly understand the problems we are solving for them. Eric Ries further popularized customer development techniques with his Lean Startup methodology, which has customer development as one of it's key components alongside minimal viable products, validated learning, and more.

In the same decade we've see a significant reduction in the cost of building a technology company. With the advent of Amazon Web Services launching in 2006 and so many other incredible developer-oriented services now available, the cost of bringing a product to market has fallen by orders of magnitude, enabling far more people to build technology startups than ever before.

And yet despite all of this innovation in the startup process, we haven't seen a substantial increase in successful large-scale startup outcomes. So why is that? I wanted to share my two theories for why this is the case. The first is that the specific customer development tactics and best practices have still not matured enough to be predictably applied in startups. The second is that despite the improved validation process, it hasn't improve it's required input: high quality idea generation.

Customer Development Maturity
One way to think about the maturity level of customer development is to compare it to another well-trodden startup technique for validated learning: A/B testing. A/B testing has been around longer, reaching incredible scale back in 2009 when Marissa Mayer at Google famously tested 41 shades of blue on the Google results page, with the winning variant netting $200M in increased annual revenue. The methodology has matured significantly, with strong implementation details specifying minimum variant population sizes, confidence levels on results, population selection techniques, minimum time-span calculators, and so much more. Techniques have gotten incredibly sophisticated in touching on topics such as improving A/B test velocity through orthogonal population testing. And the biases that one could expect from A/B testing techniques and it's limitations are well understood. If you wanted to implement an A/B test at your new startup, you could quickly implement an existing tool that completely automates the process like Google Website Optimizer or Optimizely or leverage the myriad articles online that specify in grave detail how to implement an accurate A/B test with minimum sample sizes, sample period, etc. That's a taste for how mature A/B testing is as a methodology.

When you compare that to customer development, you realize that the established best practices are very high level and leave a lot for each startup to sort through. While Steve Blank talked about "getting out of the building" to talking to customers, I've seen various success-levels with startups leveraging customer validation interviews and gleaning valuable insights from them. With such few data points, the way you ask the questions and synthesize the information is incredibly important for gaining valuable signals. It's important to construct your validation questions in such a way to get relative ranking of importance of problems, potential solutions, and more, otherwise you leave it too easy for the interviewee to say everything is important. Techniques like how would you spend $100 are valuable here. But the main point is everyone is implementing their validation interviews differently with no tried-and-true best practices yet that are broadly accepted. I shared my own techniques for doing so here. And customer validation interviews are just one aspect of the customer development process.

Until we see the established best practices for customer development mature similar to A/B testing, startups won't be able to reliably benefit from it.

High Quality Idea Generation
One key to remember for both A/B testing and customer development is that they are validation techniques that help you to assess whether various variants, concepts, ideas are going to resonate with your customer and help you achieve your goals. They are both effectively hypothesis testing techniques to either validate or invalidate your existing hypotheses, just applied at different stages of your business.

What this means is that the limitations we've ultimately realized exist for A/B testing all still apply to customer development techniques as well. A/B testing relies on coming up with a myriad of hypotheses to test and the process simply facilitates assessment of those various hypotheses. It doesn't help in the generation of hypotheses in the first place. As this has become well-established, folks have realized that often A/B testing alone can lead to a local maxima instead of a global maxima. And while you can improve your A/B testing velocity, you'll never be able to test every hypothesis out there, so coming up with high quality hypotheses in the first place remains important. Essentially the prioritized list of ideas you enter into A/B testing matters a lot for the ultimate outcome. The same is true for customer development. The quality of the ideas that you decide to validate with customer validation techniques are ultimately the limiting reagent. And the techniques themselves don't offer much improvement in the generation of the ideas in the first place.

So how do you come up with high quality ideas? No new techniques here. Being deeply immersed in a problem space for years and coming up with core insights on how to solve them is certainly a tried-and-true technique. Yet many others find their ideas in other ways. But we haven't seen a meaningful improvement in generation of high quality ideas, which remains a necessarily input to customer development and ultimately to a successful startup.

I have no doubt over time we'll improve customer development techniques to bring their maturity level up to other well-established techniques like A/B testing. I've written extensively about some of my best practices in this area. But it'll never be a panacea for ensuring the ultimate success of every startup.