Experimenting towards success

It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong.

Richard P. Feynman

We have a clear vision of what we want to build; a service that makes the lives of millions of customers simpler and fairer. The fastest way to build it would be to hire a big team, break down the thing in our heads into component parts and get going building it as fast as we can.

For some businesses that's exactly the right thing to do. Let’s use Cazoo (who sell used cars online) as an example. Carvana in the US had proved the 'Amazon for cars' model and IPO'd back in 2017. So when Alex Chesterman started Cazoo here in the UK, he had a real example of a model that had been proven to work in the US. His job was to focus on how to adjust and improve the model for the UK market and build and scale it as fast as possible– before someone else tried to. 

Another example would be if you’re starting something where you really really want the product, and you are confident there are lots of people just like you. Then you can just build the product that you want and hopefully all the other people who are just like you will also want it. You often see this in B2B businesses where someone leaves their job to build a product that will fix some part of their job that was unnecessarily difficult.

(Of course, this sometimes goes horribly wrong when an entrepreneur wildly overestimates the number of people who share their specific needs)

Nous is different - we're pioneering a new category of service; no-one has done this particular thing before. And we're not going to just build the product that specifically we want as our ambitions are much bigger than a target market of slightly mad very geeky tech people working for a startup. Therefore we need to focus on learning what our target customers really want, how we can deliver value to them and how to effectively communicate that value to get them to use our service.

We do this by running experiments. Big experiments, little experiments; all designed to help us get to Product Market Fit as quickly as possible.

The primary focus on experiments is a phase, not how we will operate forever (although we will continue to have a disciplined approach to test & learn). We started our experimental phase by documenting all of our assumptions; what are all the things that have to be true for this business to be successful? We grouped these into desirability (do customers want it), feasibility (can we do it) and viability (should we do it, or can we do it profitably at scale). After putting them on a big two-by-two grid of importance and confidence we prioritised which are the critical assumptions we want to focus on validating first.

To validate an assumption we write up a hypothesis and design an experiment to prove or disprove the hypothesis. A experiment could take many forms; techniques we've used so far include testing landing pages and marketing funnels, running surveys, measuring conversion rates of iterations of a product, testing clickable prototypes, doing tasks on behalf of our founding members and running focus groups. The critical bit is clearly articulating what you're going to measure and what the threshold to prove the hypothesis correct is BEFORE you start running it.

As much as we might wish to reduce building a great business to a disciplined process, the reality is that specifying hypotheses well, designing ingenious experiments and navigating and sequencing them is far from procedural. It involves a fair bit of judgement and a fair degree of good luck. That said, it’s great to have a method to discover truth about the world which doesn't depend on who shouts the loudest.

At the early stages, more of the experiments were small ones which sometimes could be designed, built and run in a week or two. More recently, we've been focussing on testing some bigger, deeper hypotheses which require several sprints work of development to build what's needed to enable the experiment to run. Most of the pieces we're building for experiments are core parts of our product, so we won’t have to throw too much away. However we're not afraid of building something quick and dirty to enable an experiment and then throwing it away if that’s the most efficient way to learn.

One important thing we've learnt is not to confuse the success of the experiment with the success (or truth) of the hypothesis. Some of our most valuable experiments are ones where we conclusively disproved a hypothesis. Quickly learning that your intuitions were misguided early in a business is perhaps even more important than knowing they were right.

It's not just the way we work that's optimised for learning, it's also the team we're building. Think of it like a trip to explore unknown territory. You want a small group that works exceptionally well together. We're not sure what exact challenges the group will find - so everyone needs to be world class in their speciality but also adaptable and able to pick up whatever needs doing. And perhaps most importantly a group like this can quickly absorb new information, try and understand what it means and adapt it's plans to take advantage of any new insights.

Will we be in such a focussed experimental phase forever? No. Right now we're in a phase of iterating quickly towards Product Market Fit. It's much more important to learn quickly than build quickly. As we home in on Product Market Fit the balance will change. Speed of growth and execution will rise up in importance. Being able to scale marketing, engineering and operations will become important challenges. But that is all for tomorrow. Today our priority is getting to Product Market fit (and getting some tantalising signals we might be getting close). And the way we're doing it is through experimentation.