独家优惠奖金 100% 高达 1 BTC + 180 免费旋转

What Type of Lean Startup Experiment Should I Run?

Add this to lean startup standards like smoke tests and the list gets pretty overwhelming.

Should we run a Pocket Test with Picnic in the Graveyard to follow up? Should we do a Wizard of Oz or a concierge approach? Would you like a lemon twist with that?

So what type of experiment should you run? And when?

If you’d like to cut to the chase, you can download the list and index here:

Many, many people have weighed in on which is superior.

As Laura points out in her post, it’s not a question of better. A hammer is not inherently better than a screwdriver. A hammer is better than a screwdriver for hammering in nails.

Janice Fraser introduced me to the distinction of Generative vs. Evaluative:

This distinction explains why we often get crappy results from experiments. We might run a smoke test with the hypothesis:

We advertise via twitter and come back with a paltry 1% conversion rate to email sign up. Good idea? Bad idea? We confirmed our hypothesis… “some people” did indeed sign up!

But is our conversion rate low because no one is interested? Or because we advertised via the wrong channel? Does no one want our value proposition? Or does no one understand it?

There are a hundred reasons why we might get a false negative result from this test. There are also quite a few reasons why we might get a false positive!

In this case our hypothesis is incredibly vague AND flawed.

Who are these people? People on twitter? Who are they following? Are they eco-friendly dads who bike to work? Or are they professional runners who care more about durability than being environmentally friendly?

When our hypothesis is specific and falsifiable, we can run an evaluative experiment such as a smoke test.

When our hypothesis is vague or we don’t even have a hypothesis, we need to do generative research such as getting out of the building and talking to potential customers to get new ideas or refine our hypothesis.

Some methods tell us a lot about customers, their problems, and how to reach them. For example, we can listen to our customers to help us understand their situation and what their day-to-day problems are.

Other methods tell us about the product or solution that will help solve that problem. We can do usability testing on a set of wireframes and see if our interface is usable. However, this won’t tell us anything about whether or not anyone will buy it in the first place.

These methods generally don’t overlap.

If we combine the useful distinctions of Generative Research vs. Evaluative Experiments and Market vs. Product we have four nice boxes which we can use to help us determine what we should do next:

Each of these boxes helps us answer different questions.

We could talk to customers and see what’s bothering them (Steve’s advice and always a good idea) or we might try data mining if we happen to have access to a large set of data. We could even do a broad survey with open ended questions.*

Some of these research methods are qualitative (e.g. talking to customers) and some are quantitative (e.g. data mining). That distinction is not important.

Here are some of Generative Market Research methods:

* Don’t do this. Surveys and focus groups generally suck.

Here are a few Evaluative Market Experiments if we have a clear, falsifiable hypothesis:

- Video

- Landing page

- Sales pitch

- Pre-sales

- Flyers

- Pocket test

- Event

- Fake door

- High bar

* Again, don’t use these. They generally suck.

Before we move on from here, we must remember: we’re probably wrong. Even if we have tens of thousands of users signed up to our landing page, that doesn’t mean we have a validated problem.

If the customer didn’t have to commit to anything aside from their email address or they misunderstood the value proposition, then those signups don’t signify true customer demand. It just means we make awesome landing pages.

Remember:

Once we’ve validated the market and value proposition sufficiently, we need to understand what the solution would look like.

If we truly have a validated Customer with a clear Problem and a Value Proposition, then we can start asking how.

Unfortunately, while our market hypotheses tend to be overly vague, our solution hypotheses tend to be overly specific and way too comprehensive.

To simplify our solution and help us prioritize which features to build first, we can use methods like concierge testing or solution interviews to help us generate ideas about what our MVP should be.

Here are some methods for Generative Product Research:

Fortunately, there is a list of well defined tools that have been around for decades to figure that out.

We can do user testing to look for usability issues that might prevent the solution from working. We can A/B test two alternatives to see what works better. We can use a Net Promoter Score survey (one of the few surveys that we like) to see overall satisfaction. All of these Evaluative Product Experiments tell us if our solution is doing the trick.

- Net Promoter Score

- Product/Market Fit Survey

A lot of research/experiments will blur the lines. It’s rare that we’ll do generative research without having a hypothesis in the back of our mind about who our customer is. We may inadvertently evaluate (and invalidate) that hypothesis. That’s ok.

This is just a framework to get ourselves headed in the right direction and make it more likely that we use the right tool for the job.

Choose wisely.

You can download the Real Startup Book for quick reference here:

This book isn’t finished and never will be.

Lots of people have suggested the methods listed and maybe you can suggest one that isn’t there yet. We’ll list all contributions and the whole thing is creative commons licensed.

Add a comment

What is your view?

Send

Related posts:

How to do great work?

Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do.

Rude Sounding Words

Here are some words that sound rude but they are really not. But once you start hearing the rude side to these words, a strange thing might happen to you, and many, many words of our language will…

Should teenagers drink Energy Drinks?

An issue that is hotly debated, is the consumption of Energy drinks. This essay clarifies why teenagers should not drink Energy drinks. Energy drinks are popular among young adults and teenagers…