When NOT to listen to your users; when NOT to rely on split-tests

There are three legs to the lean startup concept: agile product development, low-cost (fast to market) platforms, and rapid-iteration customer development. When I have the opportunity to meet startups, they usually have one of these aspects down, and need help with one or two of the others. The most common need is becoming more customer-centric. They need to incorporate customer feedback into the product development and business planning process. I usually recommend two things: try to get the whole team to start talking to customers ("just go meet a few") and get them to use split-testing in their feature release process ("try it, you'll like it").

However, that can't be the end of the story. If all we do is mechanically embrace these tactics, we can wind up with a disaster. Here are two specific ways it can go horribly wrong. Both are related to a common brain defect we engineers and entrepreneurs seem to be especially prone to. I call it "if some is good, more is better" and it can cause us to swing wildly from one extreme of belief to another.

What's needed is a disciplined methodology for understanding the needs of customers and how they combine to form a viable business model. In this post, I'll discuss two particular examples, but for a full treatment, I recommend Steve Blank's The Four Steps to the Epiphany.

---

Let's start with the "do whatever customers say, no matter what" problem. I'll borrow this example from randomwalker's journal - Lessons from the failure of Livejournal: when NOT to listen to your users.

The opportunity was just mind-bogglingly huge. But none of that happened. The site hung on to its design philosophy of being an island cut off from the rest of the Web, and paid the price. ... The site is now a sad footnote in the history of Social Networking Services. How did they do it? By listening to their users.

randomwalker identifies four specific ways in which LJ's listening caused them problems, and they are all variations on a theme: listening to the wrong users. The early adopters of LiveJournal didn't want to see the site become mainstream, and the team didn't find a way to stand up for their business or vision.

I remember having this problem when I first got the "listening to customers" religion. I felt we should just talk to as many customers as possible, and do whatever they say. But that is a bad idea. It confuses the tactic, which is listening, with the strategy, which is learning. Talking to customers is important because it helps us deal in facts about the world as it is today. If we're going to build a product, we need to have a sense of who will use it. If we're going to change a features, we need to know how our existing customers will react. If we're working on positioning for our product, we need to know what is in the mind of our prospects today.

If your team is struggling with customer feedback, you may find this mantra helpful. Seek out a synthesis that incorporates both the feedback you are hearing plus your own vision. Any path that leaves out one aspect or the other is probably wrong. Have faith that this synthesis is greater than the sum of its parts. If you can't find a synthesis position that works for your customers and for your business, it either means you're not trying hard enough or your business is in trouble. Figure out which one it is, have a heart-to-heart with your team, and make some serious changes.

---

Especially for us introverted engineering types, there is one major drawback to talking to customers: it's messy. Customers are living breathing complex people, with their own drama and issues. When they talk to you, it can be overwhelming to sort through all that irrelevant data to capture the nuggets of wisdom that are key to learning. In a perfect world, we'd all have the courage and stamina to perservere, and implement a complete Ideas-Code-Data rapid learning loop. But in reality, we sometimes fall back on inadequate shortcuts. One of those is an over-emphasis on split-testing.

Split-testing provides objective facts about our product and customers, and this has strong appeal to the science-oriented among us. But the thing to remember about split-testing is that it is always retrospective - it can only give you facts about the past. Split-testing is completely useless in telling you what to do next. Now, to make good decisions, it's helpful to have historical data about what has and hasn't worked in the past. If you take it too far, though, you can lose the creative spark that is also key to learning.

For example, I have often fallen into the trap of wanting to optimize the heck out of one single variable in our business. One time, I became completely enamored with Influence: The Psychology of Persuasion (which is a great book, but that's for another post). I managed to convince myself that the solution to all of our company's problems were contained in that book, and that if we just faithfully executed a marketing campaign around the principles therein, we'd solve everything. I convinced a team to give this a try, and they did tried dozens of split-test experiments, each around a different principle or combination of principles. We tried and tried to boost our conversion numbers, each time analyzing what worked and what didn't, and iterating. We were excited by each new discovery, and each iteration we managed to move the conversion needle a little bit more. Here was the problem: the total impact we were having was miniscule. It turns out that we were not really addressing the core problem (which had nothing to do with persuasion). So although we felt we were making progress, and even though we were moving numbers on a spreadsheet, it was all for nothing. Only when someone hit me over the head and said "this isn't working, let's try a radically new direction" did I realize what had happened. We'd forgotten to use the all the tools in our toolbox, and lost sight of our overarching goal.

It's important to be open to hearing new ideas, especially when the ideas you're working on are split-testing poorly. That's not to say you should give up right away, but always take a moment to step back and ask yourself if your current path is making progress. It might be time to reshuffle the deck and try again.

Just don't forget to subject the radical new idea to split-testing too. It might be even worse than what you're doing right now.

---

So, both split-testing and customer feedback have their drawbacks. What can you do about it? There are a few ideas I have found generally helpful:

• Identify where the "learning block" is. For example, think of the phases of the synthesis framework: collecting feedback, processing and understanding it, choosing a new course of action. If you're not getting the results you want, probably it's because one of those phases is blocked. For example, I've had the opportunity to work with a brilliant product person who had an incredible talent at rationalization. Once he got the "customer feedback" religion, I noticed this pattern: "Guys! I've just conducted three customer focus groups, and, incredibly, the customers really want us to build the feature I've been telling you about for a month." No matter what the input, he'd come around to the same conclusion as before.
  
  Or maybe you have someone on your team that's just not processing: "Customers say they want X, so that's what we're building." Each new customer that walks in the door wants a different X, so we keep changing direction.
  
  Or consider my favorite of all: the "we have no choice but to stay the course" pessimist. For this person, there's always some reason why what we're learning about customers can't help. We're doomed! For example, we simply cannot make the changes we need because we've already promised something to partners. Or the press. Or to some passionate customers. Or to our team. Whoever it is, we just can't go back on our promise, it'd be too painful. So we have to roll the dice with what we're working on now, even if we all agree it's not our best shot at success.
  
  Wherever the blockage is happening, by identifying it you can work on fixing it.
  
  
• Focus on "minimum feature set" whenever processing feedback. It's all too easy to put together a spec that contains every feature that every customer has ever asked for. That's not a challenge. The hard part is to figure out the fewest possible features that could possibly accomplish your company's goals. If you ever have the opportunity to remove a feature without impacting the customer experience or business metrics - do it. If you need help determining what features are truly essential, pay special attention to the Customer Validation phase of Customer Development.
  
  
• Consider whether the company is experiencing a phase-change that might make what's made you successful in the past obsolete. The most famous of these phase-change theories is Crossing the Chasm, which gives very clear guidance about what to do in a situation where you can't seem to make any more progress with the early-adopter customers you have. That's a good time to change course. One possibility: try segmenting your customers into a few archetypes, and see if any of those sounds more promising than another. Even if one archetype currently dominates your customer base, would it be more promising to pursue a different one?
  

As much as we try to incorporate scientific product development into our work, the fact remains that business is not a science. I think Drucker said it best. It's pretty easy to deliver results in the short term or the long term. It's pretty easy to optimize our business to serve one of employees, customers or shareholders. But it's incredibly hard to balance the needs of all three stakeholders over both the short and long-term time horizon. That's what business is designed to do. By learning to find a synthesis between our customers and our vision, we can make a meaningful contribution to that goal.

Validated learning about customers

Would you rather have $30,000 or $1 million in revenues for your startup? Sounds like a no-brainer, but I’d like to try and convince you that it’s not. All things being equal, of course, you’d rather have more revenue rather than less. But all things are never equal. In an early-stage startup especially, revenue is not an important goal in and of itself.

This may sound crazy, coming as it does from an advocate of charging customers for your product from day one. I have counseled innumerable entrepreneurs to change their focus to revenue, and many companies who refuse this advice get themselves into trouble by running out of iterations. And yet revenue alone is not a sufficient goal. Focusing on it exclusively can lead to failure as surely as ignoring it altogether.

Let’s start with a simple question: why do early-stage startups want revenue? We all know why big companies want revenue – it’s one of two critical halves of the formula for profit. And big companies exist to maximize profit. Don’t startups exist for the same reason? I think such reasoning is an example of the “startup dollhouse fallacy” – that startups are just shrunken-down big companies. In fact, I don’t think revenue is in and of itself a goal for startups, and neither is profit. What matters is proving the viability of the company’s business model, what investors call “traction.” Demonstrating traction is the true purpose of revenue in an early growth company. (Of course this is not at all true of many profitable small businesses, but they are not what I mean by startups.) Before I explain what I mean, let me add an important caveat: traction is not just important for investors. It should be even more important to the founders themselves, because it demonstrates that their business hypothesis is grounded in reality. More on that in a moment.

Consider this company (as always, a fictionalized composite): they have a million dollars of revenue, and are showing growth quarter after quarter. And yet, their investors are frustrated. Every board meeting, the metrics of success change. Their product definition fluctuates wildly – one month, it’s a dessert topping, the next it’s a floor wax. Their product development team is hard at work on a next-generation product platform, which is designed to offer a new suite of products – but this effort is months behind schedule. In fact, this company hasn’t shipped any new products in months. And yet their numbers continue to grow, month after month. What’s going on?

In my consulting practice, I sometimes have the opportunity to work with companies like this. Diagnosis is easy: they are exceptionally gifted salesmen. This is an incredible skill, one that most engineers overlook. True salesmen are artists, able to hone in on just those key words, phrases, features, and benefits that will convince another human being to give up their hard-earned money in exchange for even an early product. For a startup, having great sales DNA is a wonderful asset. But in this kind of situation, it can devour the company’s future.

The problem stems from selling each customer a custom one-time product. This is the magic of sales: by learning about each customer in-depth, they can convince each of them that this product would solve serious problems. That leads to cashing many checks. Now, in some situations, this over-selling would lead to a secondary problem, namely, that customers would realize they had been duped and refuse to re-subscribe. But here’s where a truly great sales artist comes in. Customers don’t usually mind a bait-and-switch if the switched-to product really does solve an important problem for them. These salesmen used their insight into what their customers really needed to make the sale and then deliver something of even greater value. They are closing orders. They are gaining valuable customer data. They are close to breakeven. What’s the problem?

This approach is fundamentally non-scalable. These founders have not managed, to borrow a phrase from Steve Blank, to create a scalable and repeatable sales process. Every sale requires handholding and personal attention from the founders themselves. This process cannot be delegated, because it’s impossible to explain to a normal person what’s involved in making the sale. The founders have a lethal combination of insight about what potential customers want and in-depth knowledge about what their current product can really deliver. As a result, potential customers are being turned away; they can only afford to engage with the customers that are best qualified.

And what of the product development team? They are busy too, but they are not creating value for the company. They are trying to build a product to an ever-changing spec, based on intuitions from the founders about what might be able to sell itself. Worse, the founders are never around – they are too busy going out and selling! Without access to customer data, or even a clear product owner, the product development team keeps building feature after feature based on what they think might be useful. But since nobody in the company can clearly articulate what the product is, their efforts result in incoherence. Worst of all, their next-generation product is so bad they are not allowed to try it out on any customers. The team is thus completely starved of any form of external feedback.

Let me describe a different company, one with only $30,000 in revenue (again, pure fiction). This company has a large long-term vision, but their current product is only a fraction of what they hope to build. Compared to the million-dollar startup, they are operating at micro-scale. How does that stack up?

First of all, they are not selling their product by hand. Instead, each potential customer has to go through a self-serve process of signing up and paying money. Because they have no presence in the market, they have to find distribution channels to bring in customers. They can only afford those (like Google AdWords) that support buying in small volume.

Compensating for these limitations is the fact that they know each of their customers extremely well, and they are constantly experimenting with new product features and product marketing to increase the yield on each new crop of customers they bring in. Over time, they have found a formula for acquiring, qualifying, and selling customers in the market segments they have targeted. Most importantly, they have lots of data about the unit economics of their business. They know how much it costs to bring in a customer and they know how much money they can expect to make on each one.

In other words, they have learned to grow renewable audiences. Given the data they’ve collected about these early customers, they are also able to estimate with modest precision how big the market is for their product in its current form. They may be at micro-scale now, but they are in a very good position to raise venture money and engage in extremely rapid growth.

Our million-dollar startup, by contrast, is stuck in the mud.

Stories like these are what has led me to this definition of progress for a startup: validated learning about customers. (Steve calls this just Customer Validation, but I like to emphasize the learning aspect, so I accept a far more awkward phrase.)

This unit of progress is remarkable in several ways. First of all, it means that most aggregate measures of success, like total revenue, are not very useful. They don’t tell us the key things we need to know about the business: how profitable is it on a per-customer basis? What’s the total available market? What’s the ROI on acquiring new customers? And how do existing customers respond to our product over time?

Secondly, this definition locates progress firmly in the heads of the people inside the company, and not in any artifacts the company produces. That’s why none of dollars, milestones, products or code can count as progress. Given a choice between what a successful team has learned and the source code they have produced, I would take the learning every time. This is why companies often get out-competed by former employees (Palm vs Handspring to name just one), even though the upstart lacks all of the familiar resources, tools, processes, and support they used to have. (Incidentally, it’s also why these upstarts often get sued for bogus reasons. Companies can’t believe they didn’t steal any of their “precious” assets.)

But learning is a tricky thing to quantify, which is why the word “validated” is so important in this definition. Validation comes in the form of data that demonstrates that the key risks in the business have been addressed by the current product. That doesn’t always mean revenue, either. Some products have relatively obvious monetization mechanisms, and the real risks are in customer adoption. Products can find sources of validation with impressive stats along a number of dimensions, such as high engagement, viral coefficient, or long-term retention. What’s important is that the data tell a compelling story, one that demonstrates that the business is on a solid economic footing. (It being so easy to convince yourself that you’re in one of these “special case” businesses, I do recommend you give revenue a long, hard look first.)

For example, I’ve talked a few times about how IMVU raised its first venture round with monthly revenues of around $10,000. This wasn’t very impressive, but we had two things going for us:

1. A hockey stick shaped growth curve. People often forget the most important part of the hockey stick: the long flat part. We had months of data that showed customers more-or-less uninterested in our product. We were limping along at a few hundred dollars a month in revenue. All this time, we were continuously changing our product, talking to customers, trying to improve on our AdWords spend. Eventually, these efforts bore fruit – and this was evident in the data. This lent our claims about learning and discovery credibility.
   
   
2. Compelling per-customer economics. We had only a small number of customers – if memory serves, only a few thousand active users. But a little math will show that we were making over a dollar per-user per-month. Our cost to acquire a customer on AdWords was only a few cents. Our eventual VC’s were quick to grasp what this meant (in fact, they understood it better than we did): that if our product achieved significant scale, it would be wildly profitable.
   

These two aspects could be plotted on one simple graph, which tells this equally simple story: if there is a market out there for this kind of product, we are the team that will find it and profit from it. That turned out to be a compelling investment thesis, despite our micro-scale results.

Let’s return to my example of the million-dollar-revenue company. If you find yourself in this kind of situation, what can you do? I’d suggest a few things, each rooted in the idea of breaking down the wall between the two halves of this company.

1. Go on an agile diet quickly. With a product development team that is not shipping, any agile methodology will surface major problems quickly. Force anyone who is in customer contact to take the role of the Product Owner and insist that they deliver something new on a short regular interval.
   
   
2. Get product into customers’ hands. The sales strategy currently leaves many customers completely un-served (those that don’t qualify for the founders’ personal time). Start using some of those customers as guinea pigs for a self-serve version of the product. Even if the product is absolutely terrible, it will establish a baseline against which the product development team can try and improve.
   
   
3. Build tools for the sales team that reduce the time investment required for each sale. Instead of devoting all product development efforts to building a full-blown product, try building just those parts of the product that would allow the current sales process to go a little faster. For example, could we develop a simple landing page that would allow customers to pre-qualify for sales time? Iterating on these kinds of features has two benefits: it frees up time for the founders and simultaneously starts getting building a feedback loop with the product development team. Pretty soon, the text on that landing page is going to become an effective explanation for what the product does, because if it’s not the salesman will have to spend time re-explaining the product to potential customers. Time-to-complete-a-sale is not a bad metric for validated learning at this stage.
   

This last point is especially important. Although this kind of team may understand their customers well, they don’t yet know how to talk to them in a standardized way. Without that, they probably won’t achieve significant scale. (For more on how this plays into the process of scaling up, see the Customer Creation stage of the customer development model.) Perhaps they’ll be able to hire someone especially skilled in the marketing skills needed to find this positioning. But in the meantime, by iterating on their product with customers, they have a chance to get there on their own.

Learning is better than optimization (the local maximum problem)

Lean startups don’t optimize. At least, not in the traditional sense of trying to squeeze every tenth of a point out of a conversion metric or landing page. Instead, we try to accelerate with respect to validated learning about customers.

For example, I’m a big believer in split-testing. Many optimizers are in favor of split-testing, too: direct marketers, landing page and SEO experts -- heck even the Google Website Optimizer team. But our interest in the tactic of split-testing is only superficially similar.

Take the infamous “41 shades of blue” split-test. I understand and respect why optimizers want to do tests like that. There are often counter-intuitive changes in customer behavior that depend on little details. In fact, the curse of product development is that sometimes small things make a huge difference and sometimes huge things make no difference. Split-testing is great for figuring out which is which.

But what do you learn from the “41 shades of blue” test? You only learn which specific shade of blue customers are more likely to click on. And, in most such tests, the differences are quite small, which is why sample sizes have to be very large. In Google’s case, often in the millions of people. When people (ok, engineers) who have been trained in this model enter most startups, they quickly get confused. How can we do split-testing when we have only a pathetically small number of customers? What’s the point when the tests aren’t going to be statistically significant?

And they’re not the only ones. Some designers also hate optimizing (which is why the “41 shades of blue” test is so famous – a famous designer claims to have quit over it). I understand and respect that feeling, too. After you’ve spent months on a painstaking new design, who wants to be told what color blue to use? Split-testing a single element in an overall coherent design seems ludicrous. Even if it shows improvement in some micro metric, does that invalidate the overall design? After all, most coherent designs have a gestalt that is more than the sum of the parts – at least, that’s the theory. Split-testing seems fundamentally at odds with that approach.

But I’m not done with the complaints, yet. Optimizing sounds bad for visionary thinking. That’s why you hear so many people proclaim proudly that they never listen to customers. Customers can only tell you want they think they want, and tend to have a very near-term perspective. If you just build what they tell you, you generally wind up with a giant, incoherent mess. Our job as entrepreneurs is to invent the future, and any optimization technique – including split-testing, many design techniques, or even usability testing – can lead us astray. Sure, customers think they want something, but how do they know what they will want in the future?

You can always tell who has a math background in a startup, because they call this the local maximum problem. Those of us with a computer science background call it the hill-climbing algorithm. I’m sure other disciplines have their own names for it; even protozoans exhibit this behavior (it's called taxis). It goes like this: whenever you’re not sure what to do, try something small, at random, and see if that makes things a little bit better. If it does, keep doing more of that, and if it doesn’t, try something else random and start over. Imagine climbing a hill this way; it’d work with your eyes closed. Just keep seeking higher and higher terrain, and rotate a bit whenever you feel yourself going down. But what if you’re climbing a hill that is in front of a mountain? When you get to the top of the hill, there’s no small step you can take that will get you on the right path up the mountain. That’s the local maximum. All optimization techniques get stuck in this position.

Because this causes a lot of confusion, let me state this as unequivocally as I can. The Lean Startup methodology does not advocate using optimization techniques to make startup decisions. That’s right. You don’t have to listen to customers, you don’t have to split-test, and you are free to ignore any data you want. This isn’t kindergarten. You don’t get a gold star for listening to what customers say. You only get a gold star for achieving results.

What should you do instead? The general pattern is: have a strong vision, test that vision against reality, and then decide whether to pivot or persevere. Each part of that answer is complicated, and I’ve written extensively on the details of how to do each. What I want to convey here is how to respond to the objections I mentioned at the start. Each of those objections is wise, in its own way, and the common reaction – to just reject that thinking outright – is a bad idea. Instead, the Lean Startup offers ways to incorporate those people into an overall feedback loop of learning and discovery.

So when should we split-test? There’s nothing wrong with using split-testing, as part of the solution team, to do optimization. But that is not a substitute for testing big hypotheses. The right split-tests to run are ones that put big ideas to the test. For example, we could split-test what color to make the “Register Now” button. But how much do we learn from that? Let’s say that customers prefer one color over another? Then what? Instead, how about a test where we completely change the value proposition on the landing page?

I remember the first time we changed the landing page at IMVU from offering “avatar chat” to “3D instant messaging.” We didn’t expect much of a difference, but it dramatically changed customer behavior. That was evident in the metrics and in the in-person usability tests. It taught us some important things about our customers: that they had no idea what an avatar was, they had no idea why they would want one, and they thought “avatar chat” was something weird people would do. When we started using “3D instant messaging,” we validated our hypothesis that IM was an activity our customers understood and were interested in “doing better.” But we also invalidated a hypothesis that customers wanted an avatar; we had to learn a whole new way of explaining the benefits of avatar-mediated communication because our audience didn’t know what that word meant.

However, that is not the end of the story. If you go to IMVU’s website today, you won’t find any mention of “3D instant messaging.” That’s because those hypotheses were replaced by yet more, each of which was subject to this kind of macro-level testing. Over many years, we’ve learned a lot about what customers want. And we’ve validated that learning by being able to demonstrate that when we change the product as a result of that learning, the key macro metrics improve.

A good rule of thumb for split-testing is that even when we’re doing micro-level split-tests, we should always measure the macro. So even if you want to test a new button color, don’t measure the click-through rate on that button! Instead, ask yourself: “why do we care that customers click that button?” If it’s a “Register Now” button, it’s because we want customers to sign up and try the product. So let’s measure the percentage of customers who try the product. If the button color change doesn’t have an impact there – it’s too small, and should be reverted. Over time, this discipline helps us ignore the minor stuff and focus our energies on learning what will make a significant impact. (It also just so happens that this style of reporting is easier to implement; you can read more here)

Next, let’s take on the sample-size issue. Most of us learn about the samples sizes from things like political polling. In a large country, in order to figure out who will win an election with any kind of accuracy, you need to sample a large number of people. What most of us forget is that statistical significance is a function of both sample size and the magnitude of the underlying signal. Presidential elections are often decided by a few percentage points or less. When we’re optimizing, product development teams encounter similar situations. But when we’re learning, that’s the rare exception. Recall that the biggest source of waste in product development is building something nobody wants. In that case, you don’t need a very large sample.

Let me illustrate. I’ve previously documented that early-on in IMVU’s life, we made the mistake of building an IM add-on product instead of a standalone network. Believe me, I had to be dragged kicking and screaming to the realization that we’d made a mistake. Here’s how it went down. We would bring customers in for a usability test, and ask them to use the IM add-on functionality. The first one flat-out refused. I mean, here we are, paying them to be there, and they won’t use the product! (For now, I won’t go into the reasons why – if you want that level of detail, you can watch this interview.) I was the head of product development, so can you guess what my reaction was? It certainly wasn’t “ooh, let’s listen to this customer.” Hell no, “fire that customer! Get me a new one” was closer. After all, what is a sample size of one customer? Too small. Second customer: same result. Third, fourth, fifth: same. Now, what are the odds that five customers in a row refuse to use my product, and it’s just a matter of chance or small sample size? No chance. The product sucks – and that is a statistically significant result.

When we switch from an optimization mindset to a learning mindset, design gets more fun, too. It takes some getting used to for most designers, though. They are not generally used to having their designs evaluated by their real-world impact. Remember that plenty of design organizations and design schools give out awards for designing products that never get built. So don’t hold it against a classically trained designer if they find split-testing a little off-putting at first. The key is to get new designers integrated with a split-testing regimen as soon as possible. It’s a good deal: by testing to make sure (I often say “double check”) each design actually improves customers lives, startups can free designers to take much bigger risks. Want to try out a wacky, radical, highly simplified design? In a non-data-driven environment, this is usually impossible. There’s always that engineer in the back of the room with all the corner cases: “but how will customers find Feature X? What happens if we don’t explain in graphic detail how to use Feature Y?” Now these questions have an easy answer: we’ll measure and see. If the new design performs worse than the current design, we’ll iterate and try again. But if it performs better, we don’t need to keep arguing. We just keep iterating and learning. This kind of setup leads to a much less political and much less arbitrary design culture.

This same approach can also lead us out of the big incoherent mess problem. Teams that focus on optimizing can get stuck bolting on feature upon feature until the product becomes unusable. No one feature is to blame. I've made this mistake many times in my career, especially early on when I first began to understand the power of metrics. When that happens, the solution is to do a whole product pivot. "Whole product" is a term I learned from Bill Davidow's classic Marketing High Technology. A whole product is one that works for mainstream customers. Sometimes, a whole product is much bigger than a simple device - witness Apple's mastery of creating a whole ecosystem around each of their devices that make them much more useful than their competitors. But sometimes a whole product is much less - it requires removing unnecessary features and focusing on a single overriding value proposition. And these kinds of pivots are great opportunities for learning-style tests. It only requires the courage to test the new beautiful whole product design against the old crufty one head-to-head.

By now, I hope you’re already anticipating how to answer the visionary’s objections. We don’t split-test or talk to customers to decide if we should abandon our vision. Instead, we test to find out how to achieve the vision in the best possible way. Startup success requires getting many things right all at once: building a product that solves a customer problem, having that problem be an important one to a sufficient number of customers, having those customers be willing pay for it (in one of the four customer currencies), being able to reach those customers through one of the fundamental growth strategies, etc. When you read stories of successful startups in the popular and business press, you usually hear about how the founders anticipated several of these challenges in their initial vision. Unfortunately, startup success requires getting them all right. What the PR stories tend to leave out is that we can get attached to every part of our vision, even the dumb parts. Testing the parts simply gives us information that can help us refine the vision – like a sculptor removing just the right pieces of marble. There is tremendous art to knowing which pieces of the vision to test first. It is highly context-dependent, which is why different startups take dramatically different paths to success. Should you charge from day one, testing the revenue model first? Or should you focus on user engagement or virality? What about companies, like Siebel, that started with partner distribution first?  There are no universally right answers to such questions. (For more on how to figure out which question applies in which context, see Business ecology and the four customer currencies.)

Systematically testing the assumptions that support the vision is called customer development, and it’s a parallel process to product development. And therein lies the most common source of confusion about whether startups should listen to customers. Even if a startup is doing user-centered design, or optimizing their product through split-testing, or conducting tons of surveys and usability tests, that’s no substitute for also doing customer development. It’s the difference between asking “how should we best solve this problem for these customers?” and “what problem should we be solving? and for which customer?” These two activities have to happen in parallel, forming a company-wide feedback loop. We call such companies built to learn. Their speed should be measured in validated learning about customers, not milestones, features, revenue, or even beautiful design. Again, not because those things aren’t important, but because their role in a startup is subservient to the company’s fundamental purpose: piercing the veil of extreme uncertainty that accompanies any disruptive innovation.

The Lean Startup methodology can’t guarantee you won’t find yourself in a local maximum. But it can guarantee that you’ll know about it when it happens. Even better, when it is time to pivot, you’ll have actual data that can help inform where you want to head next. The data doesn’t tell you what to do – that’s your job. The bad news: entrepreneurship requires judgment. The good news: when you make data-based decisions, you are training your judgment to get better over time.

The Lean Entrepreneur is here

Last May, I shared the news that long-time Lean Startup advocates Brant Cooper and Patrick Vlaskovits were working on a new book called The Lean Entrepreneur featuring illustrations by FAKEGRIMLOCK. That new book is about to hit bookstores everywhere.

I was honored that they asked me to write the foreword, and with their permission I'm posting an excerpt below. After the 2012 conference I viewed it as an opportunity to reflect on the growth and evolution of the movement as a whole.

One of the things that I love about what Brant and Patrick have done with The Lean Entrepreneur are the numerous case studies of how entrepreneurs are tackling new ventures, minimizing risk and learning their way to success - these case studies will speak to garage startups and corporate entrepreneurs alike.

A few of the detailed case studies include:

• Tech legend Bill Gross building an MVP in 1999 to test demand for online car sales, which grew into CarsDirect.com.
• LitMotors approach to using Lean Startup to create a new vehicle category.
• AppFog creating "high-hurdle" experiments to surface authentic early adopters with real pain.
• The Embrace infant warmer was developed - by getting out of the country -- and how Rob Emrich learned and scaled his non-profit, Road of Life. (Social entrepreneurs take note!)
• BetaBrand building apparel MVPs and testing them quickly with targeted customer communities.
• Telecom O2 learning to move at the speed of the internet
• 500 Startups and their accelerated feedback loops on what works, and what doesn't work in early-stage investing.
• Scott Summitt iteratively leveraging the emerging technologies of digital fabrication and 3d-scanning to change people's lives.
• PayPal, under the leadership of David Marcus and Bill Scott, re-defining and re-engineering itself by embracing Lean Startup to improve the product experience.
• KISSmetrics building and empowering cross-functional teams to attack problems in their sales funnels via hypothesis testing.
• Intuit showing large organizations how to combine Lean Startup with horizon planning to nurture internal innovation and startup experiments.
• Berkeley Pizza: from pizza in the farmer's market to a sit-down restaurant.

Another thing I love about The Lean Entrepreneur is how Brant and Patrick are treating their book like a startup. One example is in their marketing, which in today's environment requires providing value. This doesn't mean pointing to the product and describing the product's value, but rather the marketing provides value itself.

As part of their book campaign, Brant and Patrick have teamed up with General Assembly to offer 4 free online video classes, including:

Lean UX Research Techniques - Rapid Prototyping - How to Hire Developers - Growth Hacking

Every book pre-order get access to all of these classes - and quite a bit more. You can pre-order on their site until February 10^th 2013, and after that on Amazon.

Lastly, I wanted to share with you the foreword I wrote for The Lean Entrepreneur. As we head into 2013, it's a good time to reflect on how far the Lean Startup movement has come:

---

When I first started blogging in August of 2008, I had no idea what to expect. Startup blogging was hardly "cool" back then. Plenty of venture capitalists advised me against it. 

My personal background was as an engineer and my companies had been Web-based startups, so that is what I wrote about. Struggling to explain the successes and failures of those companies, I discussed principles like continuous deployment, customer development, and a hyper-accelerated form of agile. When I delved into lean manufacturing, I discovered the concepts and terminology dovetailed. The result: a new idea I called The Lean Startup. 

I started with some basic theory: that a startup is an institution designed to thrive in the soil of extreme uncertainty; that traditional management techniques rooted in forecasting and planning would not work well in the face of that uncertainty. Therefore, we needed a new management toolkit designed explicitly for iteration, scientific learning, and rapid experimentation. 

At the time, I viewed it as incidental that the theory might be tied to a particular industry, such as high-tech startups or web-based environments. Lean, after all, emerged from Toyota, a huge automobile manufacturing company. I simply stated my belief that Lean Startup principles would work in other types of startups and in other areas of business where uncertainty reigned. 

Boy, was I unprepared for what happened next. I was hopeful that we would change the way startups are built - but I didn't know.Fast-forward more than 4 years and I'm astounded by what has emerged. A nascent community has blossomed into a full-fledged movement. Entrepreneurs, both new and experienced, proudly share their Lean Startup learning in case studies, conferences, and many, many blogs. Books, workshops and courses authored by passionate practitioners relate experience, share insight, and create tools to teach students ways to make Lean Startup principles their own. Many investors, advisors, mentors and even celebrity entrepreneur icons speak the Lean Startup language. 

It's a big tent. We stand on the shoulders of giants: customer development, the theory of disruptive innovation, the technology life-cycle adoption theory, and agile development. Complementary lines of thinking, such as that of user experience professionals, design thinking practitioners and the functional disciplines of sales, marketing, operations and even accounting, come together to share practices that lift us all. 

Lean Startup has gone mainstream. I wish I could say that this was all part of some master plan, that I knew all along that companies of all sizes - far outside the high-technology world - would embrace Lean Startup. I wish I had foreseen that within a year of publishing The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Achieve Radically Successful Businesses, many large organizations, including such monsters as the United States Federal government (!) would have recognized that to cope with today's world -faster, more competitive, and inundated with data - new methods are needed to keep up. The truth is that all of this change has happened faster and more thoroughly than any of us imagined. And - as you're about to see - we're just getting started. 

That's why I am so excited by the volume you hold in your hands. The Lean Entrepreneur is about those new methods. Brant Cooper and Patrick Vlaskovits are among the earliest adopters of new ideas such as Lean Startup and customer development. Their new work turns their lens on three primary focal points: how to interact with customers, run experiments, and use actionable data to move the needle of any uncertain business endeavor. 

As with all of their work, theirs is not just a book of theory. Brant and Patrick provide great tactical depth in each of these areas. 

They endeavor to answer the question: No matter where you are as an organization, how do you know where to focus your Lean Startup activities? The Lean Entrepreneur offers new thinking, tools and activities that help organizations identify and act upon business model challenges in a waste-eliminating manner. Following the precepts of traditional lean thinking, Brant and Patrick introduce the value stream discovery process, which helps organizations hypothesize what they must do, including product development, marketing, and sales in order to create value. These business model assumptions are then ripe for testing, measuring and iterating upon. 

Further, the value you create is meaningless without a customer who needs, wants, desires and ultimately, determines the final value of your creation. Brant and Patrick spend considerable time helping you think through your customer segments. Cleverly and in the spirit of the scientific method, they even help you discover where your customer theory is wrong. 

Everyone likes a good story; Brant and Patrick interviewed dozens of entrepreneurs and documented numerous case studies both inside and outside of high-tech, in both startups and large enterprises. There's even a classic Wizard of Oz minimum viable product that dates back to 1998! 

Make no mistake, Brant and Patrick have been here since the beginning. They self-published The Entrepreneur's Guide to Customer Development in April of 2010
From the outset, they were both practitioners and mentors, urging entrepreneurs not to follow a paint-by-numbers approach, but rather to think lean: fast, agile and continuously learning. Over the last two years they've traveled around the world speaking, advising, and teaching Lean Startup. 

The Lean Entrepreneur is an important addition to the growing library of principles and practices designed to improve how we tackle innovation and uncertainty, be it in tech startups, Fortune 100, non-profits or government. 

I consider myself lucky to count Brant and Patrick as friends and colleagues. It is my hope that from this book you will gain valuable insights, make Lean Startup your own, and - much more importantly - that you are successful in changing the world for the better. 

Eric Ries,
San Francisco, December 2012

---

If you want to read more, you can find an excerpt of Chapter 6, Viability Experiments here.

MVPs and Excellence

Lately, I’ve been hearing from a lot of entrepreneurs experiencing pushback to the concept of minimum viable product. Their teams may disagree about what a product should look like, who the customer is, and which distributors to work with, but one thing they can all agree upon is: “We build high-quality products in this company. We wouldn’t even know how to go about building something ‘minimally viable.’”

How can we assure our teams that they won’t be penalized for adapting an iterative approach—even if the first version bombs? How can we make it clear that our goal is nothing less than delighting the customer? In fact, with an MVP we are not asking our teams to deliver low-quality work, we’re adopting a strategy for driving excellence throughout the organization.

An MVP is an experiment on the way to excellence.

When people hear the phrase “Minimum Viable Product” they sometimes forget to ask: minimum in regard to what? They worry they’ll need to do the same amount of work in less time by cutting corners. They fear they’re being asked to create a low-quality product that will put their reputation—or even people’s lives—at risk. 

This is a misconception. “Minimally viable” does not mean operating in a sloppy or undisciplined way, building bad code that’s going to result in a lot of technical debt, or ignoring safety or health concerns. An MVP is not an excuse to throw our beliefs about quality out the window; it’s simply an experiment on the way to excellence. 

Instead of taking one big swing with the launch of a new product—devoting months to the design of one technical feature or spending years in stealth mode developing a product without evidence that customers want it—it is an iterative approach to learn who the customer actually is, and what’s honestly required to delight them. 

Why do we think that spending more time developing a product before sharing it with customers will get us closer to discovering what they really want? Or that mistakes are something to be swept under the rug? Who is familiar with a team that produced excellent work because they got less feedback and moved slower? Where is the evidence to support that belief?

An MVP approach can help us learn about what the customer truly values before we’ve invested too much time and money into building something they don’t need or want. Our work is not done until the customer is, in fact, delighted.

Of course, that’s easier said than done. But there are concrete steps we can take to create cultures of excellence in our organizations using an MVP approach.

1. Clarify what MVPs are and why they’re important. Remember, MVPs are just one part of an iterative Build-Measure-Learn process.

We use the term “Minimum Viable Product” as a reminder to people with product development backgrounds that we’re going to build something. We are going to take something to the customer that is as real as we can make it, but we’re not going to overdo it by trying to do something too elaborate. This is a built-in cure for the human tendency to over-engineer solutions. 

It’s impossible to think about MVPs without remembering how they fit into the Build-Measure-Learn cycle:

• Enter the Build phase as quickly as possible with an MVP that will allow us to test a clear hypothesis we have about our product or strategy.
• Measure its impact in the marketplace using actionable metrics that help us analyze customer behavior.
• Learn whether our original assumptions about the product, process, and customer needs were correct or whether we need to change strategies to better meet our vision.

The Build-Measure-Learn feedback loop does not end once we’ve put our first product into the market. Every new launch of an MVP is an opportunity to gain valuable information about how well we’re meeting customer needs--and whether we need to adjust our strategy with a pivot.

2. Building a culture of excellence is our job as leaders.

I’ve long since lost track of the number of people who have given me excuses about  about why the MVP approach “won’t work in our business.” Almost all of them have gone on to discover that it can, in fact, be made to work – in businesses as diverse as software, education, high-tech healthcare equipment and gas turbine manufacturing.

While every industry has unique challenges that make an iterative approach difficult, what teams are really saying when they balk at an MVP approach is they’re not willing to do the work required to run experiments. Creating hypotheses and then putting them to the test is never an impossible feat, but it is one that will require teams to work differently.

And that means nothing less than full support from leadership. That’s true in the tiniest startup and the world’s largest companies. New thinking requires a act of leadership.

As leaders, we’re responsible for creating and supporting the platforms that will allow for experimentation. We have to make sure that our teams have the right tools and insist the work be done iteratively. And we have to hold them accountable for a high standard of success: “Show me progress along the way, but remember that you’re not done until you’ve delighted the customer.”

Sometimes this means building infrastructure that makes experimentation possible: One way of doing this is creating an innovation sandbox to protect the rest of the organization from the experiment. For example, pick a few customers to experiment with and promise to pay them penalties for any kind of fault during the experimentation period. Or select a subset of your web traffic and divert those customers to a new experience. If customers are unhappy at any time during the experiment, the startup team has to promise to make it right. That often means engineers taking support calls and sales people working with existing customers. That’s a good thing, because it enhances the learning your team will get. It also gives your team the cover to work quickly; while you may lose money in these early days, you’ll have learned a great deal about what the customer really wants. 

Often, it requires thinking creatively about what shape an MVP can take. One thing to keep in mind is that while every MVP provides some "quantum of utility" (to borrow Paul Graham's phrase) to the customer, there’s a wide range of MVPs. When your sales process is long and complicated, models and brochures can act as products. Even though they’re not “products” in the traditional sense of the word, they still offer us a chance to gauge customer interest by asking for some exchange of value, even if that exchange is not monetary. For example, we can ask customers to spend more time talking to us about the product, take part in a training program, or agree to recommend the product to other decision-makers in the buying process.

3. Cross-functional teams are key to delighting the customer.

Once we’ve set up the right conditions for experimentation, it’s important to create cross-functional teams who, together, can assess which features truly deliver value.

Say you wanted to run an experiment to test which aspects of a new appliance drive the most value. You couldn’t just send a salesperson because they would not know which features drove the most cost. Nor could you send a design team without knowledge of the supply chain or the current market landscape. This experiment requires a collaboration between the product designer, a salesperson, and someone on the manufacturing side. By working together, they can identify what drives cost and what drives value--when, prior to running such an experiment, you couldn’t really know what drives either. 

It can be challenging to make the case that all functions should be represented at both large organizations where it’s difficult to get people assigned to any project full-time and at small startups where resources are scarce. But remember that the goal of Lean Startup methodology isn’t just to build a product, but to learn how to build a sustainable business: a cross-functional approach is key to this kind of learning.

Building a culture of excellence will also require rethinking the way you measure and evaluate performance. When evaluations are tied to functional performance, a good day is one in which everyone did their job well: Did engineers strive to create the “highest-quality” product or service? Did legal limit the company’s liability? Did marketing anticipate external trends accurately enough?

But if we build products we can’t sell or processes no one uses, it doesn’t matter if we executed on time and on budget. Entrepreneurs and general managers should not simply be content with functional excellence but strive for products, processes, and systems that delight our customers. 

4. If a customer doesn’t like what you’ve made, that’s a discovery, not a failure.

Even after teams have grasped the concept of the MVP on an intellectual level, they often find it hard to pull the trigger on those first iterations; it takes them a while to set up those initial meetings with customers or launch their first MVP.

They can be held back by any number of worries: What if we show the customer something they don’t like? What if going in with a minimum viable product makes it seem as if we don’t know what we’re talking about?

As counterintuitive as it sounds, finding out that our minimum viable product is too “minimal” is good news. We can simply apologize, and use the customer feedback to build a new version that does meet their needs.

The good news about MVPs is that we can always make them more complicated as we go, so we might as well start simple and let the project grow in complexity until we have delighted the customer. If we do something we believe is too simple and customers agree, we can consider that a major win. It doesn’t mean the end of the project; it means we can use what we’ve learned to build the next iteration.

We don’t want to wait around for some mythical moment of perfection nor do we want to run around with no process. We use MVPs to test our strategy— if it’s not helping us achieve our visions, we need to make an adjustment.

If your team is concerned that creating an MVP will mean skimping on quality, it can be a good indication that they care about what they’re building. They recognize the importance of excellence and they strive to produce high-quality work —it’s why you hired them. This approach is not about asking people to ignore their skills, values and gut instincts about quality; it’s about making sure that they don’t waste their time and talent over-designing features that are not actually important to the customer. It’s about empowering your team to leave dead-end projects and activities behind so that they can invest their time into work that truly matters.

Business ecology and the four customer currencies

Lately, I’ve been rethinking the concept of “business model” for startups, in favor of something I call “business ecology.” In an ecosystem, each participant acts according to its own imperatives, but these selfish actions have an aggregate effect. Some ecosystems are stable, others malign, and others grow and prosper. A successful startup strives for this latter case. I think this concept is necessary in order to answer the truly vexing startup questions, like: “Should startups charge customers money from day one?”

Let’s begin with the four customer currencies. I had a lot of use for this concept back when I worked on game design and virtual worlds. In order to maintain game play balance, game designers have to take into account the needs of customers who have an excess of four different assets: time, money, skill, and passion.

If players with more money than others can simply buy their way to the top of the heap, a multiplayer game fails – because this makes the game un-fun for other players. The same is true if kids who have an unlimited amount of time on their hands are guaranteed the top spot – this isn’t going to be very fun for the busy professionals who want to play only casually. Chess is only fun for those who have the requisite skill to play well – and even then, only if there are ranking systems to make sure that players of relatively equal skill play each other. If you could buy a higher chess ranking or, worse, simply grow it by logging more hours, that would ruin the system for everyone. And passionate players are often the backbone of game communities – especially online. They run the clubs, forums, groups and mailing lists that make the game more fun overall. If they are barred from participating (say, because they lack the skill needed to prevent advanced players from killing them all the time), the game is worse off.

Each of these four currencies represents a way for a customer to “pay” for services from a company. And this is true outside of games. Constructing a working business model is a form of ecosystem design. A great product enables customers, developers, partners, and even competitors to exchange their unique currencies in combinations that lead to financial success for the company that organizes them.

Here’s the ecosystem we built at IMVU, just to give one example. We cultivated a passionate community that nurtured a skilled set of developers. Those developers create an incredible variety of virtual goods: 3D models, textures, homepage stickers, music, and much more – more than three million in total last time I checked. This variety entices millions of end-users to invest their time and passion with IMVU, providing many incentives for a small fraction of those users to become paying customers. Those paying customers provide IMVU with sufficient profits to reinvest in the core experience for everyone. It’s a working, growing, ecosystem.

Having a balanced ecosystem is what game designers strive for. But startups strive for something else: growth. Thus, business ecology is concerned with both ecosystem design and finding a driver of growth for that ecosystem. In a previous post, I covered the three main drivers of growth: Paid, Sticky, and Viral. When a startup finds a working value-creating ecosystem that supports one of these drivers of growth, watch out. They’re off to cross the chasm.

And this is why questions like “Should a company charge money from day one?” are nonsensical. Some companies definitely should. Others definitely shouldn’t. In order to tell which is which, you have to understand the unique ecology of the business in question.  Let’s look at some examples:

• In a traditional business, customers pay money for a physical artifact (a product) or a service. Companies use that money to market the product or service to more customers. This is the simplest ecosystem and simplest driver of growth. A business that strives for something like this should absolutely be charging money from day one, in order to establish baselines for their two key metrics: CPA (the cost to acquire a new customer) and LTV (the lifetime value of each acquired customer). In other words, the minimum viable product is designed to answer the question: does the product generate enough demand and margin to support a growing ecosystem?  
• Now consider a traditional media business. By paying money to content creators (ie writers, producers, talent), the business uses builds up assets that are of interest to other consumers. Those other consumers pay for this content sometimes with money, but more often with their attention. This attention is valuable to yet another set of people: namely, the traditional businesses (see above) who are using marketing to grow, and are looking to advertise to new prospects. The value of the attention that the media company collects determines how profitable it is. In the old days, these media companies would then themselves plow this profit back into marketing and advertising, and grow. Today, many of these businesses are suffering because the ecosystem no longer balances thanks to the Internet. (Sorry about that.) If you’re starting a new media company, does it make sense to charge from day one? Probably not – you need to be finding an audience, making sure that audience will trade you their attention for your content, and – most importantly – establishing a baseline for how much that attention is worth to advertisers. A minimum viable product in this category must answer the question: does my media content or channel command the attention of a valuable audience?
  
  
• Let’s look at a viral growth company, like Facebook. They are a classic case of a company that doesn’t seem to care about charging customers money. Here’s Andrew Chen’s description:
  
  
  

  “it strikes me that consumer internet companies often don’t care much whether or not they have viable businesses in the short run. If you are building a large, viral, ad-support consumer internet property, you just want to go big! As soon as possible!”
  

  This is a common sentiment, but I don't agree. I think it uses the phrase “viable business” in too narrow a sense. When Facebook launched early-on at college campuses, it was immediately apparent that they had a viable business, even though they weren’t charging customers for anything. Why? Because they were collecting truly massive amounts of attention and they had an amazing driver of growth. Those two factors made it relatively easy for them to raise enough money to avoid having to build a profitable business in the short term. But that doesn’t mean they didn’t have a viable one. The ecosystem worked, and was growing. Figuring out how to turn that attention into cash seems to have been pretty obvious to Mark Zuckerberg. For a true viral ecosystem, the minimum viable product is designed to answer the question: can I unlock viral growth mechanics while still keeping my ecosystem alive? As many viral companies have found to their chagrin, quite a few viral products are fundamentally useless. Although they grow, they don’t actually collect enough of any customer currency to be viable.
  
  
  In fact, the viral metaphor is actually more apt than many people realize, once you look at it from an ecological perspective. Facebook is actually quite rare – many other viral products didn’t really build their own working ecology: they colonized someone else’s. That was true for Paypal cannibalizing eBay, YouTube and MySpace, and could still be true of Slide, Zynga, or RockYou – we’ll see.
  
  Now, Andrew’s excellent piece that I quoted from above correctly diagnoses two situations where consumer internet companies often get in trouble:
  
  
  
  1. They focus too much on short-term revenue, getting caught in a local maximum via constant optimization. They aren’t really engaged in customer development, they aren’t getting inside their customers’ heads, and they aren’t crafting a robust ecosystem. For a consumer internet company in particular, this is often due to a lack of design thinking.
     
     
  2. They get focused solely on growth. This isn’t helpful either, as countless companies have shown. If you haven’t figured out the ecosystem, growth is useless – whether it is a acquisition-only viral loop, like Tagged, or an advertising blitz like countless dot-bombs.
  
  
  

• Let’s consider one last example, a sticky-growth company like eBay or World of Warcraft. Here the goal is to create a product whose ecosystem makes it hard for customers to leave. eBay offers their customers an opportunity to monetize their skill and passion via online trading for hard currency. World of Warcraft offers beautifully balanced and addictive game play, for which customers trade all four currencies in bewildering combinations. Like eBay, these investments are best understood as trades between players, which is what makes multiplayer game design so much harder than its single-player counterpart. What these products all have in common is the question their minimum viable product is attempting to answer: does this product have high natural retention built-in?

Understanding the four customer currencies allows us to avoid these problems, and also unify a number of different concepts that have been floating around. Take the minimum viable product, for starters. How should the word viable be understood? Here’s the original definition I proposed for MVP:

“the minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.”

Of course, this begs the question: what are we trying to learn? Now I think I can answer this question with some certainty: we want to learn how to construct a business ecology harnessed to the right driver of growth. And how do we validate that learning? By creating a model of the ecosystem we want, and showing that actual customer behavior conforms to that model. And, of course, if customers don’t behave the way we expect, it’s time to pivot.

That necessarily means that different types of startups will be seeking to learn different things. Minimum viable product is a tactic for mitigating risk. It doesn’t say anything about which risks it should be used to mitigate. Startup founders need to use their own judgment to ask: which is the riskiest assumption underlying my business plan? In each of the ecosystem examples I gave above, the tactics of the minimum viable product are quite different. In some cases, Tim Ferriss-style landing page tests will suffice. Others require Steve Blank-style problem/solution presentations. And others require an early product prototype. The level of design required will vary. The level of engineering quality will vary. The amount of traditional business modeling will vary.

And now we can answer the biggest question of all: how do we know it’s time to scale? Or, to borrow Steve Blank’s formulation, how do we know it’s time to move from Customer Validation to Customer Creation? I think I have a solid answer here too: when we have enough data that shows our business ecology is value-creating and also ready to grow via a specific driver of growth.

Founders struggle with this question. Successful startups don’t. In almost every case I’m aware, the question never had to be asked. When an ecosystem is thriving and growing, it takes work just to keep up with its scaling needs. This was true at Facebook, eBay, and Google – and at countless other successful startups. Marc Andreessen has already coined a phrase for what it looks like: product/market fit. One clue that you don’t have product/market fit – you’re trying to evaluate your business to see if you have it. It’s probably time to pivot.

These concepts have important implications for any lean startup. My whole goal with the lean startup movement has been to learn how to tell the difference between value-creating activities and waste in startups – and then to start eliminating waste. In an entrepreneurial situation, this is hard, because artifacts that we are creating (products, code, marketing campaigns, even revenue) are of secondary importance. The real value we create is learning how to craft profitable ecosystems. Even then, it’s the learning that’s the real value. So, when evaluating any activity, ask: is this helping me learn more about my startup’s ecosystem? If not, eliminate it. If so, ask: how could I get even more learning while doing even less work?

Most of all, beware one-size-fits-all startup advice. In order to figure out what applies to your unique situation, focus on the principles. Who are the customers? What currencies do they have? And what problems do they need solved? Look for a balanced ecosystem and a driver of growth. And be sure to hold on to the reins once you find it.

Lo, my 2295 subscribers, who are you?

I've previously written about the advantages of having a pathetically small number of customers, and how to get started with customer segmentation. Since my subscriber count has doubled again (thanks!), it seemed time to return to the subject of investigating who my customers are. Today, I'd like to spend some time on two techniques: the problem presentation and learning what channels of information reach customers.

Let's start with what I learned last time. Thanks to those of you who were willing to fill out the survey, I learned my net promoter score (about 25) as well as some clear other segmentation insights: about 80% of you are founders of or work at a startup, you read many of the same other blogs, and many of you would like to engage with Lessons Learned in formats and venues beyond this blog.

Before I go any further, here's the link to the new survey. If you are willing, please take a moment to fill it out before proceeding with the rest of the post; it'll mean less biased results, and you'll have a better idea what I'm talking about later.

Click Here to take survey

As always, I like to start with the NPS question. If this was a startup, I'd be asking it of a sample of customers even more often, because getting a regular tracking result is helpful for informing your decision loop. As I've written before, it is often a leading indicator of major problems. Another key benefit is what it enables in terms of analyzing responses to the survey. For example, those of you who classify yourselves as promoters are more likely than average to have found out about Lessons Learned from another blog. People who joined from sites like Reddit are more likely to be detractors. That's not an earth-shattering insight, but segmenting answers gets more interesting as the quality of the questions we're asking improves. In the first survey, most of the questions were open-ended because I had no idea what kinds of answers to expect, or even if anyone would answer at all.

That's a common pattern to gaining customer insight. Start with open-ended, subjective inquiry, like talking to customers on the phone. As you begin to see patterns, turn to more quantitative options like structured surveys or split-tests to confirm those patterns without as much observation bias.

In the new survey, you'll find more structured questions. They are centered around two themes. The first is asking after problems that you're experiencing. The challenge with these sorts of questions is to keep them open-ended, and to read the answers carefully. Every product ultimately is satisfying a need in the customers who buy it or use it. Products that don't solve satisfy any need for any customers tend to die out quickly. The more painful curse for startups is the product that satisfies a need or solves a problem - but that problem is not very important.

For startups, it is critical that the problem you are trying to solve is top of mind for at least some customers. To see if you're on the right track, consider this hierarchy of customer pain, courtesy of the Four Steps to the Epiphany:

1. Customer has a problem.
2. Customer knows they have a problem.
   
3. Customer knows they have a problem and is actively trying to solve it.
4. Customer has a budget allocated to the problem or is otherwise trying to spend money solving it. Ideally, they are trying to spend money but can't find a vendor to solve it for them.
   
5. Customer is actively spending money, time or effort on solving the problem and failing. This is likely to be an early adopter, because they have the vision to see that the problem is important and they are already putting together a solution out of piece parts. But if that piece-parts solution is great, then it's unlikely they are going to need to buy any additional products.

Because this language may lead you to believe this concept is for enterprise sales only, I thought I'd walk you through an example from the world of consumer electronics. One of my favorite gadgets at home is the Harmony Universal Remote. This device let's you control all your home entertainment devices without requiring you to "train" it from your existing remotes. It connects to the internet to automatically download the necessary codes. But the real breakthrough is a usability insight: that people don't really want to control devices. They want to do activities. So the remote has buttons for simple high-level activities like "watch TV" or "watch a movie." When pressed, the remote configures all the right settings automatically to fulfill the customer's intention. It works like magic.

Imagine what it must have been like for Harmony in their early days. Most customers they talked to have a problem: their coffee table is littered with tons of obscure remotes. Do they know they have a problem? Not in that many cases. Sure, people like to complain about their consumer electronics, but most people have way bigger problems in their lives. But there are some people who have this problem in a more severe form, and since we have the benefit of hindsight, we can quote them. Here's a representative view from an actual Amazon customer review:

We all know how to operate our own entertainment center, but what happens when you have to explain it to your babysitter, mother-in-law, or your wife? Before this remote, I had a Sony RM-AV3000. It was a decent remote, except that it took hours to set up the macros, and the instructions that I had to leave for the babysitter were longer than the instructions for my kid!
This remote took 15 minutes to set up 6 different components (including lighting) on Windows Vista. Long lasting battery, great feel, a little pricey but anything to make it easier on me. After all, a happy wife is a happy husband. A happy mother-in-law...is a happy? Well, I wouldn't go that far!

That's an example of someone who had a problem, knew they had a problem, and had previously tried to solve it with a complex and inadequate solution. If Harmony had sat down with this person in their early days and asked them about their "remote control situation" I am confident they would have heard an earful. And if they had sketched out the concept for the activity-based remote they were building, I'm pretty sure he would have signed up on the spot. That's an early adopter.

If you are grappling with understanding what problem your product solves, try building a "problem presentation." This is way of talking to potential customers in which you do not mention your product at all. If it's appropriate, build a few slides. Or create an online survey. Or just talk to people in person. However you do it, focus on explaining the problem you think you will eventually solve and how important it is. If your customer is nodding along, keep getting more and more detailed about how the problem affects their life or their company. When they look confused, stop and ask them to explain, clarify, and correct you. When you get to the point where you can get three consecutive customers to nod through your whole presentation, congratulations. You've got an accurate read on the problem. Then, and only then, should you proceed to start trying to sell people your solution.



The second set of questions in the survey are focused on learning where you currently go to find solutions. In particular, I've focused on the two channels of communication that were most commonly suggested in the first survey: conferences and books. In a startup, these channels of communication are critical to understand for advertising, launch and PR purposes. But beyond the obvious, it's also important to know where customers go to get answers to pressing questions. If they are willing to pay big bucks, for example, to attend a conference on a certain topic, that bodes well for your ability to sell them a product related to that topic. And, for conferences, it also tells you where to go to talk to them. This is especially important for startups that have a small number of target customers, like those that sell only to large companies in specific industries.

I'll mention one last thing about knowing what channels of communication your customers engage with. It also gives you insight into their language. This is important, especially if you are a very early stage company. If you're used to pitching your company in a startup hub, like here in Silicon Valley, you may suffer from a debilitating disease: the inability to describe your product to normal people. That's no big deal if all you want to do is sell to people who live in startup hubs. But for most of us, that's devastating.

For example, when pitching IMVU, I used to say things like: IMVU is an 3D avatar IM platform with a micropayment economy powered by user-generated content. No customers understood what that meant. If we had bothered to ask them where they get their information, none of them would have said TechCrunch or Mashable. If we had spent even a tiny amount of time engaging them on this question in our early days, we could have saved a lot of time. Of course, we would have had to spend some time reading teen magazines. But isn't that a small price to pay to change the odds of your startup making it?

So thank you so much for subscribing. And an extra thank you to those of you who can spare a few minutes more to let me know what you're thinking. Have suggestions for other questions to ask in future surveys? Go ahead and leave your thoughts in the comments.