A month is fifteen weekends

Lean Startup Machine is the brainchild of Trevor Owens, Josh Horn, and Ben Fisher, a hackathon-style competition where teams come together on a Friday evening and build a brand new startup – by Sunday. It’s an impossibly short amount of time. To make it more difficult, unlike your standard issue hackathon, the judging on Sunday is not just about who can make a cool-looking prototype. Teams launch real products to real customers. The judging criteria is all about validated learning. Teams are expected to talk to customers, build minimum viable products, and even pivot as necessary. Luckily, the teams are assisted by some of the best startup mentors in the industry.

Lean Startup Machine came to San Francisco a few weeks ago, after successful debut runs in New York and Chicago. I’ve had the privilege of being a judge in previous weekends, but this is the first time I was able to attend in person.

When I first heard about Lean Startup Machine, I’ll admit I was skeptical. In my consulting practice, I struggle to get teams to do validated learning when they have months or years of runway. How much learning could a team do in only 48 hours, with no budget whatsoever?

But the results were amazing. Out of eleven teams, the three finalists we picked all had discovered honest-to-goodness viable businesses. Each had pivoted several times and had built one or more MVPs. One even had $100 in revenue (in escrow) from real live customers. Beyond the three finalists another few teams had made solid discoveries and would probably have been contenders if they’d just had a little more time. I bet more than half the teams wish they had just one more day – and could have achieved something great in that time.

Think about that for a second. If only they had one more day. Think how valuable a single day is, when  used to its maximum potential. And now think how casually we throw a day of work away, when it’s just one tiny part of a huge batch, as in a monthly release cycle.

And that brings me to the fuzzy math that forms the title of this post: a month is fifteen weekends. Normally, we consider a month-long development cycle to be a short time. We can barely get anything done. We feel rushed. We are too busy to learn anything. We think we’re acting fast. The Lean Startup Machine puts the lie to that idea. I forced me to think about how much validated learning these teams could accomplish if they had a whole month - fifteen times longer - to work.

It wasn’t just that the winning teams were so productive. It was the wide disparity between the most and least productive teams. Remember, everyone had the same amount of time. Most teams were about the same size. And every team started with a promising idea (which were chosen by voting on the first day). But some teams managed to get only a single landing page-style MVP done over the course of the whole weekend. Some managed little more than a survey of customers, or a few conversations. Their learning was decidedly un-validated: stories, anecdotes, maybe some strong thinking at the whiteboard.

Talking to attendees afterwards, I got the sense that these laggards learned the most from the weekend. I bet when those teams disperse and go back to their day jobs (many of whom work at startups), they will have the biggest impact on their teams and colleagues. Because they saw, first hand, just how hard it is to make progress, even with a great team and great idea, if you’re not flying through the Build-Measure-Learn feedback loop.

For example, one team managed to put together a very decent looking minimum viable product, in the form of a landing page with a “click to signup button” that basically did nothing but collect data about who was clicking. And their MVP even had a reasonably high click rate. But is a 25% click rate validation of the idea? That depends on who’s clicking and why. Do they understand the product? Are they eager to try it, or were they just enticed by the shiny button? Unfortunately, the team had no way of answering these questions. They weren’t even collecting contact information from these first customers. They were just counting clicks.

I don’t mean to pick on this team. After all, I can easily imagine how they got in this situation. They were trying to find the minimum viable product, after all. I can almost hear the argument in my imagination: “guys, let’s just ship this thing and see what happens – we can always add contact fields later.” This is a classic startup fallacy: “ship it and see what happens.” Whenever you use this plan, you are guaranteed to succeed – at seeing what happens. Unfortunately, if you cannot fail, you cannot learn. And so this team had succeeded in building and shipping something; they even had some measuring. But they could not learn, and – most importantly – they could not get another turn through the feedback loop. Remember, an MVP is not an event in itself; it is the beginning of a process. (Dare I say it, "you still have to iterate?")

I don’t know how many real companies will come out of a weekend. After all, the scientific method makes it easier to disprove a bad idea than “prove” a good idea. In one notable case, a team was able to conclusively invalidate a business that I have been pitched by venture-backed entrepreneurs many times – with a full day to spare. Compared to entrepreneurs who’ve blown millions of dollars pursuing the same vision, this is a way better outcome. Since they had extra time, they tried a pivot into a much more promising idea. By the time of the judging, they had an MVP in the market with real customers signed up. Will this new idea ultimately prove profitable? Only time will tell.

Regardless, I’m a believer in Lean Startup Machine because of its educational impact. Reading is good. Going to conferences is good. Watching videos is good. But taking action is better. Lean Startup Machine offers everyone the opportunity to learn some of our key tenets, by immersion, rather than by rote: getting out of the building, pivoting, minimum viable product, and validated learning. In real life, the most comfortable thing is to stay in the building, leave assumptions untested, and hope for the best. LSM forces participants to go outside the comfort zone, or fail.

I was surprised by the number of people who attend Lean Startup Machine who already work in a startup full-time. They are not there to learn if their current startup idea is good or bad. They are there to learn a new, faster process that they can bring back to their company.

As one participant wrote,

"The Lean Startup Machine was a synthesis of a lot of my frustrations with how the companies I've worked for over the past few years have created products, and showed me that a much better way was possible. A hugely eye opening experience for me, and I had a blast working with really smart, energetic people."

If you’re reading this right now, and wondering if maybe your company could go a little faster, pay attention to Lean Startup Machine. It may be coming soon to a city near you. The next LSM is scheduled for Boston on Feb 25, 2011. Early bird registration just started, and it’s sure to sell out.

Lean Startup junkies

Offered without comment. Enjoy.

Warning: NSFW



(I did not create this video and I have no idea who did. Whoever you are, get in touch and I will heap copious praise upon you.)

Case Study: UX, Design, and Food on the Table

(One of the common questions I hear is how to reconcile design and user experience (UX) methods with the Lean Startup. To answer, I asked one of my favorite designers to write a case study illustrating the way they work, taking us step-by-step through a real life redesign.

This is something of an IMVU reunion. The attendees at sllconf 2010 were wowed by Food on the Table's presentation. If you weren't there, be sure to watch the video. Manuel Rosso was IMVU's first VP of Marketing, and is now CEO of Food on the Table, one of the leading lean startups in Austin. I first met Laura Klein when we had the good fortune of hiring her at IMVU to join our interaction design team. Since then, she's gone on to become one of the leading experts implementing UX and design in lean startups. 

In this case study, Laura takes us inside the design process in a real live startup. I hope you'll find it illuminating. -Eric)

A lot of people ask me whether design fits into the lean startup process. They're concerned that if they do any research or design up front that they will end up in a waterfall environment.

This is simply not true. Even the leanest of startups can benefit from design and user research. The following is a great example of how they can work together.

A couple of months ago, Manuel Rosso, the CEO of Food on the Table came to me with a problem. He had a product with a great value proposition and thousands of passionate customers. That wasn't the problem. The problem was activation.

As a bit of background, Food on the Table helps people plan meals for their families around what is on sale in their local grocery stores. The team defined an activated user as someone who made it through all the steps of the first time user experience: selecting a grocery store, indicating food preferences, picking recipes, and printing a grocery list.

Users who made it through activation loved the product, but too many first time users were getting lost and never getting all the way to the end.

Identifying The Problem

More than any startup I've worked with, Food on the Table embraces the lean startup methodology. They release early and often. They get tons of feedback from their users. And, most importantly, they measure and a/b test absolutely everything.

Because of their dedication to metrics, they knew all the details of their registration funnel and subsequent user journey. This meant that they knew exactly how many people weren't finishing activation, and they knew that number was higher than they wanted.

Unfortunately, they fell into a trap that far too many startups fall into at some point: they tried to measure their way out of the problem. They would look at a metric, spot a problem, come up with an idea for how to fix it, release a change, and test it. But the needle wasn't moving.

After a couple of months, Manuel had a realization. The team had always been dedicated to listening to users. But as they added new features, their conversations with users had changed - they became more narrowly focused on new features and whether each individual change was usable and useful. Somewhere along the way, they'd stopped observing the entire user experience, from end to end. This didn't last very long - maybe a month or two, but it was long enough to cause problems.

As soon as he realized what had happened, Manuel went back to talking directly to users about their overall experiences rather than just doing targeted usability tests, and within a few hours he knew what had gone wrong. Even though the new features were great in isolation, they were making the overall interface too complicated. New users were simply getting lost on their way to activation.

Now that they knew generally why they were having the problem, Manuel decided he needed a designer to identify the exact pain points and come up with a way to simplify the interface without losing any of the features.

Key Takeaways:

• Don't try to measure your way out of a problem. Metrics do a great job of telling you what your problem is, but only listening to and observing your users can tell you why they're having trouble.
• When you're moving fast enough, a product can become confusing in a surprisingly short amount of time. Make sure you're regularly observing the user experience.
• Adding a new feature can be useful, but it can also clutter up an interface. Good design helps you offer more functionality with less complexity.

Getting an Overview of the Product

When I first came on board, the team had several different experiments going, including a couple of different competing flows. I needed to get a quick overview of the entire user experience in order to understand what was working and what wasn't.

Of course, the best way to do that is to watch new and current customers use the product. In the old days, I would have recruited test participants, brought them into an office, and run usability sessions. It would have taken a couple of weeks.

Not anymore! I scheduled UserTesting.com sessions, making sure that I got participants in all the main branches of the experiments. Within a few hours, I had a dozen 15 minute videos of people using the product. The entire process, including analysis, took about one full day.

Meanwhile, we set up several remote sessions with current users and used GoToMeeting to run fast observational sessions in order to understand the experience of active users. That took another day.

Key Takeaway: Get feedback fast. Online tools like GoToMeeting and UserTesting.com (and about a hundred others) can help you understand the real user experience quickly and cheaply.

Low Hanging Fruit

Once we had a good idea of the major pain points, we decided to split the design changes into two parts: fixing low hanging fruit and making larger, structural changes to the flow. Obviously, we weren't going to let engineering sit around on their hands while we made major design changes.

The most important reason to do this was that some of the biggest problems for users were easy to fix technically and could be accomplished with almost no design input whatsoever.
For example, in one unsuccessful branch of a test, users saw a button that would allow them to add a recipe to a meal plan. When user test participants within the office pressed the button, it would very quickly add the recipe to the meal plan, and users had no problem understanding it. When we observed users pressing the button on their own computers with normal home broadband connections, the button took a few seconds to register the click.

Of course, this meant that users would click the button over and over, since they were getting no feedback. When the script returned, the user would often have added the recipe to their meal plan several times, which wasn't what they meant to do.

This was, by all accounts, a bad user experience. Why wasn't it caught earlier?

Well, as is the case with all software companies, the computers and bandwidth in the office were much better than the typical user's setup, so nobody saw the problem until we watched actual users in their natural environments.

What was the fix? We put in a "wait" spinner and disabled the button while the script was processing. It took literally minutes to implement and delivered a statistically significant improvement in the performance of that branch of the experiment.

Giving immediate feedback drastically reduced user error

Manuel told me that, immediately after that experience, the team added a very old, slow computer to the office and recently caught a nasty problem that could add 40 seconds to page load times. Needless to say, all usability testing within the office is now done on the slowest machine.

Key Takeaways:

• Sometimes big user problems don't require big solutions.
• To truly understand what your user is experiencing, you have to understand the user's environment.
• Sometimes an entire branch of an experiment can be killed by one tiny bug. If your metrics are surprising, do some qualitative research to figure out why!

A Redesign

While the engineering team worked on the low-hanging fruit, we started the redesign. But we didn't just chuck everything out. We started from the current design and iterated. We identified a few critical areas that were making the experience confusing and fixed those.

For example, we started with the observation that people were doing ok for the first couple of screens, but then they were getting confused about what they were supposed to do next. A simple "Step" counter at the top of each page and very clear, obvious "Next" and "Back" buttons told users where they were and what they should do next.

Users also claimed to want more freedom to select their recipes, but they were quickly overwhelmed by the enormous number of options, so we put in a simple and engaging way to select from recommended recipes while still allowing users to access the full collection with the click of one button.

Users were confused by how to change their meal plan

Recommended recipe carousels made choosing a meal plan fun and easy to understand

One common problem was that users asked for a couple of features that were actually already in the product. The features themselves were very useful and well-designed; they just weren't discoverable enough. By changing the location of these features, we made them more obvious to people.

Most importantly, we didn't just jump to Photoshop mockups of the design. Instead, we created several early sketches before moving to interactive wireframes, which we tested and iterated on with current users. In this case, I created the interactive wireframes in HTML and JavaScript. While they were all grayscale with no visual design, they worked. Users could perform the most important actions in them, like stepping through the application, adding meals to their meal plan, and editing recipes. This made participants feel like they were using an actual product so that they could comment not just on the look and feel but on the actual interactions.

By the end of the iterations and tests, every single one of the users liked the new version better than the old, and we had a very good idea why.

Did we make it perfect? No. Perfection takes an awful lot of time and too often fails to be perfect for the intended users.

Instead, we identified several areas we'd like to optimize and iterate on going forward. But we also decided that it was better to release a very good version and continue improving it, rather than aim for absolute perfection and never get it out the door.

The redesign removed all of the major pain points that we'd identified in the testing and created a much simpler, more engaging interface that would allow the team to add features going forward. It improved the user experience and set the stage for lots more iteration and experimentation in the future. In fact, the team currently has several more exciting experiments running!

Key Takeaways:

• Interactive prototypes and iterative testing let you improve the design quickly before you ever get to the coding stage.
• Targeting only the confusing parts of the interface for redesign reduces the number of things you need to rebuild and helps make both design and development faster.
• Lean design is about improving the user experience iteratively! Fixing the biggest user problems first means getting an improved experience to users quickly and optimizing later based on feedback and metrics.

The Metrics

Like any good lean startup, we released the new design in an a/b test with new users. We had a feeling it would be better, but we needed to know whether we were right. We also wanted to make sure there weren't any small problems we'd overlooked that might have big consequences.

After running for about 6 weeks and a few thousand people, we had our statistically significant answer: a 77% increase in the number of new users who were making it all the way through activation.

My entire involvement with the project to do the research, design, and usability testing was just under 90 hours spread over about 6 weeks.

Key Takeaway: Design - even major redesigns - can be part of an agile, lean startup environment, if done in an efficient way with a lot of iteration and customer involvement.



Laura Klein has been working in Silicon Valley as both an engineer and a UX professional for the last 15 years. She currently consults with lean startups to help them make their products easier to use. She frequently blogs about design, usability, metrics, and product management at Users Know. You can follow her on Twitter at @lauraklein.

Why we need to teach MBA’s about modern entrepreneurship (and what Harvard Business School is doing about it)

This week, the startup tribe from Harvard Business School is making their annual trek to Silicon Valley. They’ll hear from a variety of experts and get to see many startups firsthand. While they’re here, I’m sure they’ll be received warmly. But I don’t think they’d be too happy to hear what gets said about them when they leave the room.

It’s a common refrain around Silicon Valley to disparage the role of MBA’s in entrepreneurship. We still have some collective scar tissue: the idea conjures up the hordes of dot-com hopefuls that descended on VC’s and angel investors with little more than a business plan. Even today, I routinely hear MBA’s advised to remove the degree from their resume when applying to startup jobs. We also cavalierly lampoon the “suits” that get brought in to run startups after the founders are fired by callous VC’s. I used to call them “executioners” because their attempts to “execute” the standard general management playbook in the startup context of extreme uncertainty usually led to disaster.

But this anti-MBA bias harms the entrepreneurship ecosystem and limits opportunities even for startups that don’t employ a single MBA.

I spend a lot of time thinking and writing about what a theory of entrepreneurship needs to do. One of my beliefs is that such a theory should be addressed to entrepreneurs and the people who hold them accountable. It’s this latter criteria that I think tends to get overlooked in most writing about entrepreneurs.

Who holds entrepreneurs accountable? One obvious answer is startup investors: angels and venture capitalists - many of whom have MBA’s. They have the ultimate responsibility in a venture-backed company of deciding whether to fire the founders and bring in “professional management.” I believe one of the reasons that this often goes badly is that the investors have nothing more than standard general management tools for evaluating the startup’s success. For example, when a startup is making more money month after month, I often hear VC’s say something to the effect of, “well, you can’t argue with success.” I hear similar things for pre-revenue startups that are on schedule, on time, and on budget - even though they are busy building something that nobody wants. (In fact, this crisis was at the heart of Steve Blank’s original impetus to develop customer development as an alternative set of milestones to use for startups.)

I also frequently see the reverse. General management is supposed to be orderly, “strategic” and mostly calm. I have seen founders replaced because their style seemed too chaotic, even though what’s really happening is that they are operating at startup speed. Pivots are disorienting, but necessary. Except when a startup is busy “pivoting” all the time, running around in circles. That's a waste of time. How do you tell the difference? General management doesn't have a good answer. As a result, founders get removed prematurely or even entirely exiled.

And there’s a worse fate, too. Far too many venture-backed companies correctly conclude that the founders are being counter-productive during their high-growth phase. But neither they nor the founders can think of anything for them to do: the MBA’s think they should be purged and the founders think they should be in charge. It rarely occurs to either party that the founders could be busy inventing the next disruptive product, but doing so in a non-disruptive way. As a result, many of these companies get caught by surprise when their optimization activities hit a plateau and competitors who have a true portfolio approach race past them. Facebook vs. Myspace, anyone?

These dynamics harm startups at all stages, because they pressure founders to engage in “success theatre” – trying to make themselves look successful by general management standards by focusing on vanity metrics, product milestones, and whiz-bang demos. In entrepreneur circles, this is goes by the innocuous-sounding term “board management” – but it is actually a terrible waste of energy and focus.

And it’s not just investors who are having trouble. The tech press loves to celebrate when startups get acquired, and founders make lots of money. But who do you think makes those acquisition decisions? In many cases, it’s MBA’s inside large companies. Far too many of these acquisitions go nowhere. Companies are acquired for hundreds of millions only to be worth tens of millions a few years later. I know plenty of people that have nothing but disdain for the “suits” who make these decisions. These entrepreneurs are only too happy to engage in success theatre, cash out, and move on to the next venture. What happens after they leave is of little concern. If it goes badly, we all know it’s the behemoth who will get the blame.

But that’s damaging, too. The core moral tenet of capitalism is that voluntary exchange is the ultimate test of whether an economic transaction is value creating or value destroying. Fraud, deception, and dishonesty undermine this moral calculus. Vanity metrics and success theater are in a moral grey area; they are masking the fact that some of our industry’s most “successful” ventures are actually value destroying. Even worse, this breeds tremendous mistrust.

Back in my IMVU days, I met with an investment banker who wanted to help us understand the M&A landscape. When we walked him through our business model and results, he explained that our options would be limited, because the main M&A community was feeling burned by companies like ours. He explained, “the big companies have bought a bunch companies with a client download or engagement business model, like Xfire and Neopets, and haven’t seen the returns they had hoped. Therefore, if you want to sell IMVU one day, you’ll need to abandon your business model, even though it’s generating a lot of revenue per customer. You’d be better off just selling ads.” There’s a lot wrong with statement, not the least of which is that “client download” is not a business model and that he was advocating that we abandon a real business model for an unprofitable one. But I believe he was representing his clients’ honest beliefs; they were confused about how to classify startups. When they get burned by a wave of acquisitions that turn out to be value destroying, the rest of us suffer the resulting lack of liquidity that is so important to the startup ecosystem.

It’s tempting to blame the MBA’s for all these mistakes. But I think the root cause of these mistakes is the fact that most MBA’s are not adequately educated about entrepreneurship. The problem afflicts general managers who try to innovate within big companies, too. In fact, I hope longtime readers will recognize these as the exact same mistakes that afflict us as entrepreneurs when we try to hold ourselves and our teams accountable. Are we making progress? Is what I’m working on creating value? What should I work on next? These are the enduring startup questions.

That’s why I am so excited about a new course that is debuting this year at Harvard Business School.

I’ll be honest. I get a lot of raised eyebrows here in Silicon Valley when people hear that I am an Entrepreneur in Residence at Harvard Business School – and not just because I continue to be physically resident here in San Francisco. Some people here are skeptical of spending time with MBA’s. One tweet read, “well, if HBS is investing in the lean startup we know it has jumped the shark.”

I’m glad to be able to do something about the divide between entrepreneurs and MBA's. As I’ve written before, academia has an important role to play in the new startup movement that’s afoot: documenting case studies, identifying emerging new practice, and doing original research to validate or refute theories. Harvard is, in some ways, a late entrant to this project; important work has taken place at Stanford in both the business and engineering schools; Berkeley's Haas School of Businesses was the place where Steve Blank first taught customer development.

Professor Tom Eisenmann is pioneering a novel approach at HBS, with a new class called Launching Technology Ventures:

Launching Technology Ventures uses case studies to examine lean startup practices. LTV focuses on the integration of marketing and engineering functions and emphasizes implementation rather than strategy formulation issues. The course does not examine financing options or the composition of founding teams. LTV draws heavily on the ideas of Eric Ries, Steve Blank, Marty Cagan and other practitioners.

The class debuts in a few weeks. I’m excited to be joining him for one of the first sessions on January 26 (for those who don’t attend HBS, I’ll also be doing a public event at the Boston Lean Startup Circle). Prof. Eisenmann has just published a series of blog posts describing the class, and they include three new resources that will benefit entrepreneurs and MBA’s everywhere:

1. A comprehensive reading list of the best sources on the new entrepreneurship: books, blogs, ebooks - everything. It’s the best such compilation I’ve ever seen. It should be considered the definitive (and mandatory) reading list for anyone who wants to understand modern entrepreneurship.
   
2. A series of new business cases documenting Lean Startup principles in a variety of companies. Many will be familiar to those who attended sllconf: IMVU, Dropbox, Aardvark. But he’s also documented cases of these principles applied in a variety of other situations. For example, for years Steve Blank and I have been saying things like, “if you’re working on a cure for cancer, these rules don’t apply.” But it turns out that this is not exactly true, as a company called Predictive Biosciences discovered. It’s a phenomenal case.
   
3. A compilation of tools and skills. The class also requires that the students explore – and master – some specific tools and techniques that entrepreneurs use every day. You cannot learn about entrepreneurship just at the blackboard. You have to get your hands dirty. This post outlines the skills that MBA’s who want to understand entrepreneurship have to master. For example, in my work with MBA’s who want to go into software entrepreneurship, I often encourage them to learn how to program. Managing a software company without knowing how to program is just as ridiculous as a factory manager who refuses to walk the factory floor. If you don’t understand how work is done, you cannot manage it. The same logic applies across a whole host of skills, which is why this compilation is so important - and why it'll be a long term project. (I'm hoping we'll get this hosted on a wiki soon.)

This class is just the beginning. If you’re an MBA student, I strongly encourage you take a class like this one – even if you’re not planning to become an entrepreneur yourself. First of all, if you pursue a career in general management, it’s extremely likely you’re going to find yourself managing entrepreneurs. Second of all, you never know – you might find yourself facing extreme uncertainty in your job, no matter what industry or company you work for. If you’re at Harvard or one of the schools where Steve Blank teaches, consider yourself lucky. If you’re not, advocate for your program to try something new.

And for my friends in Silicon Valley, I hope you get excited about this, too. Imagine a world where MBA’s are actually helpful and not just “suits” that get in our way. Wouldn’t that be a cool hack?

2011

Thank you, 2010.

I'm excited for 2011, and I want to share some of my plans for the coming year. But before I do, I also feel obligated to take a look back on my plans for last year.

I can't believe a whole year has passed since I wrote a blog post called "Towards a new entrepreneurship" laying out my priorities for 2010. Back then, I wrote:

"[Here is] an idea that I don't think is too widespread yet: that entrepreneurship is an industry. Sure, when entrepreneurs create startups that grow up into mature companies, they become part of an established industry, with its own ecosystem, norms, partners and best practices. But until that happens, we entrepreneurs have our own ecosystem, of investors and service providers, norms and even some "best" practices. The two ecosystems have diverged significantly in the past fifty years - and especially in the past ten. The reason is that the underlying theory that powers established business, the theory of general management, is increasingly inadequate for managing startups. And yet, so far, we lack a coherent theory to replace it. My belief is that the lean startup is that theory. Together, we are part of a movement that is redefining entrepreneurship."

This idea has become a reality in many ways this year. Our ideas have entered the mainstream of startup thinking and even the popular culture. I mean, who ever thought we'd see this cartoon in the New Yorker magazine?

Lean Startup Meetups are now in more than 75 cities, with more than 12,000 combined members. More and more, I am meeting entrepreneurs and managers from companies large and small who agree on this one point: entrepreneurship is management. By applying the same scientific principles that gave rise to general management in the first place to entrepreneurship and innovation, we are unleashing incredible creativity. But we still have a long way to go.

In that spirit, I want to review the four priorities I laid out at the start of this year. For 2010, I announced four main projects. Here's how I laid them out (in their original embarrassing order), and here's how each one turned out.

1. The Lean Startup Cohort program. Verdict: FAILURE. This seemed like such a promising idea at the time. Take a small number of high-growth companies and have them pay a premium price to learn from me and from each other how to apply Lean Startup ideas in depth. My main hypothesis was that making the program expensive would act as a quality filter, and that if we could find smart, committed companies to participate, they would all benefit tremendously. Thus, I assumed the biggest risk was finding participants who could afford the price.
   
   Unfortunately, I was completely wrong. Finding participants was no problem; the program quickly filled up. And the quality of participants was way higher than I imagined. And yet, when we actually started to run the program, it still failed. Teaching Lean Startup concepts in a fixed order really didn't work, since all active companies face different challenges at different times. And even in a strict, high-quality filtered room, most companies didn't want to share their problems and internal data, nor did they particularly want to engage with other companies' problems. In retrospect, that should have been obvious to me - as an entrepreneur, I would never have had the patience for a program like that.
   
   What's that you say? Even "gurus" have to get out of the building, build a minimum viable product, and pivot? Why, yes, they do. Embarrassing, but at least we failed fast. (Peter Drucker thought people used the term guru because it was easier to spell than charlatan.)
   
   
2. Teaching in academia. Verdict: MIXED. I started the year co-teaching a Lean Startup class for MBA's at Berkeley with Steve Blank. In some ways, it was a big success: the class was oversubscribed, had a record number of auditors, and received positive reviews. But the experience left me with doubts about whether that is the right way to engage with academia, for me.
   
   I strongly believe academia has an important role to play in transforming the practice of entrepreneurship. Luckily, Steve has been leading the charge to bring a new way of teaching entrepreneurship into academic programs, and 2010 saw the debut of his Durant School of Entrepreneurship at sllconf, as well as new programs like the Lean Launch Pad at Stanford and the Business Model Competition at BYU.
   
   I believe significant new research also needs to be done. What we know today is just the tip of the iceberg about this new entrepreneurial management. How many of our beliefs are just tactics that sound good, or that only work in certain situations? Much more is needed, and 2011 will see the first few buds of that research project flower. My colleagues at Harvard Business School will debut a new Lean Startup-themed course for MBAs this spring, as well as a new $50,000 Minimum Viable Product Fund. As part of that project, HBS has commissioned a series of new case studies on Lean Startup practices, both in and (importantly) outside the software industry. (You can see a little taste at Jeffrey Busgang's blog here.) I've also begun a collaboration with Nathan Furr at BYU to research actual practitioners, following them over time with an eye towards discovering ways to test some of our beliefs about Lean Startup ideas empirically. You can follow our work (and volunteer to be studied) here.
   
   Also along these lines, I've worked with a variety of collaborators to produce case studies right here on Startup Lessons Learned. Hopefully, more will come in the new year. You can see our efforts so far.
   
   
3. Startup Lessons Learned Conference (sllconf). Verdict: SUCCESS. This was a project I almost didn't do this year, because the prospect made me so nervous. Boy am I glad I did. I still receive regular feedback from people who were there live or in one of our 60+ simulcast locations around the world. It was always a dream of mine to produce a conference where knowledge - not hype - was king, where information was presented in a useful order, and where success theatre and vanity metrics were banned. I believe we succeeded on all three counts.
   
   In case you missed it, here's a little taste of the event itself, courtesy of my friends at Micro-Documentaries:
   
   
   
   And don't forget, you can watch full video of the entire conference courtesy of our sllconf Justin.tv channel.
   
   In 2011, we will do sllconf again, probably in mid-May. As always, I will look to you readers for guidance and suggestions of what we should do different. Stay tuned for details. If you are interested in speaking or mentoring at sllconf 2011, we will accept suggestions and nominations. If you would like to nominate someone, please post a video of them giving a talk (with slides if possible).  Grainy low-def youtube videos are perfectly adequate. We had far too many submissions last year on behalf people I didn't know. I had to be confident they would meet the standards I laid out above, but I couldn't take the time to meet them all. Therefore, if you'd like to speak this year at sllconf, a great way to get a leg up would be to speak at a Lean Startup Meetup, and ask someone to record the session. And if you are a meetup organizer, and have had a great speaker who you'd like to see at sllconf 2011, please let me know.
   
   In other conference news, we'll also have an event at SXSW. We'll make details available soon, I promise. If you're going to be in town for SXSW, and might like to join as a speaker, sponsor, or attendee, please let me know.
   
   
4. Writing a book. Verdict: TBD. I am in the final weeks of preparing a manuscript for The Lean Startup Book which will be published by Crown (one of the largest business book publishers in the world) in 2011. I sincerely hope you'll like the final result; it has been a labor of love for me all year.
   
   Deciding to publish this book through traditional channels took a lot of thought. I believe it is time for our movement to Cross the Chasm into mainstream awareness. Our early successes have been impressive, but we are still just at the beginning. In my talks all this year I have been exhorting audiences to Stop Wasting People's Time. Our modern economy is full to the brim of waste: building products that have few customers, that produce negative returns for investors, or companies stuck in the land of the living dead. And yet, the people who are responsible for this waste are not generally early adopters of new ideas about entrepreneurship. They are not scouring blogs for the latest gems in innovation thinking. They are overwhelmed, doing the best they can, and get information from only a few sources. They don't want avant garde advice, they want to be reading the same things everyone else is reading.
   
   My belief is that, in order to reach this mainstream audience, we need to produce a book that is accessible to them, and then make that book a bestseller. That's one of my main goals for 2011, and I will be asking you to help many, many times in the coming year. I hope you'll continue to support me as you have this past year.
   
   As a reader, the rational thing to do with a new book is to wait until the book comes out, see if your friends and colleagues read it, and if they do, see if they think it's any good. That's classic mainstream customer thinking. Hopefully, the early adopters and visionaries among you will disregard this advice, and agree to pre-order the book instead. The more of you who do that, the more people we'll be able to reach when it debuts next year. Remember, mainstream customers will be looking to you to see if it's worth buying.
   
   You can pre-order it from me directly, or get an even better price at Amazon.
   
   (If you'd like to help, I'm still looking for test readers, case studies, and - most importantly - help bringing traffic to the book website. We're running constant A/B tests there; anyone who is able to donate traffic, ads, or a link from your own blog/website will have my gratitude.)

So that was 2010. I believe 2011 will be even better.

It's an auspicious time. Entrepreneurship is in a new renaissance. There are more startups operating today than at any time in history. New ideas about entrepreneurship are in the air. And the dominant management paradigm of the past century has run its course. Literally.

2011 will mark the one hundredth anniversary of the idea of management. I date its origin to the publication, in 1911, of Frederick Winslow Taylor's The Principles of Scientific Management, one of the most important management books ever written. Management's second century will be very different than its first. Our problems are more complex, faster moving, and we face greater uncertainty. In other words, we need entrepreneurs to solve them. I'm excited to see what comes next.

I hope you've all had a happy holidays, and I wish you the best for a new and exhilarating New Year. Here's to 2011!

Why do we do this?

On April 1, 2009, I was as nervous as I have ever been in my life. It was just minutes before I was supposed to go on stage for the very first time and present The Lean Startup to a large audience, at a big conference. To that point, I’d been talking about Lean Startup concepts only on a seldom-read blog and with people in my immediate network. I had advised startups and VC’s, guest lectured a few time, and met with some small groups of extremely early adopters, like Sean Murphy’s Bootstrapper’s Breakfast. This was different.

I’d had plenty of public speaking experience in my life. This was not the largest audience I’d ever spoken in front of, and I’ve since stood before much larger. I had pitched startups and products, raised money, and lived through some tough negotiations. But this time, I was presenting an idea, not a product. And I was presenting on my own behalf, not on behalf of a company, risking ridicule and hoping for acceptance.

Don’t think I wasn’t prepared. This is a Lean Startup talk we’re speaking of, people. I had a Customer Advisory Board, made up of the type of people who attended this conference, to give me feedback on my talk and slides. I had beta tested the talk with a smaller audience of entrepreneurs at Stanford. And I had been testing variations on the underlying stories on just about anyone who would listen.

My anxiety stemmed from a question that kept occurring to me in the hours before I walked on stage: why am I doing this? For some people, getting up on stage is the most natural thing in the world. Not me. Some people love going to conferences, mixers, and summits. Not me. Some people thrive in the “startup scene.” Not me. When I was a real live practicing entrepreneur, I never had the time or the desire for that stuff. And yet, here I was, about to try and convince nearly a thousand people that I had something valuable to say about entrepreneurship. Why?

It was a defining moment for me. I sought a quiet spot, away from the stage and the crowd. I asked myself why over and over. Startups should be more successful, I answered. Why do I care? Because preventable failures waste time and money. Why do I care? Because entrepreneurs are following “best practices” that don’t work for them. Why do I care? Because I was once one of those failed entrepreneurs. But I'm past that failure now, why do I still care? Because it doesn’t have to be that way.

I remembered a specific moment from my very first startup. It was the moment I realized my company was going to fail. My cofounder and I were at our wits’ end. The dot-com bubble had crashed, and we had spent all of our money. We were trying desperately to raise more, and we could not. The scene was perfect: it was raining, we were arguing in the street. We literally couldn’t agree on where to walk next, and so we parted, in anger, heading in opposite directions. As a metaphor for our company's failure, this image of the two of us, lost in the rain and drifting apart, is perfect.

It remains a painful memory. We had begun as friends, and ended as enemies. The company limped along for months after, but our situation was hopeless. Looking back, I know our failure was inevitable, because we had no clue. It seemed we were doing everything right: we had a great product, a brilliant team, amazing technology, and the right idea at the right time. And, as I’ve mentioned previously, we really were on to something. We were building a way for college kids to create online profiles for the purpose of sharing… with employers. Oops. But despite a promising idea, we were nonetheless doomed from day one, because we did not know the process we would need to use to turn our product insights into a great company.

If you’ve never experienced a failure like this, it is hard to describe the feeling. It’s as if the world is falling out from under you. You realize you’ve been duped: the stories in the magazines are lies, hard work and perseverance don’t lead to success, and – worst of all – the many, many, many promises you’ve made to employees, friends, and family are not going to come true. Everyone who said you were an idiot to do this will be proved right.

Looking back, the idea that this failure was preventable makes me ill. That is the memory I conjured up right before going on stage that April 1st. And I thought, nobody should ever, ever, ever have to go through that. If I can reach just one entrepreneur and help them find a different path, that will be a success. If I get laughed off stage, if I never give another talk, if nobody ever reads my blog, none of that will matter if, in the back of the room, there’s just one person who can use what I have to offer to save their dream, their vision, their startup.

That was the mission that carried me onstage. It helped me stay calm in the face of the unexpectedly large crowd and the incredible response that followed. That day, I had absolutely no idea what would come later – that Lean Startup would become a movement, that it would take over my life, that I'd be writing a book about it, that you would be reading my words today. (You can read my original post-conference report here, including a scratchy iPhone recording of the talk itself.)

All of this is by way of saying, thank you. You are the reason I took those first steps. You are the reason I find this work meaningful. And you keep me going whenever I despair of being able to figure out what to do next.

And there's been plenty of despair, lately. I've been mostly absent from the blog and keeping a much reduced public schedule, as all my energy is devoted to the book. Writing a book is a slog, in the same way that writing a large piece of software is. There are many parts, they all depend on each other, and none is at all valuable unless the whole comes together with high quality. Naturally, quality is exclusively in the eye of the beholder. So, while I'm writing, I can never be quite sure if I've hit the mark. It's been hard, but I am optimistic. This project has allowed me to meet so many entrepreneurs who are building extraordinary organizations. It is their efforts which form the backbone of the Lean Startup movement. The world is going to hear a lot about it next year. And the mission I discovered as a nervous wreck in 2009 continues to motivate me to keep going.

I am grateful to you entrepreneurs, who test ideas in the crucible of daily practice. You are making incredible things happen. You are changing the world. I wish you a happy Thanksgiving.

Case Study: Rapid iteration with hardware

(I am often asked to explain how to apply Lean Startup approaches to domains beyond software. In order to answer, I have taken to drawing a two-axis diagram. 

On one axis we have the degree of market uncertainty for a given industry. For "cure for cancer" type businesses, there is no question about who the customer is and what the customer wants, and therefore there is no market uncertainty. On the other extreme, modern web-based applications face almost no technical risk, and are governed by high market uncertainty.

On the other axis we have the underlying cycle time of the industry in question. Slow-moving cycles, like drug discovery or new automobile models, govern the slow part of the axis. On the extreme opposite end are rapid iteration businesses like software or fashion.

The key to understanding Lean Startup is to recognize two things:

1. Lean Startup techniques confer maximum benefit in the upper-right quadrant, namely high market uncertainty coupled with fast cycle time.
2. Every industry on Earth is currently undergoing a disruption that is causing it to move along both axes: more uncertainty and faster cycle times.

I am aware of no industries that are moving "backwards" on either dimension. Thus, more and more industries are starting to look like the software business. Of course, the underlying root cause of this worldwide disruption is the software and semiconductor revolution. Industries are disrupted as their traditional work process is "infected" by software. And, as a result, more and more companies are able to benefit from Lean Startup practices. 

The following case study looks at one such industry, consumer electronics, where the pace of iteration has taken a marked turn towards high speed. It is written by Ronald Mannak, who is currently the CEO of a startup named Yobble. What follows are solely his opinions. -Eric)

In a bar in Amsterdam in 2005, my two cofounders and I came to the sad conclusion that startup we tried to built for two years was doomed. In 2003 we started developing a martial arts motion sensing toy, a full three years before the Nintendo Wii changed the world of motion sensing. The toy (we called it Ninja Master) consisted of two hardware units, attached to both wrists. When a child would perform a perfect karate move (or better yet: a combo of several karate moves in a row), Bruce Lee-like karate sounds would emerge from a small speaker in the device. We loved the product. Test users loved the product. It was way ahead of its time. We thought we were visionaries and believed the future was motion control. Yet, we failed to sell the toy. We talked to every toy company imaginable, but none wanted to license our toy. " Kids nowadays don't want to move, they play Playstation" was the most often heard reply, even though our user tests suggested otherwise. To make matters worse, we lived in a country (Holland) without a proper functioning startup, VC and angel ecosystem. The company was doomed. My co-founders decided startup life wasn't for them.

However, one new idea emerged at that meeting. What if we could make an air drum? Drum sticks with sensors in them. Now that was an idea. Music is much easier to sell (to toy companies) than the abstract martial arts Ninja Master toy. Besides, we could easily expand the line with with an air guitar and a device to link the air instruments to a PC. How cool. I loved the idea so much that I decided to pursue the idea.

I envisioned the product would be popular with 8 to 12 year old boys. I thought the price couldn't be higher than $40. I already knew how the product would be used. Boy, was I wrong.

Waterfall
I previously worked on a couple of IT projects that used the 'waterfall model' where specifications were written down by one team, thrown over an imaginary wall and implemented by another team. Every single waterfall project I encountered turned out to be a disaster in every way. Specifications turned out to be open to multiple interpretations, usability was the last priority (if a priority at all). It Just Did Not Work. As a beta tester of the first Borland Delphi, I learned the wonders of rapid prototyping and fast iterations. I wondered if we could do the same for hardware development. It turned out we indeed could.

The first hire
The first hire was critical. I wanted somebody who was creative first and technical secondly. I found the perfect person at the department of Industrial Design Engineering of the Delft University of Technology. Joris. Joris was creative and eager to learn. Better yet, he plays drums. Even better than that: he likes to tinker with electronics. Hiring him was a no brainer, and he didn't disappoint.

The internship only lasted six months. That's not much time, considering the scope of the project. I convinced the university that Joris should not be writing specifications and other nonsense first, but start right away building prototypes. And he did.

Joris suggested that before he started working on electronics, we should invite children, give them wooden drum sticks and let them pretend they were playing air drums. It turned out to be an excellent idea. Children are perfect test subjects. To our surprise, every single child did something we didn't anticipate. Without any exception, they all whacked the wooden drum sticks *sideways* and made 'crashing sounds'. I certainly didn't think of sideways movements when I created the first ideas, but apparently it was a good idea to implement.

The prototypes
The next day we started building the first prototype to see if the sensors actually behaved like they were supposed to, and to see if we could measure the sideway movements. The prototype was crude. Joris taped sensors on his arms with duct tape and started drumming in the air with wooden drum sticks (that did not contain any electronics). We connected the sensors to a seven year old pc with an Arduino-like interface that ran a simple drum program we developed. The results were amazing. It actually worked. (A video of the first prototype can be found here.)

We now knew what kids liked and we knew the product was technically feasible. Yet, I still felt we didn't know all we needed to know and wanted to test more. And I'm glad we did.

For the next prototypes we placed the sensors in PVC pipes to optimize sensor angles and added features to the pc software.

We made another discovery we did not anticipate. We found out that parents (who came along with their children for user testing) often liked the prototype as much as their kids. We decided to interview the parents and quickly found out that the parents who like our product were video games players. Of course we liked our product, but we never would have guessed other grown ups would like our product too. Knowing this, we invited test users from 12 to 30. They also loved the prototypes. Our target audience just exploded in size. We decided to make a few changes that would make the product less 'toy' and more 'gadget'.

Over a period of six months, we made eight generations of prototypes, each version adding more features and making the product more reliable. By testing each generation, we learned that a lot of hypotheses were correct, but a large number of hypotheses were incorrect. By testing early and often, we were able to adjust the product. I believe that we demonstrated that it is indeed possible to iterate fast and often with hardware development.

Product Launch
After some financing-related delays, the products went on sale in Europe and Asia in the summer of 2008. The retail selling price was $40, exactly what we targeted. In less than six months, we sold over 90,000 units. All shops sold out our products two months before christmas, all without spending one penny on marketing. The products were voted 'best music gadget' on television program The Gadget Show, became the best selling music toys on Amazon.co.uk and the best selling products on Firebox.com. Best of all, users love the products. On Firebox.com, the average user rating (740 users) is 4.5 out of 5 stars (link). We couldn't have been happier.

Post mortem
We demonstrated it is possible to iterate often and fast. I believe a lot of the product's success can be attributed to the iterative development process. We didn't find every issue though. We didn't test the price and we didn't see the Nintendo Wii or Guitar Hero coming. We chose to enter the market though the low margin toy market, where (in hindsight) we should have positioned the products as video games with higher video games margins.

Another thing we missed: after we launched we received many requests to add double bass drum, as often used in metal. The drums include two drum pedals and a double bass drum could have been added by a simple and minor change to the embedded software. However, updating the embedded software in sold devices isn't possible with the microcontroller we used. We could have included the feature in a 1.1 version of the product, but the toy manufacturer we licensed the toy to, wasn't interested in a new version, as the original version continues to sell well to this day.

Tools to develop hardware get better and cheaper. Open source projects like Arduino and SuperCollider make iterative hardware development cheaper and faster than ever. We learned that connecting the prototypes to PCs and user the PC to run the program is a very good way to test hardware (developing on a PC is still much faster than embedded developing on a standalone hardware device).

In the summer of this year, I moved to San Francisco and founded a new startup that makes music related games and hardware controllers that connect to the iPhone. There are a lot of new opportunities. New cheap flashable micro controllers make firmware updates possible for low cost hardware. With hardware connected to the internet (in our case through the iPhone) it should be possible to use continuous deployment: small and very frequent updates of the firmware instead of less frequent large updates. Bugs in firmware could be fixed within minutes or hours instead of weeks or months.

(Continuous deployment of hardware is an exciting new capability. In addition to continuous deployment of firmware via the Internet, it is also possible to do continuous deployment by taking advantage of a small-batch production process. When the complete cycle time of assembly is low and the design can be specified mostly through software, it's conceivable that each batch rolling off the line could have a different design. -Eric)

As a final thought, I am convinced iterative design depends mostly on the mindset of the team and the company culture and less on the tools. I was lucky to have a great team of A-players that were willing to take responsibility and risk. If the company culture is such that mistakes are punished, I am pretty sure iterative development won't work.

The Lean Startup Bundle

People often ask me if they can buy "Lean Startup in a box" from me. It's a strange thing, to be asked by a potential customer if they can give you copious amounts of money, and then to have to refuse. Startups run into this problem all the time: not every possible way of making money is equally useful. On the other hand, I try to keep in mind the idea that feedback is always about them and never about you. I recognize the need being expressed by this request. People want to get started with Lean Startup but aren't exactly how. If you're in that boat, today my friends at Appsumo are trying a new experiment that just might help you out.

It's called the "The Lean Startup Bundle" and it is available this week only. It is the ultimate product development get-started guide to Lean Startup. We did our best to package as much content (including almsot a dozen books and ebooks) as well as tools and apps, into one low-priced magic offer. Naturally, the bundle includes every essay I have written for this blog, as part of the Startup Lessons Learned ebook series.

But I am even more excited to report that it will include a hardcover copy of my new Lean Startup book. That's right, I am writing an old-fashioned, traditionally-published book full of all new material. Because of the traditional publishing industry's commitment to rapid iteration, the book won't come out until Fall, 2011. But if you buy The Lean Startup Bundle this week, you'll get a pre-order of the book included. That means you will get a hardcover copy of the book in the mail when it is eventually published. (Believe me, you will be hearing a lot more about this book over the course of the next year, so that's all I will say about it right now.)

But that's not all! I'm even more excited to announce two brand-new ebooks from two of my favorite bloggers: Sean Ellis and Andrew Chen. These ebooks were created by my friends at Leanpub especially for this bundle, and include the best essays written by both authors (and a foreword from yours truly). Also included is the incredible Venture Hacks Bible, which is worth the price of admission on its own, and The Entrepreneur's Guide to Customer Development (which I previously reviewed here).

And there's so much more. For two awesome books which have been traditionally published, we have several ebook chapters: Inbound Marketing by Dharmesh Shah, Do More Faster: TechStars Lessons To Accelerate Your Startup by Brad Feld & David Cohen. And also making its debut, you'll get a sample of Ash Maurya's forthcoming ebook Getting Lean.

"But," I hear you say, "so far this sounds like just a bunch of gurus trying to sell me books." Indeed, but - as they say - that is not all. We also reached out to companies that make products useful to accelerating your Build-Measure-Learn feedback loop, from cloud-hosted Selenium provider Sauce Labs (disclaimer: where I have equity) to ScrumPad (where I don't) and many in-between (a sampling is below, but you should really just click through to see the whole smorgasbord). (And even if you don't buy this bundle, one beneficial side-effect of my research for it is this cool list of Lean Startup tools curated by twtpick.in. Because of the short timeline in putting this all together, we couldn't include them all. But, if this bundle is successful, perhaps we'll have a chance to do another.)

Phew, I'm exhausted. "But," you are sure to ask, "how much will all this bundling cost me?" How about $1000? $500? $250? No! All of those prices are much too high. "Ok," you surely say, "how about $100?" Absolutely not. "$75?" No. "$65?" Absurd.

No, all of those prices are incorrect. If you act now, you will pay not one dollar more than $42. Don't panic. Buy now.

Stop lying on stage

Entrepreneurs crave information about successful startups, and they should. Most of the received wisdom about business and entrepreneurship is simply wrong. Many journalists and conference organizers attempt to fill this demand by giving successful entrepreneurs the opportunity to tell their stories: in magazines, on blogs, and on stage. And yet, most of the time, those opportunities are wasted, because the protagonists tell lies. And while this may sound like a harmless phenomenon – after all, most of these are simple lies of omission – I think the consequences are quite harmful.

I know the word “lie” sounds harsh. But I think our industry has to face up to this unpleasant reality. Luckily, some people have started to collect evidence of misleading behavior on the part of speakers. Paul Graham wrote this just the other day:

I didn't consciously realize how much speakers at more public events censored themselves till I was able to compare the same people speaking off the record at YC dinners and on the record at Startup School. YC dinner talks are much more useful, because the details people omit in more public talks tend to be the most interesting parts of their stories. About half the interesting things I know about famous startups, I learned at YC dinners.

I commend Paul for his honesty, something I have always admired about him. Paul has been – intentionally or unintentionally – running a perfect science experiment to answer the question: do startup speakers tell the whole truth on stage? His results are the same as I’ve observed on many occasions: the simple answer is no. I think everyone who’s been around these events long enough knows this to be true. None of the really successful people I know take what’s said at public events seriously. This is the same issue we see with vanity metrics: companies are giving the appearance of sharing information while actually engaging in spin or outright deception.

I call this the vanity ratio: the amount of apparently interesting information given divided by the amount of useful information contained therein. The higher the vanity ratio, the more effective the PR. Unfortunately – also – the more misleading the story is as a help to others.

And it’s not as if the stories we hear about entrepreneurs are biased in a random way. Paul quotes one of his founders like this: “That's the actual beauty in the off-the-record-ness: you hear just how screwed up most of these successful startups were on the way up.” In my experience, the official stories are always more linear, make the founders and investors look smarter, and dramatically overstate the level of certainty everyone had at every stage of the process. Failures, pivots, and crappy minimum viable products are generally elided. And the kinds of failures that do get airtime are usually failures to adequately plan, anticipate, or design in advance. So, naturally, the kinds of inferences we make from these stories are: we need heroic entrepreneurs, with absolute certainty in a brilliant idea, and we need to plan and execute well.

I call this the mythological-industrial complex, because it serves the interests of many players in preserving the status quo. It sells newspapers and magazines. It helps investors boost their profile and convince entrepreneurs that they offer value-add. It helps companies with PR. It makes successful founders famous. I've certainly been a beneficiary of these forces. Yet I worry that it deters new entrants from disrupting incumbents: if your idea looks a little dubious, your career a little messy, your team a little dysfunctional, if you lack superhuman design skills, maybe you should just give up. You don’t have the Right Stuff to become an entrepreneur.

The reason this is harmful is that it benefits another kind of person even more: the startup advisor. Yup, that’s me. When all of the stories floating around are bogus, there is a huge opportunity for people to claim to have arcane or esoteric knowledge. Paul does it right in his essay: “About half the interesting things I know about famous startups, I learned at YC dinners.” Doesn’t that sound good? Don’t you believe that having that special knowledge would help Paul give you better advice? I emphatically do. Most of my mentors have had this arcane knowledge, and I have benefited from it immensely. They know how the game is really played. They know what really happened in those pivotal moments that shaped famous companies. And I have done my best to pay it forward, by sharing that arcane knowledge with as many startups as I can.

However, the problem is this: how can you tell if someone who claims to have arcane knowledge is telling you the truth? After all, the fact that what they advise you contradicts all public sources of knowledge is part of the pitch. And, of course, it sounds good. All talking heads (including me!) face an overwhelming incentive to say whatever it is that will sound good to their audience, regardless of whether it is true. Believe me, it is hard to resist (as far as I can tell, that’s why cable news is so awful: pretty much everyone on cable news is giving in to this temptation every day). But I have met many startups that are making key decisions based on arcane knowledge that is simply not true. It is extremely hard to help them, because they are following voodoo advice that is nearly impossible to disprove. In fact, in some of my early entrepreneurial ventures, I was taken in by people just like that.

I have tried really hard to hold myself to a high standard in all of my public presentations: to give the unvarnished truth about what it was really like to succeed and fail. I do my best and yet I probably fail sometimes, too. In fact, I rely on readers and attendees to ask me the hard questions that challenge me to root out and eradicate these errors. And this was an absolute requirement of the speakers at sllconf. The reason we had such a non-diverse lineup (much to my chagrin) is that I insisted on hosting speakers whose stories I already knew – because I had been there, as a friend, advisor, or investor. Thus, I knew they were telling the truth and I knew they shared a commitment to speak with integrity even when it’s uncomfortable. This is also why I get so outraged when people treat Lean Startup like a religion. The whole point of transforming advice into the form of a testable theory is to allow people to evaluate whether it works. You can test theories like Lean Startup in your own practice, and discover if they deliver the outcomes they predict – and you can do it in small batches.

That’s why I find lying on stage so upsetting. By misleading future entrepreneurs, we put them in an untenable position. Either take the stuff you read and hear at face value or gain access to a wizard who can guide you to the true path (and hope you don’t get taken in). But this false choice is not the only way forward. As usual, I think there’s an opportunity for our industry to do better.

To be clear, I don’t blame any particular actor in the system for doing what they’re doing. Paul Graham is getting his startups the best advice he can in the best way he knows: by giving them exclusive access to off-the-record conversations that nobody else is privy to. And it’s not reasonable to blame him for their behavior when he gives them the stage at events like Startup School. Similarly, journalists print vanity metrics because that’s all companies will release. And companies crave positive PR and control over their message – all for rational reasons.

And yet you who are reading this right now have tremendous power. You are the intended audience for all of those expenditures of energy. You are the “hits” that websites crave, the followers and the RSS subscribers. Where you put your attention and what you do with it matters.

So I’d like to suggest the following: let’s stop giving lying on stage and vanity metrics a free pass. I think if we can delegitimize this behavior, we can make it stop. We’ll need to ask tougher questions, though: how do you know your company’s success was caused by X? what else was happening around that time that might have caused Y? what did you think was going to happen when you did Z?

Journalists have the highest obligation to ask these kinds of questions, and conferences organized by journalists ought to be the exemplars the rest of us look up to. And yet, today many are not. I am sympathetic, because I have faced this problem, too. If you ask tough questions, are unwilling to help people craft their message, and are skeptical of vanity metrics, you can’t get the high profile guests. That means less attention, less coverage, fewer readers, and lower sponsorship dollars. Assuming, of course, that all of you give your attention disproportionately to famous people.

But if you’re attending a conference or reading a magazine, you aren’t bound by those rules. You can ask any question you want. And you can reward the speakers who tell the whole truth with your support.  Every time you do that, you’re helping make our industry a better place.

(Have a favorite real startup story? Share it in the comments and we can start giving those people some much-deserved attention right away.)

Good enough never is (or is it?)

One of the sayings I hear from talented managers in product development is, “good enough never is.” It’s inspirational, always calling the team to try harder and do better. It works to undermine excuses for poor or shoddy work. And, most importantly, it helps team members develop the courage to stand up for these values in stressful situations. Especially in teams that are managing by objectives (or OKR's), the pressure to deliver is intense. Under such pressure, the temptation to cut corners, to quit prematurely, or to hand off shoddy work to another department is overwhelming. It requires courage to stand up and say: "this work is simply not good enough. Sure, we could get away with it, but that's not how we work." Good managers work hard to create an environment where this courage thrives.

On the other hand, there are many stories of companies achieving a breakthrough by shipping something that was only "good enough." One such rumor, which I’ve heard from several sources, tells of the launch of Google Maps. The team was demoing their AJAX-powered map solution, the first of its kind, to senior management at Google. They were impressed, even though the team considered it still an early prototype. Larry and Sergey, so the legend goes, simply said: “it is already good enough. Ship it.” The team complied, despite their reservations and fear. And the rest is history: Google Maps was a huge success. This success was aided by the fact that it did just one thing extremely well – its lack of extra features emphasized its differentiation. Shipping sooner accentuated this difference, and it took competitors a long time to catch up.

So which is it? Is "good enough" good enough? Rules of thumb can be infuriatingly unhelpful. When should you settle for good enough and when should you push yourself to do your best?

This is precisely the dilemma that the doctrine of minimum viable product is designed to solve. And it’s really hard.

Most of us intuitively have a “split the difference” attitude when faced with recurring difficult choices. That is not a long-term solution. The reason: it actively encourages factional strife. Everyone naturally falls along a spectrum, from “ship anything soonest” to “always build it right, no matter what it takes.” When members of a team realize that the final answer will be some kind of average, they face an overwhelming incentive to express desires in the strongest possible terms. After all, someone else’s view will be averaged in, too. Any excesses are likely to be moderated by others. Of course, this logic applies to members of all factions. Over time, such teams either explode due to irreconcilable differences or dramatically slow down. The latter is actually more dangerous. Divided teams usually can’t agree on facts or interpretations. Yet startups rely on collective learning in order to find their way. Factional strife is learning kryptonite. I believe this is one reason why the myth of the dictatorial startup founder has such enduring appeal. Faced with these kinds of disagreements, strong arbitrary action is much superior to paralysis.

But action/paralysis are not the only options. As in many false dichotomies, we can find a third way that gives both factions a positive message to rally around.

Without an affirmative message, managers can cause lasting harm. I certainly have. When people start using quality, reliability, or design as an excuse to delay, it used to make me nervous, even when these suggestions were well intentioned. After all, how would Craig Newmark’s life (and the rest of ours, too) be different today if he had waited to build something with a high-quality design before starting his famous list? Rather than having this repeated argument, I sometimes found it easier to play dictator on the other side, forcing teams to ship sooner than they were comfortable with. As I found out to my dismay, this is a dangerous game: in many cases, you’re asking trained professionals to violate their own code of best practices, for the good of the company. Once you go down that road, you risk opening a Pandora’s box of possible bad behaviors. And yet, it does not have to be that way.

Almost everything we know today about how to build quality products in traditional management has its origins with W. Edwards Deming, the original quality guru. He had two concepts that are especially important to this discussion. The first is that “best efforts are not enough.” Despite what it seems in the moment, most quality problems are not caused by people slacking off or acting maliciously. (It seems that way only because of a psychological phenomenon called the fundamental attribution error.) In reality, most quality problems are systemic in nature. They have to be solved in the boardroom by making a company-wide commitment to building quality into the very systems the company uses to build products. Lean manufacturing, agile software development, and Theory of Constraints are all examples of this idea in action.

However, a commitment to quality alone is not enough. In old school manufacturing, quality was defined as reliability: parts and products that did not wear out, break down, or fail unexpectedly. And so Deming’s contribution was especially prescient, as he saw that “the customer is the most important part of the production line.” This means that quality is defined in the eye of the customer, not necessarily by arbitrary standards loved by insiders to the production process. In today’s world, this is increasingly important, as quality is often defined by factors beyond reliability: design, ease of use, aesthetic appeal, and convenience.

Now we come to the heart of the minimum viable product issue: how can we build quality in if we do not yet know who the customer is? All of our professional standards that lead us to want to get it right the first time – all of them were developed originally in a non-startup context, one where the customer was known in advance. Startups are different, leading to this axiom: if you do not know who the customer is, you do not know what quality is.

Which takes us right back to the original definition of minimum viable product:

the minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning with the least effort.

In other words, the minimum viable product is a test of a specific set of hypotheses, with a goal of proving or disproving them as quickly as possible. One of the most important of these hypotheses is always: what will the customer care about? How will they define quality?

One common worry is that this might lead companies to “release crap,” shipping too soon with a product of such low quality that it alienates potential customers and, thus,  causes entrepreneurs to abandon their vision. This critique combines two misunderstandings in one.

First, I want to explore the idea of releasing crap: that our product is of such low quality that we will release it, customers will hate it, and we’ll have accomplished nothing but alienating them. But notice how many hypotheses are baked into this supposedly simple scenario: we believe we have already solved the distribution problem for our product (or else how could customers try it?). We already know who to distribute the product to (or else why would we care what they think?). Naturally, we already know the standard of quality that they will use to judge our product. And, of course, we already know that they will care enough to be offended. In fact, we know so much that we already know what they will care enough about (namely, the product’s quality – as opposed to, say, missing features).

Even better, this is a falsifiable hypothesis. It is entirely possible that we can ship “crap” and have one of the aforementioned facts fail to materialize. In fact, that is one of the best possible outcomes, because it will force us to learn something. What if customers actually like the “crap” product? Or what if we can’t get any of them to even try it? Or what if the features they demand we build are different from the ones we were planning to build? In those cases, we can’t help but learn a great deal. Remember, the minimum in minimum viable product does not mean that you should ship just anything at the nearest possible date. It means to ship as soon as it is possible to learn what you need to learn.

The second misunderstanding is a concern for what will happen if things turn out exactly as we originally predicted (namely, badly). Entrepreneurs, faced with an early defeat, might lose their commitment to seeing their vision through. I understand this fear. It is a direct consequence of the reality distortion field, that ability most visionaries have to get people to believe in a vision as if it was already true. Data can undermine this field. It's easier to believe in a glorious future when you have only zeroes, for everyone: founders, investors, and employees.

But this fear is way overblown, in my experience. The great visionaries I’ve worked with can incorporate a commitment to iteration into their process. However, there are some important ground rules. As I wrote in Don’t Launch, it’s essential to remember that these early minimum viable product launches are not marketing launches. No press should be allowed. No vanity metrics should be looked at. If there are investors involved, they should be fully briefed on the expectation that these early efforts are designed to fail.

Again, even if they do "fail," it is improbable that they will fail in the way we originally expected. In fact, in all of the startups I have worked with, I have never seen this happen. There is always something unexpected when customers react to a product in the real world: we thought they’d be offended by low quality, but actually they refused to download it; we thought they’d share it with their friends, but actually they wanted us to provide the friends; we thought they’d care a lot about our beautiful design, but actually they wanted more features. As in any experiment, the important thing is not the bare fact that the hypothesis was invalidated. More important is to understand the reasons why. This is not an academic exercise; the goal of these experiments is to immediately get up off the mat and design the next one. And the next, and the next, until we have not just learned but proved our learning with hard facts: through the attainment of validated learning.

Minimum viable product is an attempt to get startups to simplify, but it is not itself simple. How do you know which features are essential and which should go? There is no formula, it requires judgment. Any scientific method requires the choice of a hypothesis to test. This leads to two questions:

1. By what standard is this hypothesis to be chosen? Minimum viable product proposes a clear standard: the hypothesis that seems likely to lead to the maximum amount of validated learning.
   
2. How do you train your judgment to get better over time? Again, the answer is derived from the hard-won wisdom of the scientific method: making specific, concrete predictions and then testing them via experiments that are supposed to match those predictions helps scientists train their intuition towards the truth. 

(Fans of the history of science will recognize this as Thomas Kuhn’s theory of scientific paradigms. Minimum viable products are not a single hypothesis. They should therefore be properly understood as product paradigms. As in science, the paradigms that survive will be those that allow practitioners to discover the most productive experiments to try, during the period Kuhn calls “normal science.” A paradigm crisis is analogous to a pivot.)

I told you it wasn’t simple. And this leads to a last criticism of minimum viable product that I hear from time to time: it’s just too complicated. Most people prefer simple, short, pithy startup advice. I remember this acutely from my debate with David Heinemeier Hansson, of 37 Signals fame. As I was explaining the MVP concept, I could see the look of horror on his face. His answer, to paraphrase, was something like this: “that’s way too complicated. Just build something awesome, something that you yourself would love, and ship it.”

Other similar forms of this advice abound: “release early, release often,” “build something people want,” “just build it,” etc. This Nike school of entrepreneurship is not entirely misguided. Compared to "not doing it," I think “just do it” is a superior alternative.

But the teams I meet in my travels are often one step beyond this. What do you do the day after you just did it? It really doesn’t matter if you took a long time to build it right or just threw the first iteration over the wall. Unless you achieve instantaneous overnight success, you will be faced by difficult decisions. Pivot or persevere? Add features or remove them? Charge money or give it away for free? Freemium or subscription or advertising?

I won’t apologize for this aspect of the Lean Startup methodology. These are complicated questions. We are drawn to easy answers because we look at the landscape of successful companies with a biased lens. We see examples of startups who did things “our way” and were successful. Unfortunately, that’s true no matter which way we prefer. Even in the narrow field of giant tech companies, their early products were wildly different. Compare eBay and Google, Apple and Sun, Oracle and Seibel. And, of course, there’s incredible selection bias. For every successful company we think we know that “built it right” or “shipped crap” from the start, there are plenty we’ve never heard of, because they followed that same strategy and promptly died. That’s the deep flaw in most startup advice: it argues from selective examples.

So what about the question of whether good enough really is? What’s needed, I believe, is an alternative discipline that teams can get excited about. When we’re talking about being disciplined, following our methodology with rigor, continuous improvement, there is no such thing as good enough. Our pursuit of learning is ongoing and our commitment is absolute. But when it comes to the specific of a product release, business plan, or marketing launch, all that matters is: do we have a strong hypothesis that will enable us to learn? If so, execute, iterate, and learn. We don’t need the best possible hypothesis. We don’t need the best possible plan. We need to get through the build-measure-learn feedback loop with maximum speed.

Over time, I believe we will build a new professional discipline that will seek excellence at this kind of product-centric learning. And then that new breed of managers will, I'm sure, confidently go around saying: good enough never is.