Friday, December 31, 2010


Thank you, 2010.

I'm excited for 2011, and I want to share some of my plans for the coming year. But before I do, I also feel obligated to take a look back on my plans for last year.

I can't believe a whole year has passed since I wrote a blog post called "Towards a new entrepreneurship" laying out my priorities for 2010. Back then, I wrote:
"[Here is] an idea that I don't think is too widespread yet: that entrepreneurship is an industry. Sure, when entrepreneurs create startups that grow up into mature companies, they become part of an established industry, with its own ecosystem, norms, partners and best practices. But until that happens, we entrepreneurs have our own ecosystem, of investors and service providers, norms and even some "best" practices. The two ecosystems have diverged significantly in the past fifty years - and especially in the past ten. The reason is that the underlying theory that powers established business, the theory of general management, is increasingly inadequate for managing startups. And yet, so far, we lack a coherent theory to replace it. My belief is that the lean startup is that theory. Together, we are part of a movement that is redefining entrepreneurship."
This idea has become a reality in many ways this year. Our ideas have entered the mainstream of startup thinking and even the popular culture. I mean, who ever thought we'd see this cartoon in the New Yorker magazine?

Lean Startup Meetups are now in more than 75 cities, with more than 12,000 combined members. More and more, I am meeting entrepreneurs and managers from companies large and small who agree on this one point: entrepreneurship is management. By applying the same scientific principles that gave rise to general management in the first place to entrepreneurship and innovation, we are unleashing incredible creativity. But we still have a long way to go.

In that spirit, I want to review the four priorities I laid out at the start of this year. For 2010, I announced four main projects. Here's how I laid them out (in their original embarrassing order), and here's how each one turned out.

  1. The Lean Startup Cohort program. Verdict: FAILURE. This seemed like such a promising idea at the time. Take a small number of high-growth companies and have them pay a premium price to learn from me and from each other how to apply Lean Startup ideas in depth. My main hypothesis was that making the program expensive would act as a quality filter, and that if we could find smart, committed companies to participate, they would all benefit tremendously. Thus, I assumed the biggest risk was finding participants who could afford the price.

    Unfortunately, I was completely wrong. Finding participants was no problem; the program quickly filled up. And the quality of participants was way higher than I imagined. And yet, when we actually started to run the program, it still failed. Teaching Lean Startup concepts in a fixed order really didn't work, since all active companies face different challenges at different times. And even in a strict, high-quality filtered room, most companies didn't want to share their problems and internal data, nor did they particularly want to engage with other companies' problems. In retrospect, that should have been obvious to me - as an entrepreneur, I would never have had the patience for a program like that.

    What's that you say? Even "gurus" have to get out of the building, build a minimum viable product, and pivot? Why, yes, they do. Embarrassing, but at least we failed fast. (Peter Drucker thought people used the term guru because it was easier to spell than charlatan.)

  2. Teaching in academia. Verdict: MIXED. I started the year co-teaching a Lean Startup class for MBA's at Berkeley with Steve Blank. In some ways, it was a big success: the class was oversubscribed, had a record number of auditors, and received positive reviews. But the experience left me with doubts about whether that is the right way to engage with academia, for me.

    I strongly believe academia has an important role to play in transforming the practice of entrepreneurship. Luckily, Steve has been leading the charge to bring a new way of teaching entrepreneurship into academic programs, and 2010 saw the debut of his Durant School of Entrepreneurship at sllconf, as well as new programs like the Lean Launch Pad at Stanford and the Business Model Competition at BYU.

    I believe significant new research also needs to be done. What we know today is just the tip of the iceberg about this new entrepreneurial management. How many of our beliefs are just tactics that sound good, or that only work in certain situations? Much more is needed, and 2011 will see the first few buds of that research project flower. My colleagues at Harvard Business School will debut a new Lean Startup-themed course for MBAs this spring, as well as a new $50,000 Minimum Viable Product Fund. As part of that project, HBS has commissioned a series of new case studies on Lean Startup practices, both in and (importantly) outside the software industry. (You can see a little taste at Jeffrey Busgang's blog here.) I've also begun a collaboration with Nathan Furr at BYU to research actual practitioners, following them over time with an eye towards discovering ways to test some of our beliefs about Lean Startup ideas empirically. You can follow our work (and volunteer to be studied) here.

    Also along these lines, I've worked with a variety of collaborators to produce case studies right here on Startup Lessons Learned. Hopefully, more will come in the new year. You can see our efforts so far.

  3. Startup Lessons Learned Conference (sllconf). Verdict: SUCCESS. This was a project I almost didn't do this year, because the prospect made me so nervous. Boy am I glad I did. I still receive regular feedback from people who were there live or in one of our 60+ simulcast locations around the world. It was always a dream of mine to produce a conference where knowledge - not hype - was king, where information was presented in a useful order, and where success theatre and vanity metrics were banned. I believe we succeeded on all three counts.

    In case you missed it, here's a little taste of the event itself, courtesy of my friends at Micro-Documentaries:

    And don't forget, you can watch full video of the entire conference courtesy of our sllconf channel.

    In 2011, we will do sllconf again, probably in mid-May. As always, I will look to you readers for guidance and suggestions of what we should do different. Stay tuned for details. If you are interested in speaking or mentoring at sllconf 2011, we will accept suggestions and nominations. If you would like to nominate someone, please post a video of them giving a talk (with slides if possible).  Grainy low-def youtube videos are perfectly adequate. We had far too many submissions last year on behalf people I didn't know. I had to be confident they would meet the standards I laid out above, but I couldn't take the time to meet them all. Therefore, if you'd like to speak this year at sllconf, a great way to get a leg up would be to speak at a Lean Startup Meetup, and ask someone to record the session. And if you are a meetup organizer, and have had a great speaker who you'd like to see at sllconf 2011, please let me know.

    In other conference news, we'll also have an event at SXSW. We'll make details available soon, I promise. If you're going to be in town for SXSW, and might like to join as a speaker, sponsor, or attendee, please let me know.

  4. Writing a book. Verdict: TBD. I am in the final weeks of preparing a manuscript for The Lean Startup Book which will be published by Crown (one of the largest business book publishers in the world) in 2011. I sincerely hope you'll like the final result; it has been a labor of love for me all year.

    Deciding to publish this book through traditional channels took a lot of thought. I believe it is time for our movement to Cross the Chasm into mainstream awareness. Our early successes have been impressive, but we are still just at the beginning. In my talks all this year I have been exhorting audiences to Stop Wasting People's Time. Our modern economy is full to the brim of waste: building products that have few customers, that produce negative returns for investors, or companies stuck in the land of the living dead. And yet, the people who are responsible for this waste are not generally early adopters of new ideas about entrepreneurship. They are not scouring blogs for the latest gems in innovation thinking. They are overwhelmed, doing the best they can, and get information from only a few sources. They don't want avant garde advice, they want to be reading the same things everyone else is reading.

    My belief is that, in order to reach this mainstream audience, we need to produce a book that is accessible to them, and then make that book a bestseller. That's one of my main goals for 2011, and I will be asking you to help many, many times in the coming year. I hope you'll continue to support me as you have this past year.

    As a reader, the rational thing to do with a new book is to wait until the book comes out, see if your friends and colleagues read it, and if they do, see if they think it's any good. That's classic mainstream customer thinking. Hopefully, the early adopters and visionaries among you will disregard this advice, and agree to pre-order the book instead. The more of you who do that, the more people we'll be able to reach when it debuts next year. Remember, mainstream customers will be looking to you to see if it's worth buying.

    You can pre-order it from me directly, or get an even better price at Amazon.

    (If you'd like to help, I'm still looking for test readers, case studies, and - most importantly - help bringing traffic to the book website. We're running constant A/B tests there; anyone who is able to donate traffic, ads, or a link from your own blog/website will have my gratitude.)

So that was 2010. I believe 2011 will be even better.

It's an auspicious time. Entrepreneurship is in a new renaissance. There are more startups operating today than at any time in history. New ideas about entrepreneurship are in the air. And the dominant management paradigm of the past century has run its course. Literally.

2011 will mark the one hundredth anniversary of the idea of management. I date its origin to the publication, in 1911, of Frederick Winslow Taylor's The Principles of Scientific Management, one of the most important management books ever written. Management's second century will be very different than its first. Our problems are more complex, faster moving, and we face greater uncertainty. In other words, we need entrepreneurs to solve them. I'm excited to see what comes next.

I hope you've all had a happy holidays, and I wish you the best for a new and exhilarating New Year. Here's to 2011!

Thursday, November 25, 2010

Why do we do this?

On April 1, 2009, I was as nervous as I have ever been in my life. It was just minutes before I was supposed to go on stage for the very first time and present The Lean Startup to a large audience, at a big conference. To that point, I’d been talking about Lean Startup concepts only on a seldom-read blog and with people in my immediate network. I had advised startups and VC’s, guest lectured a few time, and met with some small groups of extremely early adopters, like Sean Murphy’s Bootstrapper’s Breakfast. This was different.

I’d had plenty of public speaking experience in my life. This was not the largest audience I’d ever spoken in front of, and I’ve since stood before much larger. I had pitched startups and products, raised money, and lived through some tough negotiations. But this time, I was presenting an idea, not a product. And I was presenting on my own behalf, not on behalf of a company, risking ridicule and hoping for acceptance.

Don’t think I wasn’t prepared. This is a Lean Startup talk we’re speaking of, people. I had a Customer Advisory Board, made up of the type of people who attended this conference, to give me feedback on my talk and slides. I had beta tested the talk with a smaller audience of entrepreneurs at Stanford. And I had been testing variations on the underlying stories on just about anyone who would listen.

My anxiety stemmed from a question that kept occurring to me in the hours before I walked on stage: why am I doing this? For some people, getting up on stage is the most natural thing in the world. Not me. Some people love going to conferences, mixers, and summits. Not me. Some people thrive in the “startup scene.” Not me. When I was a real live practicing entrepreneur, I never had the time or the desire for that stuff. And yet, here I was, about to try and convince nearly a thousand people that I had something valuable to say about entrepreneurship. Why?

It was a defining moment for me. I sought a quiet spot, away from the stage and the crowd. I asked myself why over and over. Startups should be more successful, I answered. Why do I care? Because preventable failures waste time and money. Why do I care? Because entrepreneurs are following “best practices” that don’t work for them. Why do I care? Because I was once one of those failed entrepreneurs. But I'm past that failure now, why do I still care? Because it doesn’t have to be that way.

I remembered a specific moment from my very first startup. It was the moment I realized my company was going to fail. My cofounder and I were at our wits’ end. The dot-com bubble had crashed, and we had spent all of our money. We were trying desperately to raise more, and we could not. The scene was perfect: it was raining, we were arguing in the street. We literally couldn’t agree on where to walk next, and so we parted, in anger, heading in opposite directions. As a metaphor for our company's failure, this image of the two of us, lost in the rain and drifting apart, is perfect.

It remains a painful memory. We had begun as friends, and ended as enemies. The company limped along for months after, but our situation was hopeless. Looking back, I know our failure was inevitable, because we had no clue. It seemed we were doing everything right: we had a great product, a brilliant team, amazing technology, and the right idea at the right time. And, as I’ve mentioned previously, we really were on to something. We were building a way for college kids to create online profiles for the purpose of sharing… with employers. Oops. But despite a promising idea, we were nonetheless doomed from day one, because we did not know the process we would need to use to turn our product insights into a great company.

If you’ve never experienced a failure like this, it is hard to describe the feeling. It’s as if the world is falling out from under you. You realize you’ve been duped: the stories in the magazines are lies, hard work and perseverance don’t lead to success, and – worst of all – the many, many, many promises you’ve made to employees, friends, and family are not going to come true. Everyone who said you were an idiot to do this will be proved right.

Looking back, the idea that this failure was preventable makes me ill. That is the memory I conjured up right before going on stage that April 1st. And I thought, nobody should ever, ever, ever have to go through that. If I can reach just one entrepreneur and help them find a different path, that will be a success. If I get laughed off stage, if I never give another talk, if nobody ever reads my blog, none of that will matter if, in the back of the room, there’s just one person who can use what I have to offer to save their dream, their vision, their startup.

That was the mission that carried me onstage. It helped me stay calm in the face of the unexpectedly large crowd and the incredible response that followed. That day, I had absolutely no idea what would come later – that Lean Startup would become a movement, that it would take over my life, that I'd be writing a book about it, that you would be reading my words today. (You can read my original post-conference report here, including a scratchy iPhone recording of the talk itself.)

All of this is by way of saying, thank you. You are the reason I took those first steps. You are the reason I find this work meaningful. And you keep me going whenever I despair of being able to figure out what to do next.

And there's been plenty of despair, lately. I've been mostly absent from the blog and keeping a much reduced public schedule, as all my energy is devoted to the book. Writing a book is a slog, in the same way that writing a large piece of software is. There are many parts, they all depend on each other, and none is at all valuable unless the whole comes together with high quality. Naturally, quality is exclusively in the eye of the beholder. So, while I'm writing, I can never be quite sure if I've hit the mark. It's been hard, but I am optimistic. This project has allowed me to meet so many entrepreneurs who are building extraordinary organizations. It is their efforts which form the backbone of the Lean Startup movement. The world is going to hear a lot about it next year. And the mission I discovered as a nervous wreck in 2009 continues to motivate me to keep going.

I am grateful to you entrepreneurs, who test ideas in the crucible of daily practice. You are making incredible things happen. You are changing the world. I wish you a happy Thanksgiving.

Monday, October 11, 2010

Case Study: Rapid iteration with hardware

(I am often asked to explain how to apply Lean Startup approaches to domains beyond software. In order to answer, I have taken to drawing a two-axis diagram. 

On one axis we have the degree of market uncertainty for a given industry. For "cure for cancer" type businesses, there is no question about who the customer is and what the customer wants, and therefore there is no market uncertainty. On the other extreme, modern web-based applications face almost no technical risk, and are governed by high market uncertainty.

On the other axis we have the underlying cycle time of the industry in question. Slow-moving cycles, like drug discovery or new automobile models, govern the slow part of the axis. On the extreme opposite end are rapid iteration businesses like software or fashion.

The key to understanding Lean Startup is to recognize two things:
  1. Lean Startup techniques confer maximum benefit in the upper-right quadrant, namely high market uncertainty coupled with fast cycle time.
  2. Every industry on Earth is currently undergoing a disruption that is causing it to move along both axes: more uncertainty and faster cycle times.
I am aware of no industries that are moving "backwards" on either dimension. Thus, more and more industries are starting to look like the software business. Of course, the underlying root cause of this worldwide disruption is the software and semiconductor revolution. Industries are disrupted as their traditional work process is "infected" by software. And, as a result, more and more companies are able to benefit from Lean Startup practices. 

The following case study looks at one such industry, consumer electronics, where the pace of iteration has taken a marked turn towards high speed. It is written by Ronald Mannak, who is currently the CEO of a startup named Yobble. What follows are solely his opinions. -Eric)

In a bar in Amsterdam in 2005, my two cofounders and I came to the sad conclusion that startup we tried to built for two years was doomed. In 2003 we started developing a martial arts motion sensing toy, a full three years before the Nintendo Wii changed the world of motion sensing. The toy (we called it Ninja Master) consisted of two hardware units, attached to both wrists. When a child would perform a perfect karate move (or better yet: a combo of several karate moves in a row), Bruce Lee-like karate sounds would emerge from a small speaker in the device. We loved the product. Test users loved the product. It was way ahead of its time. We thought we were visionaries and believed the future was motion control. Yet, we failed to sell the toy. We talked to every toy company imaginable, but none wanted to license our toy. " Kids nowadays don't want to move, they play Playstation" was the most often heard reply, even though our user tests suggested otherwise. To make matters worse, we lived in a country (Holland) without a proper functioning startup, VC and angel ecosystem. The company was doomed. My co-founders decided startup life wasn't for them.

However, one new idea emerged at that meeting. What if we could make an air drum? Drum sticks with sensors in them. Now that was an idea. Music is much easier to sell (to toy companies) than the abstract martial arts Ninja Master toy. Besides, we could easily expand the line with with an air guitar and a device to link the air instruments to a PC. How cool. I loved the idea so much that I decided to pursue the idea.

I envisioned the product would be popular with 8 to 12 year old boys. I thought the price couldn't be higher than $40. I already knew how the product would be used. Boy, was I wrong.

I previously worked on a couple of IT projects that used the 'waterfall model' where specifications were written down by one team, thrown over an imaginary wall and implemented by another team. Every single waterfall project I encountered turned out to be a disaster in every way. Specifications turned out to be open to multiple interpretations, usability was the last priority (if a priority at all). It Just Did Not Work. As a beta tester of the first Borland Delphi, I learned the wonders of rapid prototyping and fast iterations. I wondered if we could do the same for hardware development. It turned out we indeed could.

The first hire
The first hire was critical. I wanted somebody who was creative first and technical secondly. I found the perfect person at the department of Industrial Design Engineering of the Delft University of Technology. Joris. Joris was creative and eager to learn. Better yet, he plays drums. Even better than that: he likes to tinker with electronics. Hiring him was a no brainer, and he didn't disappoint.

The internship only lasted six months. That's not much time, considering the scope of the project. I convinced the university that Joris should not be writing specifications and other nonsense first, but start right away building prototypes. And he did.

Joris suggested that before he started working on electronics, we should invite children, give them wooden drum sticks and let them pretend they were playing air drums. It turned out to be an excellent idea. Children are perfect test subjects. To our surprise, every single child did something we didn't anticipate. Without any exception, they all whacked the wooden drum sticks *sideways* and made 'crashing sounds'. I certainly didn't think of sideways movements when I created the first ideas, but apparently it was a good idea to implement.

The prototypes
The next day we started building the first prototype to see if the sensors actually behaved like they were supposed to, and to see if we could measure the sideway movements. The prototype was crude. Joris taped sensors on his arms with duct tape and started drumming in the air with wooden drum sticks (that did not contain any electronics). We connected the sensors to a seven year old pc with an Arduino-like interface that ran a simple drum program we developed. The results were amazing. It actually worked. (A video of the first prototype can be found here.)

We now knew what kids liked and we knew the product was technically feasible. Yet, I still felt we didn't know all we needed to know and wanted to test more. And I'm glad we did.

For the next prototypes we placed the sensors in PVC pipes to optimize sensor angles and added features to the pc software.

We made another discovery we did not anticipate. We found out that parents (who came along with their children for user testing) often liked the prototype as much as their kids. We decided to interview the parents and quickly found out that the parents who like our product were video games players. Of course we liked our product, but we never would have guessed other grown ups would like our product too. Knowing this, we invited test users from 12 to 30. They also loved the prototypes. Our target audience just exploded in size. We decided to make a few changes that would make the product less 'toy' and more 'gadget'.

Over a period of six months, we made eight generations of prototypes, each version adding more features and making the product more reliable. By testing each generation, we learned that a lot of hypotheses were correct, but a large number of hypotheses were incorrect. By testing early and often, we were able to adjust the product. I believe that we demonstrated that it is indeed possible to iterate fast and often with hardware development.

Product Launch
After some financing-related delays, the products went on sale in Europe and Asia in the summer of 2008. The retail selling price was $40, exactly what we targeted. In less than six months, we sold over 90,000 units. All shops sold out our products two months before christmas, all without spending one penny on marketing. The products were voted 'best music gadget' on television program The Gadget Show, became the best selling music toys on and the best selling products on Best of all, users love the products. On, the average user rating (740 users) is 4.5 out of 5 stars (link). We couldn't have been happier.

Post mortem
We demonstrated it is possible to iterate often and fast. I believe a lot of the product's success can be attributed to the iterative development process. We didn't find every issue though. We didn't test the price and we didn't see the Nintendo Wii or Guitar Hero coming. We chose to enter the market though the low margin toy market, where (in hindsight) we should have positioned the products as video games with higher video games margins.

Another thing we missed: after we launched we received many requests to add double bass drum, as often used in metal. The drums include two drum pedals and a double bass drum could have been added by a simple and minor change to the embedded software. However, updating the embedded software in sold devices isn't possible with the microcontroller we used. We could have included the feature in a 1.1 version of the product, but the toy manufacturer we licensed the toy to, wasn't interested in a new version, as the original version continues to sell well to this day.

Tools to develop hardware get better and cheaper. Open source projects like Arduino and SuperCollider make iterative hardware development cheaper and faster than ever. We learned that connecting the prototypes to PCs and user the PC to run the program is a very good way to test hardware (developing on a PC is still much faster than embedded developing on a standalone hardware device).

In the summer of this year, I moved to San Francisco and founded a new startup that makes music related games and hardware controllers that connect to the iPhone. There are a lot of new opportunities. New cheap flashable micro controllers make firmware updates possible for low cost hardware. With hardware connected to the internet (in our case through the iPhone) it should be possible to use continuous deployment: small and very frequent updates of the firmware instead of less frequent large updates. Bugs in firmware could be fixed within minutes or hours instead of weeks or months.

(Continuous deployment of hardware is an exciting new capability. In addition to continuous deployment of firmware via the Internet, it is also possible to do continuous deployment by taking advantage of a small-batch production process. When the complete cycle time of assembly is low and the design can be specified mostly through software, it's conceivable that each batch rolling off the line could have a different design. -Eric)

As a final thought, I am convinced iterative design depends mostly on the mindset of the team and the company culture and less on the tools. I was lucky to have a great team of A-players that were willing to take responsibility and risk. If the company culture is such that mistakes are punished, I am pretty sure iterative development won't work.

Tuesday, October 5, 2010

The Lean Startup Bundle

People often ask me if they can buy "Lean Startup in a box" from me. It's a strange thing, to be asked by a potential customer if they can give you copious amounts of money, and then to have to refuse. Startups run into this problem all the time: not every possible way of making money is equally useful. On the other hand, I try to keep in mind the idea that feedback is always about them and never about you. I recognize the need being expressed by this request. People want to get started with Lean Startup but aren't exactly how. If you're in that boat, today my friends at Appsumo are trying a new experiment that just might help you out.

It's called the "The Lean Startup Bundle" and it is available this week only. It is the ultimate product development get-started guide to Lean Startup. We did our best to package as much content (including almsot a dozen books and ebooks) as well as tools and apps, into one low-priced magic offer. Naturally, the bundle includes every essay I have written for this blog, as part of the Startup Lessons Learned ebook series.

But I am even more excited to report that it will include a hardcover copy of my new Lean Startup book. That's right, I am writing an old-fashioned, traditionally-published book full of all new material. Because of the traditional publishing industry's commitment to rapid iteration, the book won't come out until Fall, 2011. But if you buy The Lean Startup Bundle this week, you'll get a pre-order of the book included. That means you will get a hardcover copy of the book in the mail when it is eventually published. (Believe me, you will be hearing a lot more about this book over the course of the next year, so that's all I will say about it right now.)

But that's not all! I'm even more excited to announce two brand-new ebooks from two of my favorite bloggers: Sean Ellis and Andrew Chen. These ebooks were created by my friends at Leanpub especially for this bundle, and include the best essays written by both authors (and a foreword from yours truly). Also included is the incredible Venture Hacks Bible, which is worth the price of admission on its own, and The Entrepreneur's Guide to Customer Development (which I previously reviewed here).

And there's so much more. For two awesome books which have been traditionally published, we have several ebook chapters: Inbound Marketing by Dharmesh Shah, Do More Faster: TechStars Lessons To Accelerate Your Startup by Brad Feld & David Cohen. And also making its debut, you'll get a sample of Ash Maurya's forthcoming ebook Getting Lean.

"But," I hear you say, "so far this sounds like just a bunch of gurus trying to sell me books." Indeed, but - as they say - that is not all. We also reached out to companies that make products useful to accelerating your Build-Measure-Learn feedback loop, from cloud-hosted Selenium provider Sauce Labs (disclaimer: where I have equity) to ScrumPad (where I don't) and many in-between (a sampling is below, but you should really just click through to see the whole smorgasbord). (And even if you don't buy this bundle, one beneficial side-effect of my research for it is this cool list of Lean Startup tools curated by Because of the short timeline in putting this all together, we couldn't include them all. But, if this bundle is successful, perhaps we'll have a chance to do another.)

Phew, I'm exhausted. "But," you are sure to ask, "how much will all this bundling cost me?" How about $1000? $500? $250? No! All of those prices are much too high. "Ok," you surely say, "how about $100?" Absolutely not. "$75?" No. "$65?" Absurd.

No, all of those prices are incorrect. If you act now, you will pay not one dollar more than $42. Don't panic. Buy now.

Monday, October 4, 2010

Stop lying on stage

Entrepreneurs crave information about successful startups, and they should. Most of the received wisdom about business and entrepreneurship is simply wrong. Many journalists and conference organizers attempt to fill this demand by giving successful entrepreneurs the opportunity to tell their stories: in magazines, on blogs, and on stage. And yet, most of the time, those opportunities are wasted, because the protagonists tell lies. And while this may sound like a harmless phenomenon – after all, most of these are simple lies of omission – I think the consequences are quite harmful.

I know the word “lie” sounds harsh. But I think our industry has to face up to this unpleasant reality. Luckily, some people have started to collect evidence of misleading behavior on the part of speakers. Paul Graham wrote this just the other day:
I didn't consciously realize how much speakers at more public events censored themselves till I was able to compare the same people speaking off the record at YC dinners and on the record at Startup School. YC dinner talks are much more useful, because the details people omit in more public talks tend to be the most interesting parts of their stories. About half the interesting things I know about famous startups, I learned at YC dinners.
I commend Paul for his honesty, something I have always admired about him. Paul has been – intentionally or unintentionally – running a perfect science experiment to answer the question: do startup speakers tell the whole truth on stage? His results are the same as I’ve observed on many occasions: the simple answer is no. I think everyone who’s been around these events long enough knows this to be true. None of the really successful people I know take what’s said at public events seriously. This is the same issue we see with vanity metrics: companies are giving the appearance of sharing information while actually engaging in spin or outright deception.

I call this the vanity ratio: the amount of apparently interesting information given divided by the amount of useful information contained therein. The higher the vanity ratio, the more effective the PR. Unfortunately – also – the more misleading the story is as a help to others.

And it’s not as if the stories we hear about entrepreneurs are biased in a random way. Paul quotes one of his founders like this: “That's the actual beauty in the off-the-record-ness: you hear just how screwed up most of these successful startups were on the way up.” In my experience, the official stories are always more linear, make the founders and investors look smarter, and dramatically overstate the level of certainty everyone had at every stage of the process. Failures, pivots, and crappy minimum viable products are generally elided. And the kinds of failures that do get airtime are usually failures to adequately plan, anticipate, or design in advance. So, naturally, the kinds of inferences we make from these stories are: we need heroic entrepreneurs, with absolute certainty in a brilliant idea, and we need to plan and execute well.

I call this the mythological-industrial complex, because it serves the interests of many players in preserving the status quo. It sells newspapers and magazines. It helps investors boost their profile and convince entrepreneurs that they offer value-add. It helps companies with PR. It makes successful founders famous. I've certainly been a beneficiary of these forces. Yet I worry that it deters new entrants from disrupting incumbents: if your idea looks a little dubious, your career a little messy, your team a little dysfunctional, if you lack superhuman design skills, maybe you should just give up. You don’t have the Right Stuff to become an entrepreneur.

The reason this is harmful is that it benefits another kind of person even more: the startup advisor. Yup, that’s me. When all of the stories floating around are bogus, there is a huge opportunity for people to claim to have arcane or esoteric knowledge. Paul does it right in his essay: “About half the interesting things I know about famous startups, I learned at YC dinners.” Doesn’t that sound good? Don’t you believe that having that special knowledge would help Paul give you better advice? I emphatically do. Most of my mentors have had this arcane knowledge, and I have benefited from it immensely. They know how the game is really played. They know what really happened in those pivotal moments that shaped famous companies. And I have done my best to pay it forward, by sharing that arcane knowledge with as many startups as I can.

However, the problem is this: how can you tell if someone who claims to have arcane knowledge is telling you the truth? After all, the fact that what they advise you contradicts all public sources of knowledge is part of the pitch. And, of course, it sounds good. All talking heads (including me!) face an overwhelming incentive to say whatever it is that will sound good to their audience, regardless of whether it is true. Believe me, it is hard to resist (as far as I can tell, that’s why cable news is so awful: pretty much everyone on cable news is giving in to this temptation every day). But I have met many startups that are making key decisions based on arcane knowledge that is simply not true. It is extremely hard to help them, because they are following voodoo advice that is nearly impossible to disprove. In fact, in some of my early entrepreneurial ventures, I was taken in by people just like that.

I have tried really hard to hold myself to a high standard in all of my public presentations: to give the unvarnished truth about what it was really like to succeed and fail. I do my best and yet I probably fail sometimes, too. In fact, I rely on readers and attendees to ask me the hard questions that challenge me to root out and eradicate these errors. And this was an absolute requirement of the speakers at sllconf. The reason we had such a non-diverse lineup (much to my chagrin) is that I insisted on hosting speakers whose stories I already knew – because I had been there, as a friend, advisor, or investor. Thus, I knew they were telling the truth and I knew they shared a commitment to speak with integrity even when it’s uncomfortable. This is also why I get so outraged when people treat Lean Startup like a religion. The whole point of transforming advice into the form of a testable theory is to allow people to evaluate whether it works. You can test theories like Lean Startup in your own practice, and discover if they deliver the outcomes they predict – and you can do it in small batches.

That’s why I find lying on stage so upsetting. By misleading future entrepreneurs, we put them in an untenable position. Either take the stuff you read and hear at face value or gain access to a wizard who can guide you to the true path (and hope you don’t get taken in). But this false choice is not the only way forward. As usual, I think there’s an opportunity for our industry to do better.

To be clear, I don’t blame any particular actor in the system for doing what they’re doing. Paul Graham is getting his startups the best advice he can in the best way he knows: by giving them exclusive access to off-the-record conversations that nobody else is privy to. And it’s not reasonable to blame him for their behavior when he gives them the stage at events like Startup School. Similarly, journalists print vanity metrics because that’s all companies will release. And companies crave positive PR and control over their message – all for rational reasons.

And yet you who are reading this right now have tremendous power. You are the intended audience for all of those expenditures of energy. You are the “hits” that websites crave, the followers and the RSS subscribers. Where you put your attention and what you do with it matters.

So I’d like to suggest the following: let’s stop giving lying on stage and vanity metrics a free pass. I think if we can delegitimize this behavior, we can make it stop. We’ll need to ask tougher questions, though: how do you know your company’s success was caused by X? what else was happening around that time that might have caused Y? what did you think was going to happen when you did Z?

Journalists have the highest obligation to ask these kinds of questions, and conferences organized by journalists ought to be the exemplars the rest of us look up to. And yet, today many are not. I am sympathetic, because I have faced this problem, too. If you ask tough questions, are unwilling to help people craft their message, and are skeptical of vanity metrics, you can’t get the high profile guests. That means less attention, less coverage, fewer readers, and lower sponsorship dollars. Assuming, of course, that all of you give your attention disproportionately to famous people.

But if you’re attending a conference or reading a magazine, you aren’t bound by those rules. You can ask any question you want. And you can reward the speakers who tell the whole truth with your support.  Every time you do that, you’re helping make our industry a better place.

(Have a favorite real startup story? Share it in the comments and we can start giving those people some much-deserved attention right away.)

Monday, September 27, 2010

Good enough never is (or is it?)

One of the sayings I hear from talented managers in product development is, “good enough never is.” It’s inspirational, always calling the team to try harder and do better. It works to undermine excuses for poor or shoddy work. And, most importantly, it helps team members develop the courage to stand up for these values in stressful situations. Especially in teams that are managing by objectives (or OKR's), the pressure to deliver is intense. Under such pressure, the temptation to cut corners, to quit prematurely, or to hand off shoddy work to another department is overwhelming. It requires courage to stand up and say: "this work is simply not good enough. Sure, we could get away with it, but that's not how we work." Good managers work hard to create an environment where this courage thrives.

On the other hand, there are many stories of companies achieving a breakthrough by shipping something that was only "good enough." One such rumor, which I’ve heard from several sources, tells of the launch of Google Maps. The team was demoing their AJAX-powered map solution, the first of its kind, to senior management at Google. They were impressed, even though the team considered it still an early prototype. Larry and Sergey, so the legend goes, simply said: “it is already good enough. Ship it.” The team complied, despite their reservations and fear. And the rest is history: Google Maps was a huge success. This success was aided by the fact that it did just one thing extremely well – its lack of extra features emphasized its differentiation. Shipping sooner accentuated this difference, and it took competitors a long time to catch up.

So which is it? Is "good enough" good enough? Rules of thumb can be infuriatingly unhelpful. When should you settle for good enough and when should you push yourself to do your best?

This is precisely the dilemma that the doctrine of minimum viable product is designed to solve. And it’s really hard.

Most of us intuitively have a “split the difference” attitude when faced with recurring difficult choices. That is not a long-term solution. The reason: it actively encourages factional strife. Everyone naturally falls along a spectrum, from “ship anything soonest” to “always build it right, no matter what it takes.” When members of a team realize that the final answer will be some kind of average, they face an overwhelming incentive to express desires in the strongest possible terms. After all, someone else’s view will be averaged in, too. Any excesses are likely to be moderated by others. Of course, this logic applies to members of all factions. Over time, such teams either explode due to irreconcilable differences or dramatically slow down. The latter is actually more dangerous. Divided teams usually can’t agree on facts or interpretations. Yet startups rely on collective learning in order to find their way. Factional strife is learning kryptonite. I believe this is one reason why the myth of the dictatorial startup founder has such enduring appeal. Faced with these kinds of disagreements, strong arbitrary action is much superior to paralysis.

But action/paralysis are not the only options. As in many false dichotomies, we can find a third way that gives both factions a positive message to rally around.

Without an affirmative message, managers can cause lasting harm. I certainly have. When people start using quality, reliability, or design as an excuse to delay, it used to make me nervous, even when these suggestions were well intentioned. After all, how would Craig Newmark’s life (and the rest of ours, too) be different today if he had waited to build something with a high-quality design before starting his famous list? Rather than having this repeated argument, I sometimes found it easier to play dictator on the other side, forcing teams to ship sooner than they were comfortable with. As I found out to my dismay, this is a dangerous game: in many cases, you’re asking trained professionals to violate their own code of best practices, for the good of the company. Once you go down that road, you risk opening a Pandora’s box of possible bad behaviors. And yet, it does not have to be that way.

Almost everything we know today about how to build quality products in traditional management has its origins with W. Edwards Deming, the original quality guru. He had two concepts that are especially important to this discussion. The first is that “best efforts are not enough.” Despite what it seems in the moment, most quality problems are not caused by people slacking off or acting maliciously. (It seems that way only because of a psychological phenomenon called the fundamental attribution error.) In reality, most quality problems are systemic in nature. They have to be solved in the boardroom by making a company-wide commitment to building quality into the very systems the company uses to build products. Lean manufacturing, agile software development, and Theory of Constraints are all examples of this idea in action.

However, a commitment to quality alone is not enough. In old school manufacturing, quality was defined as reliability: parts and products that did not wear out, break down, or fail unexpectedly. And so Deming’s contribution was especially prescient, as he saw that “the customer is the most important part of the production line.” This means that quality is defined in the eye of the customer, not necessarily by arbitrary standards loved by insiders to the production process. In today’s world, this is increasingly important, as quality is often defined by factors beyond reliability: design, ease of use, aesthetic appeal, and convenience.

Now we come to the heart of the minimum viable product issue: how can we build quality in if we do not yet know who the customer is? All of our professional standards that lead us to want to get it right the first time – all of them were developed originally in a non-startup context, one where the customer was known in advance. Startups are different, leading to this axiom: if you do not know who the customer is, you do not know what quality is.

Which takes us right back to the original definition of minimum viable product:
the minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning with the least effort.
In other words, the minimum viable product is a test of a specific set of hypotheses, with a goal of proving or disproving them as quickly as possible. One of the most important of these hypotheses is always: what will the customer care about? How will they define quality?

One common worry is that this might lead companies to “release crap,” shipping too soon with a product of such low quality that it alienates potential customers and, thus,  causes entrepreneurs to abandon their vision. This critique combines two misunderstandings in one.

First, I want to explore the idea of releasing crap: that our product is of such low quality that we will release it, customers will hate it, and we’ll have accomplished nothing but alienating them. But notice how many hypotheses are baked into this supposedly simple scenario: we believe we have already solved the distribution problem for our product (or else how could customers try it?). We already know who to distribute the product to (or else why would we care what they think?). Naturally, we already know the standard of quality that they will use to judge our product. And, of course, we already know that they will care enough to be offended. In fact, we know so much that we already know what they will care enough about (namely, the product’s quality – as opposed to, say, missing features).

Even better, this is a falsifiable hypothesis. It is entirely possible that we can ship “crap” and have one of the aforementioned facts fail to materialize. In fact, that is one of the best possible outcomes, because it will force us to learn something. What if customers actually like the “crap” product? Or what if we can’t get any of them to even try it? Or what if the features they demand we build are different from the ones we were planning to build? In those cases, we can’t help but learn a great deal. Remember, the minimum in minimum viable product does not mean that you should ship just anything at the nearest possible date. It means to ship as soon as it is possible to learn what you need to learn.

The second misunderstanding is a concern for what will happen if things turn out exactly as we originally predicted (namely, badly). Entrepreneurs, faced with an early defeat, might lose their commitment to seeing their vision through. I understand this fear. It is a direct consequence of the reality distortion field, that ability most visionaries have to get people to believe in a vision as if it was already true. Data can undermine this field. It's easier to believe in a glorious future when you have only zeroes, for everyone: founders, investors, and employees.

But this fear is way overblown, in my experience. The great visionaries I’ve worked with can incorporate a commitment to iteration into their process. However, there are some important ground rules. As I wrote in Don’t Launch, it’s essential to remember that these early minimum viable product launches are not marketing launches. No press should be allowed. No vanity metrics should be looked at. If there are investors involved, they should be fully briefed on the expectation that these early efforts are designed to fail.

Again, even if they do "fail," it is improbable that they will fail in the way we originally expected. In fact, in all of the startups I have worked with, I have never seen this happen. There is always something unexpected when customers react to a product in the real world: we thought they’d be offended by low quality, but actually they refused to download it; we thought they’d share it with their friends, but actually they wanted us to provide the friends; we thought they’d care a lot about our beautiful design, but actually they wanted more features. As in any experiment, the important thing is not the bare fact that the hypothesis was invalidated. More important is to understand the reasons why. This is not an academic exercise; the goal of these experiments is to immediately get up off the mat and design the next one. And the next, and the next, until we have not just learned but proved our learning with hard facts: through the attainment of validated learning.

Minimum viable product is an attempt to get startups to simplify, but it is not itself simple. How do you know which features are essential and which should go? There is no formula, it requires judgment. Any scientific method requires the choice of a hypothesis to test. This leads to two questions:

  1. By what standard is this hypothesis to be chosen? Minimum viable product proposes a clear standard: the hypothesis that seems likely to lead to the maximum amount of validated learning.
  2. How do you train your judgment to get better over time? Again, the answer is derived from the hard-won wisdom of the scientific method: making specific, concrete predictions and then testing them via experiments that are supposed to match those predictions helps scientists train their intuition towards the truth. 

(Fans of the history of science will recognize this as Thomas Kuhn’s theory of scientific paradigms. Minimum viable products are not a single hypothesis. They should therefore be properly understood as product paradigms. As in science, the paradigms that survive will be those that allow practitioners to discover the most productive experiments to try, during the period Kuhn calls “normal science.” A paradigm crisis is analogous to a pivot.)

I told you it wasn’t simple. And this leads to a last criticism of minimum viable product that I hear from time to time: it’s just too complicated. Most people prefer simple, short, pithy startup advice. I remember this acutely from my debate with David Heinemeier Hansson, of 37 Signals fame. As I was explaining the MVP concept, I could see the look of horror on his face. His answer, to paraphrase, was something like this: “that’s way too complicated. Just build something awesome, something that you yourself would love, and ship it.”

Other similar forms of this advice abound: “release early, release often,” “build something people want,” “just build it,” etc. This Nike school of entrepreneurship is not entirely misguided. Compared to "not doing it," I think “just do it” is a superior alternative.

But the teams I meet in my travels are often one step beyond this. What do you do the day after you just did it? It really doesn’t matter if you took a long time to build it right or just threw the first iteration over the wall. Unless you achieve instantaneous overnight success, you will be faced by difficult decisions. Pivot or persevere? Add features or remove them? Charge money or give it away for free? Freemium or subscription or advertising?

I won’t apologize for this aspect of the Lean Startup methodology. These are complicated questions. We are drawn to easy answers because we look at the landscape of successful companies with a biased lens. We see examples of startups who did things “our way” and were successful. Unfortunately, that’s true no matter which way we prefer. Even in the narrow field of giant tech companies, their early products were wildly different. Compare eBay and Google, Apple and Sun, Oracle and Seibel. And, of course, there’s incredible selection bias. For every successful company we think we know that “built it right” or “shipped crap” from the start, there are plenty we’ve never heard of, because they followed that same strategy and promptly died. That’s the deep flaw in most startup advice: it argues from selective examples.

So what about the question of whether good enough really is? What’s needed, I believe, is an alternative discipline that teams can get excited about. When we’re talking about being disciplined, following our methodology with rigor, continuous improvement, there is no such thing as good enough. Our pursuit of learning is ongoing and our commitment is absolute. But when it comes to the specific of a product release, business plan, or marketing launch, all that matters is: do we have a strong hypothesis that will enable us to learn? If so, execute, iterate, and learn. We don’t need the best possible hypothesis. We don’t need the best possible plan. We need to get through the build-measure-learn feedback loop with maximum speed.

Over time, I believe we will build a new professional discipline that will seek excellence at this kind of product-centric learning. And then that new breed of managers will, I'm sure, confidently go around saying: good enough never is.

Monday, September 20, 2010

The visionary’s lament

“But customers don’t know what they want!”

It’s an anguished cry that I have heard often from startup founders. In a way, I don’t blame them. I’ve been there myself. If we’re not attempting something truly new and innovative – what’s the point? If we’re just going to conduct the world’s biggest focus group to decide what to do, why couldn’t any old idiot do it instead? Isn’t the whole point of devoting our life to this enterprise to show the world that we have a unique and visionary idea?

I remember one conversation with a visionary quite well. He had just come back to the office after a few days away, and he was filled with big news. “I have incredible data to share!” which was pretty unusual – a visionary with data? He carefully explained that he had conducted a number of one-on-one customer interviews, showing them an existing product and then documenting their reactions. His conclusions were well thought out, coherently based in the data he was presenting, and painted an alluring picture of a new way forward. His team almost exploded on the spot.

“That’s the same idea you’ve been pushing for months!” “What were the odds? Customers explained to you that we need to do exactly what you wanted to do anyway? Wow!” It was an ugly scene.

We all know that great companies are headed by great visionaries, right? And don’t some people just have a natural talent for seeing the world the way it might be, and convincing the people around them to believe in it as if it was real?

This talent is called the reality distortion field. It’s an essential attribute of great startup founders. The only problem is that it’s also an attribute of crazy people, sociopaths, and serial killers. The challenge, for  people who want to work with and for startups, is learning to tell the difference. Are you following a visionary to a brilliant new future? Or a crazy person off a cliff?

True visionaries spend considerable energy every day trying to maintain the reality distortion field. Try to see it from their point of view – none of the disruptive innovations in history were amenable to simple ROI calculations and standard linear thinking. In order to do something on that scale, you need to get people thinking, believing, and acting outside the box. Their greatest fear is categorically not that their vision is wrong. Their real fear is that the company will give up without ever really trying.

This is where data, focus groups, customer feedback, and collaborative decision-making get their bad rap. In many cases, these activities lead to bad outcomes: watered down vision, premature abandonment, and local maxima.

When visionaries say “but customers don’t know what they want!” they are right. That’s the problem with false dichotomies: each side has a kernel of truth within it. You cannot build a great product simply by obeying what customers say they want. First of all, how do you know which customers to listen to? And what do you do when they say contradictory things?

And yet, the people who resist visionaries also have a point. Isn’t a bit scary, maybe even suicidal, to risk everything on a guess – even if it is emotionally compelling?

Like all false dichotomies, if either side “wins” this argument, the whole enterprise loses. If we just follow the blind mantra of “release early, release often” and then become purely reactive, we’re as likely to be chasing our tail as to be making progress. Similarly, if we pursue our vision without regard to reality, we’re almost guaranteed to get some aspects of it wrong.

The solution is synthesis: to never compromise two essential principles. One, that we always have a vision that is clearly articulated, big enough to matter, and shared by the whole team. Second, that our goal is always to discover which aspects of this vision are grounded in reality, and to adapt those aspects that are not.

A vision is like a sculpture buried in a block of stone. When the excess is chipped away, it will become a work of art. But the challenge in the meantime is to discover which parts are essential, and which are extraneous. The only way to do this is to continuously test the vision against reality and see what happens.

So what should you do if you find yourself working with a visionary? Almost every successful visionary has found partners to work with that help them stay grounded in reality. To do this you have to find ways to be supportive of the vision at the same time as reporting the bad news about where the vision falls short. I recommend a mantra that I learned from Steve Blank: always consider your job to find out if there is a market for the product as currently specified. Don’t try and change the vision every time you get new data. Instead, get out of the building and look for customers for whom your product vision is a slam-dunk fit. If and only if, after exhaustive searching, you cannot find any customers that fit the profile, is it time to have a serious conversation about whether and how the vision should be modified (a pivot).

And what should a good visionary do to help find synthesis? Based on the successful visionaries I have had the opportunity to work with up close, I'd like to offer two suggestions for the role a visionary should take on:

  1. Identify an acute pain point that others don’t see. It’s important to specify the vision as much as possible in terms of the problem we’re trying to solve, rather than a specific solution. (Or, to use Clay Christensen's formulation, of the "job" customers are hiring us to do.) Even though the visionary surely has some concrete ideas which are to be tried, he or she should always be asking, “would I rather solve the problem, or have this specific feature?”
  2. Hold the team to high standards. Despite Steve Jobs' incredible talents, he doesn’t personally design and ship every Apple product. It’s much more likely that his main function is to hold everyone who works for him to the same high standard. Once they’ve agreed to try and solve a dramatic problem, it’s the visionary’s job to hold each provisional result up to the light of that vision, and help the team remember that although trade-offs and compromises are always necessary – the real payoff is in solving that acute pain. This can help avoid the trap of the false negative: even if the first few iterations don’t get it right, the vision inspires us to learn from our failures and keep trying.

Let me close with a specific story of a visionary at work. I’ve heard from several sources a story about Jeff Bezos and the invention of one-click shopping. It may be apocryphal, but it’s illustrative anyway. Amazon had tasked a team with building their new one-click shopping feature, which was designed to reduce the friction required to make an impulse purchase. The purpose of naming the feature “one-click” was to clearly communicate to everyone the vision of maximum simplicity. When Bezos was meeting with the team to review their first version of the feature, so the story goes, after he clicked to make his purchase, he was prompted with a confirmation dialog box. He had to click “yes” to continue. In other words, one-click shopping required two clicks!

Now, it’s really important to see this story from both sides. Bezos was surely infuriated that the team had missed so obvious a point about his vision. But see his team’s point of view: they were immersed in a culture of protecting the customer. It was probably considered too dangerous to let someone “shoot themselves in the foot” and make an unintended purchase that could have serious economic consequences.

But by actually building a version of this feature, and doing some simple testing with customers and with Bezos, this team surfaced an issue that probably wasn’t really clear in Bezos’ vision from the get-go. Namely, how are we going to handle the case of customers one-clicking by accident? The synthesis solution is so simple, I’m sure it seems obvious in retrospect (and I’m sure dozens of people, for all I know including Bezos himself, are now sure they came up with it on their own): since mistakes are the uncommon case, give the customer several opportunities to realize and correct them after the fact, rather than trying to prevent them with a confirmation dialog box.

Those are the attributes I admire in successful visionaries: a determination to see the vision through, holding their teams to high standards, and a commitment to iterate in order to get there.

Monday, September 13, 2010

The Superbowl ad test

I am a firm believer in the danger of vanity metrics, numbers that give the illusion of progress but often mask the true relationship between cause and effect. Since I first started writing about vanity metrics, I’ve met more and more entrepreneurs who are struggling with a simple question: how can I tell a vanity metric when I see it?

From the outside, vanity metrics are a lot easier to see than from the inside, precisely because of the psychology behind them. Everyone wants to believe that the work they are doing is making a difference. So it’s easy to read positive causes into noisy data, whether it’s really happening or not. (This is called “the illusion of cause” and is discussed at length in the extremely readable book The Invisible Gorilla). Even worse, entrepreneurs are faced with a constant barrage of vanity metrics from competitors and other companies engaged in PR. Vanity metrics are generally bigger. And everyone knows bigger is better, right?

News publications print vanity metrics because they want to give their readers information about the companies they cover. Companies want the coverage, but they don’t actually want to reveal anything useful about their operations. The solution? Vanity metrics. By only releasing vanity metrics, companies co-opt the press into helping them mislead others. Is that really news? I’ll leave that to professional journalists to sort out. For the general public, it’s probably OK to treat company updates as entertainment. But for entrepreneurs, investors, analysts and competitors, it’s quite dangerous.

Here’s my quick heuristic for telling if a given number, graph, or chart is a vanity metric: could it have been caused by the company secretly running a Superbowl ad and not disclosing it?

If yes, it’s very likely to be a vanity metric. Let’s take a look at an example, one of my favorites, the "billions of messages" claim.

Here's Mashable's coverage of Facebook chat reaching "a billion messages a day." Or take a recent TechCrunch article about a startup I won't name: “X billion messages sent since June 2009.” These articles treat this as a huge number, and it is. Probably, it represents tremendous success for the company in question. It’s side-by-side with a number of other vanity metrics. But notice what’s not listed: messages sent per person, churn rates of active users, or activation rate of new user. Even worse, we have no indication of how these numbers are moving over time. Is the company growing because of an amazing viral loop paired with a strong engagement loop? It’s possible, but the article doesn’t say. Most of the article is about the features – new and old – of the product. The unstated implication is that these features are what are leading to this tremendous growth. But is that true? Isn’t it equally possible that this company is spending more money on advertising or marketing than it’s competitors? Or that there is some other external factor at work?

I have no insight into these questions, and I don’t mean to pick on these companies in particular. My point is that this article does not contain the kind of information we’d need to draw reasonable inferences, which is by design. That’s what you pay PR firms for: to get an article written that is entirely factual and yet still provides positive spin for your company. (For context "some 740 billion text messages were sent in the first half of 2009." The PR firm helpfully left out that context.)

So, could these numbers have been generated by a Superbowl ad? Of course. We have no idea when the billions of messages were sent. They could all have been sent very recently. That’s the magic of vanity metrics – you never know what’s really going on. The trouble comes when companies and investors come to rely on these numbers to make consequential business decisions. How should a company like the one above prioritize their next set of features? Hopefully, they have internal reports that show the true correlation between their features and customer results. Are employees paying more attention to those reports than to the positive press coverage? I sure hope so.

Notice that cohort and conversion based metrics do not suffer from this problem. When we look at the same conversion percentage for cohort after cohort, we are effectively getting a new, independent, report card for our efforts each period. Each cohort is mostly unaffected by the behavior of earlier cohorts. And it is much more insulated from external effects, like an advertising or PR blitz, than your typical vanity metric.

It is not difficult to translate a gross metric like total messages sent into cohort terms. Since I’m picking on the TechCrunch example from above, we’re talking about more than a year’s worth of data. Let’s divide it into monthly cohorts. For each month, messages are sent by two kinds of people: new customers and returning customers. In order to make each cohort as meaningful as possible, let’s define them as follows:

New customer: someone who registered for the service in a given month
Returning customer: someone who used the service in the immediately preceding month.

I choose the “preceding month” definition in order to give us a sense for individual people’s behavior. A huge advertising blitz might cause a temporary winback effect by bringing in lots of old customers, but this is generally the kind of effect we want to ignore (unless we’re measuring the short-term effectiveness of the advertising).

Now, let’s plot a single number for each cohort, the percentage of customers in that cohort who sent at least one message in that time period. That makes our numbers denominated in people, not messages, which is much easier to understand. (remember, metrics are people, too). If we wanted to get fancy, we could also plot the average number of messages sent per person in each cohort.

If these numbers are flat month-to-month, then we can draw some strong conclusions about the product features we’re working on: they are basically having no effect on customer behavior. Hopefully, that’s not the case. Hopefully, the numbers are steadily improving month after month.

The data needed to generate this simple graph already exists: it’s the same basic data you’d need to get an accurate count of the total number of messages sent, just presented in a different form. For understanding what’s really going on with a product, this alternate form is far superior. Is it any wonder companies don’t want the press to have it?

It’s my hope that, in time, our industry will start to reject vanity metrics as a serious part of the discourse about customers. But this will take a long time. Investors and journalists have the most leverage to start making this change. Entrepreneurs have a part to play, too. Playing with vanity metrics is a dangerous game. Even if you intend to “only” give that sugar rush to publicists or investors, it’s all-too-easy to be taken in yourself. Your employees probably read the same press you are trying to influence. Your investors may be taken in today, but they will use those same vanity metrics to hold you accountable tomorrow. It’s much easier to rely on actionable metrics in the first place.

Monday, September 6, 2010

Lo, my 57692 subscribers, who are you?

Since this blog's earliest days, I have made a habit of surveying you, my subscribers. I did it originally as a demonstration of the advantages of having a pathetically small number of customers, but I found the actual info so incredibly helpful, I have done it several times since. Since the last time, your ranks have grown tremendously, and I thank you all for this incredible support.

So, to celebrate Labor Day here in the US, I've created another survey. If you're willing to take five minutes to fill it out, I would be most grateful:

As usual, I've added a small minimum viable product (I'm starting to think of this technique as the "survey MVP") at the end, as yet another customer validation exercise. I'll post about the results later; to say anything here would bias the survey. 

Tuesday, August 24, 2010


Next year will officially mark one-hundred years since scientific management, the first great management paradigm, burst into the national consciousness. It invented many concepts we now take for granted: efficiency, productivity, and the idea of management itself. We owe that movement an incalculable debt of gratitude. Have you decide already how to celebrate this centennial? I have: I'm going to mark the occasion at SXSW Interactive in March, 2011. You're invited (details below).

I'm starting to experiment with new ways of talking about the Lean Startup movement and the impact you all are having on the practice of entrepreneurship across countries, industries, and even sizes of companies. We are collectively bringing a new level of scientific rigor to the act of innovation itself, and our revolution is just beginning. We still have much to learn. So take a look at the ideas below, and please leave your thoughts as a comment. As always, I welcome your feedback.

The seed of this idea was planted by many of you on Twitter over the past few weeks. In fact, both I and the SXSW organizers got many emails and tweets about the necessity of having Lean Startup be part of SXSWi. As a result, they were kind enough to allow a very late Lean Startup submission to their user-generated PanelPicker system. They use crowdsourcing to figure out which speakers to invite and what topics are of interest to their audience. Even if you've never been to SXSW or don't know what I'm talking about, you can still go vote - it takes less than five minutes. Because we're getting a very late start on the other panels, our submission is way behind. We only have a few days to catch up, as voting ends on August 27 - just three days from now. So please vote, comment, tweet, and help make this happen. Thank you.

Here's the submission itself, with my first attempt at a new framing for Lean Startup as a rebirth of scientific management. I'd love to know what you think:

The Lean Startup: innovation through experimentation

2011 will mark the one-hundredth anniversary of Frederick Winslow Taylor's "Principles of Scientific Management." The tremendous material abundance we enjoy today is the result of the productivity revolution he unleashed by bringing the tools of science to the study of work itself. Management today is rigorous, scientific, and effective -at the production of physical goods. 
In other areas the picture is bleak, especially for innovative new products. We fail spectacularly in startups and big companies alike. Too often we're building something nobody wants. There is a movement that is trying to eradicate this disease. 
We are at the beginning of a second scientific management revolution that will bring science, rigor, and discipline to the process of innovation itself. It has already begun to transform the way startups are built around the world. It is called the Lean Startup. 
All entrepreneurs face these challenges: 
How do we know if we’re making progress? 
How do we know if customers will want what we’re building? 
How do we know what kind of value we can create? 
Answering requires more than just disciplined thinking at the whiteboard. It requires the coordination of people. In other words, it requires management. The Lean Startup is a management science for entrepreneurs of all kinds. It enables rapid customer-centric iteration. It helps startups test their vision before it's too late. It is a tool for people who want to change the world.
Regardless of what the SXSW organizers decide, I intend to host an event in Austin to coincide with the conference. I'm hoping we'll be able to top last year's Lean Startup Smackdown, which was put on by the Austin Lean Startup Circle. I don't know if it will be more like a party, or more like a mini-conference. In fact, I encourage you to weigh in with a comment. Would you be interested in attending? Co-sponsoring or co-organizing? Or just getting drunk? Let me know.

Most importantly, I want to continue to send you all my thanks for your tremendous support and encouragement. I can honestly say this is something I would never have imagined attempting on my own, and it is the latest in a series of amazing experiences you've all made possible. Thank you. 

Monday, August 16, 2010

Case Study: SlideShare goes freemium

(Normally, I do not write about companies that are doing a marketing launch. But I have decided to make an exception today, for two reasons. First, SlideShare is a fantastic product (that I use on a regular basis) and an impressive company example of Lean Startup practices in action. Second, their story illustrates a key Lean Startup idea: proving the business in micro-scale. It requires separating the product launch from the marketing launch (see Don't Launch) as well as other staple Lean Startup tactics: minimum viable product, split-testing, customer development and the pivot. This story especially demonstrates that these techniques are not reserved only for tiny startups just starting out. When SlideShare began the journey you're about to read, they already had more than a million visitors a day. Because the stakes were high, they had to successfully use a technique I call Innovation inside the box which is important for entrepreneurs inside established companies of all sizes.

Once again, this case study is a collaboration with Sarah Milstein, who conducted the interviews and wrote the post itself, with some minor edits and commentary from me. As this is a new initiative for this blog, we especially welcome your feedback. Did you find this post useful? One recurring request I hear from Lean Startup practitioners around the world is a desire to see examples of the ideas in action. How are we doing?

In the meantime, take a look at how SlideShare performed a significant pivot while still moving at full speed. -Eric]

“The first user experience was actually terrible.” Rashmi Sinha, co-founder and CEO of SlideShare, describes an early version of the analytics package that’s part of the Pro accounts the company announced today.

If your company is using minimum viable product, you’ve probably said the same thing yourself. A lot. SlideShare, founded in 2007, started experimenting with MVPs and A/B testing this year. Those tools, combined with focused customer interviews, have turbo-charged the company’s ability to learn.

What prompted the process change? Early this year, SlideShare launched custom channels. Designed for large businesses, the channels let a company share several types of documents, brand the channel with their own design elements, and then include display advertising, contest promotions, blog aggregation, social media integration and metrics reporting. The idea seemed to SlideShare to be a natural direction. Except it didn’t take off. [I was an early adopter of this feature, and participated in the last marketing launch, as you can see here. Alas, even brilliant marketing adorned with a giant picture of me can't fix the wrong product. -Eric]

Big companies said they liked the idea, but SlideShare found it hard to close deals. Meanwhile, individuals and smaller companies emailed by the hundreds to say that they wanted the features of custom channels, but the sales model—arranged like a media buy—didn’t make sense to them.

SlideShare’s existing customers had needs that the company’s new product—along with its pricing and positioning—simply weren’t solving. Realizing it had taken a wrong turn, SlideShare rethought its approach to premium accounts and ultimately performed what we’d call a value capture pivot, one where the company changes the way it collects revenue from customers.

The process started with a few moving parts. First, the company began quietly testing subscription pricing plans, initially positing a basic plan and an enterprise version. Second, when an individual or small company signed up, Sinha would email them to ask if they’d be willing to hold a phone interview with her to discuss their experience of the product. Despite the fact that SlideShare's product is well-established with many customers, Sinha still took the critical step of (to use Steve Blank's famous phrase) getting out of the building, a particularly important job for founders. Third, SlideShare started holding sales calls with large companies to learn what would prompt them to buy the enterprise version.

“Individuals and small companies wanted analytics, they wanted to know what was happening in social media [for their content], they wanted ad removal and lead gen. Branding was less important to them,” says Sinha. Big companies had other needs. “We didn’t anticipate at all the control features. For instance, we worked with Pfizer, and they wanted the comments turned off. I hadn’t thought that would be a feature. But they’re regulated, so it makes sense.” SlideShare used the two streams of information to segment their market and come up with three plans that recombined the custom channel features in meaningful ways.

But that’s just part of the story. As SlideShare was pivoting, it was also trying out two processes to get better results: 1) A/B testing to refine the pricing plans and the page describing them; and 2) MVPs to hone the actual premium features. The combo helped SlideShare learn a lot in short order. [This is the essential approach to testing a big vision that avoids the "local maximum" trap. See Learning is better than optimization. -Eric]

The company ran landing page splits every two or three days (they initially used Unbounce to generate the pages) and measured them carefully with KISSmetrics. They also used SnapABug for live chat on their site. Between the metrics and the direct customer questions, SlideShare had what Sinha calls “minor learnings and then major shifts.”

For example, early iterations of their pricing page included the original, free version of SlideShare. “We realized that was really confusing people,” says Sinha. “We don’t give you all this Pro plan information right away when you join SlideShare. It’s more like, ‘If you’re already using SlideShare, you might want to try this.’” They removed the core plan, and conversions went up.

The A/B testing did have its challenges. Because SlideShare has more than a million visitors a day, the team is used to developing features that at least 100,000 people will use. “You get used to having a big impact,” says Sinha. With the split tests, maybe 500 people would see an iteration (SlideShare drove traffic with calls to action around their own site). “You have to get ready to deal with much smaller numbers.”

The MVPs were tricky to implement for emotional reasons, too. Because the SlideShare team was used to giving away a high-value product, engineers balked at charging for a clearly imperfect product. The analytics package, for instance, launched in what Sinha calls “a very crude version; we started off and sold it before we were comfortable with it.”

The saving grace was follow-up interviews. SlideShare asked customers what they had expected in the product; the responses were often literal descriptions. People consistently said they were dying for analytics and specifically that they wanted to track social media and understand the people visiting their content (SlideShare eventually discovered that showing visitors’ locations and timing satisfied people’s needs).

“Charging for something half-baked is really interesting,” says Sinha. “It makes the product team uncomfortable. At the same time, you make sure that you get honest feedback. If the product doesn’t meet customers’ expectations, they cancel. It’s a very honest relationship. On analytics, we got a lot of feedback that it was half-baked, that we sold it under false pretense. But we would just respond honestly and fast and say, ‘Tell us what you want.’ Then we’d get back to them when we had built it.” Customers appreciated the follow-up, and many bought again after the feature had evolved. In this regard, SlideShare used the early adopter feedback not only to improve the product, but too improve its understanding of what subsequent customers would want. [That is customer development. -Eric]

The marketing launch for SlideShares Pro accounts is today. But the product launch has been happening iteratively over the past months—which means the company is confident in its new offerings. “When we launched custom channels in February, a lot of people reached out and said, “We’d love to buy’,” recalls Sinha. “But it never happened.” [Alas, customers don't know what they want! -Eric] Since creating and refining its premium accounts, SlideShare has closed a number of deals, including high-profile accounts like Dell, Cisco and Pfizer.

Sinha notes that Eric Schmidt, in a recent interview, said that you find out whether people truly like a product in the second phase after launch. In the first phase, you get a lot of curious people. Only after the buzz has died down do you truly understand what’s going on. With careful and continuous learning processes, SlideShare is inverting that idea and going to market with a validated product. That is the essence of proving the business in micro-scale.

[We'll see if the marketing launch results follow the predictions of SlideShare's validated business model. We wish them the best of luck, and hope we can convince them to share their results - positive or negative - in the near future. In the meantime, good luck and thanks for letting us share your story. -Eric]