Thursday, October 30, 2008

John Doerr's 10 lean startup tips

I just saw video of John Doerr's talk yesterday at VentureBeat’s “How to manage your start-up in the downturn” roundtable event. The tips are based on advice JD solicited from great KPCB entrepreneurs. I was impressed enough to transcribe (and paraphrase) the list. You can see the whole video at the end of this post.
  1. Act now, act with speed.
  2. Protect the vital core of the business.Use a scalpel to make strategic cuts.
  3. Get 18 months or more of cash (runway) in the business against a conservative forecast.
  4. Defer all facilities expansions, capital expenditures. Use Google Apps. Reprioritize & rerationalize all R&D. Defer.
  5. Negotiate. Everything is negotiable in this climate.
  6. Everybody should be selling. Selling is an honorable profession. Everyone from the receptionist to engineers is selling. Not just about expenses, about increasing revenue.
  7. Offer equity instead of cash. Voluntary salary reduction program.
  8. Pay attention to where your cash is. All cash in most secure possible instruments.
  9. Make sure for planned revenues you have "leading indicators" to know if you will hit it.
  10. Over-communicate with employees, investors, customers. Don't sugar coat.
What I find remarkable about this list is how many of them apply to lean startups in good times and in bad. For example, on the "leading indicators" front, I think companies should use a funnel analysis to drive decisions and get rapid feedback on how they are progressing. This can take the form of a traditional sales pipeline or a registration-activation-revenue chart. I've also used the voluntary salary reduction tool - it's particularly useful as a way to get passionate employees who don't have a lot of personal expenses to buy into the mission of the company at a deep level. And even for those employees who can't afford to take much reduction, it's sobering to go through the exercise of deciding if they believe in the company enough to put their own money into it. At IMVU we strongly believed that everyone in the company had to be involved in selling. There's no better way to learn what customers want (or hate!). And it's hard to know if you're truly succeeding without a focus on revenue.

I hope the startups who are struggling with the current downturn will use it as a motivator to make cuts that actually increase their tempo and speed. A crisis can clarify what's important, and getting clear about what's important is the criticial first step to seeing waste clearly (see Lean Thinking). And once you can see waste, you can start to get it out of the way of your execution, and become a truly lean company.



The Entire Video of John Doerr Giving 10 Tips for Start-ups to Avoid the Econalypse

Reblog this post [with Zemanta]

Tuesday, October 28, 2008

A hierarchy of pitches

Every company will need to pitch itself from time to time. Usually we think of pitches in the context of raising money, but that is only one of many pitch situations. We pitch to potential partners, vendors, publishers, conferences, employees, and even lawyers. It's different from selling a product, because it is not part of our regular business practice, is not something that relates to our core competence, and tends not to happen in a repeatable and scalable way. (I'll exclude those non-lean startups who basically exist for the purpose of raising bigger and bigger sums of money. You're not one of those are you?)

Most of the times I have seen pitches fail, it is not because they are poorly written, or that the entrepreneur lacks passion. It is because they don't answer the right question. My favorite example of all time comes from students in an entrepreneurship class. Their idea was to build a next-generation autonomous robot, that could be used by defense and security agencies around the world. The whole pitch was about how valuable robots could be in the future. They even included a slide with The Transformers on it. Now there was nothing wrong with their analysis: anyone who invents a technology as sophisticated as The Transformers is definitely going to make a lot of money. But these students completely failed to address the one and only question on their audience's mind: can you three guys really build the robots of the future? (Turns out, they were incredibly well-credentialed graduate students who had, in fact, developed some interesting new robotics technology. But you wouldn't have known that from their pitch.)

I have come to believe that there is a hierarchy of pitches, and that understanding where your pitch falls on this spectrum can assist in making decisions about what information to highlight. Pitches higher in the hierarchy tend to be more successful, and so if you can fit your company into one of those categories, you can get better results or better terms. Now, just because you can do a thing, doesn't mean you should - and there are plenty of other great resources out there that can help you think through whether and when to raise money (or do other kinds of deals).

With that disclaimer out of the way, here's how I order the hierarchy of pitches:

Printing money
Key questions: are those numbers real? how big is the market? can your team execute the growth plan?
Most important slide: valuation

Promising results
Key questions: can you monetize that traffic? (or drive traffic to that profitable destination?) do you know why you've achieved those results?
Most important slide: hockey stick

Micro-scale results
Key questions: who is the customer, and how do you know? what is the potential market size? what are the business economics?
Most important slide: lessons learned

Working product
Key questions: what does the product do? what's the launch plan? who's on the marketing team?
Most important slide: live demo

Prototype product
Key questions: what will it take to ship a working product? how do you know anyone would want it? who's on the engineering team?
Most important slide: demo (if the product solves an obvious problem), engineering resumes (if the product is nearly impossible to build), "day in the life of a customer" (if neither of the above)

Breakthrough technology
Key questions: who owns the patents? can we make a product out of this technology? are there any good substitutes?
Most important slide: barriers to entry

All-star team
Key questions: has the team made money for their investors in the past? are they domain experts? are they committed to an idea in their domain of expertise?
Most important slide: problem we are trying to solve

Good product idea
Key questions: what kinds of risk does this company need to mitigate (technology risk, market risk, team risk, funding risk)? is it a revolutionary and novel idea? is this team the one to back? can the team bring the product to market? who is the customer? who is the competition? will they fail fast?
Most important slide: about the founders


In a pitch meeting, try to spend as much time as possible talking about the key questions for your pitch. If you find yourself getting asked non-key questions, try to use your answers to steer the conversation back to the key questions. But here's the most important part: if you keep getting non-key questions over and over again, something is wrong with your pitch. Either you misunderstand where your pitch fits into the hierarchy, or you are not using the early part of your pitch to establish it. Don't keep banging your head against the wall - if you can't convince your potential partners that your startup is printing money, try to figure out why. Experiment with different narratives. If you still can't do it, move one level down the hierarchy and see if you can make that story stick.

One last piece of advice: don't forget that potential partners are evaluating the strength of your pitch, not you. It's not true that companies with pitches further up the hierarchy are better, in some absolute sense, than companies further down. It's just that they have an easier time closing these kinds of deals. If you can't close the deal, maybe your company is at the wrong stage of its development, and it's time to try a different tack.

Wednesday, October 22, 2008

Chuck's Code & Learning: Learning to write tests that matter

Chuck's Code & Learning: Learning to write tests that matter
Don't write tests for scenarios just because you can. Write tests that support a specific business need.
Amen.

Tuesday, October 21, 2008

Lean startups vs lean companies

Venture Hacks has a great article today on lean startups. They quote extensively from two of our most important thinkers: Taiichi Ohno, creator of the Toyota Production System, and Kent Beck, creator of extreme programming. If you're not familiar with their work, and how it relates to startups, this is a great place to start. Go read it.

I want to take up one of the questions posed in the Hacker News discussion of the article:
The question in my mind against most of these efficiency driven programs is simple. If you're always looking to eliminate waste and become super efficient, you're not spending any time being creative or chasing radical ideas that may or may not be worth the effort spent on executing them.

Innovation sometimes requires a lot of wasteful experimentation and it looks like that is completely anti-thetical to the whole efficiency argument.

This is a common sentiment that derails many efficiency programs. But I do think it is a misconception of what lean startups are all about. Part of the problem is the distinction (that I owe to Steve Blank) between lean startups and lean companies. In a typical lean company, waste is defined as "every activity that does not create value for the customer." And this is 100% correct. By driving this kind of waste out of your company, you actually boost creativity by eliminating beaurocracy, busy work, unnecessary hierarchy, and, of course, excess inventory.

But statups require a special kind of creativity: disruptive innovation. And, as the commenter rightly points out, this is not really a matter of efficiency. By the standard of "customer value," most innovation-seeking experiments are waste. Lean startups operate by a different standard. I suggest they define waste as "every activity that does not contribute to learning about customers." (aka "how you get to product/market fit.") This is why I find the concept of the Ideas-Code-Data feedback loop so helpful. Any activity that actively promotes us getting through each iteration (including the learning phase) faster, is value-creating. Everything else (including a lot of traditional "agile" or "lean" tactics) is waste.

Of course, lean startups and lean companies both follow the same underlying principles. It's just the implementation that changes, as focus shifts from learning to execution. And, as many commenters noted, there are great companies that successfully incorporate innovation into their execution discipline (Toyota first among them). I would say that the true standard of waste is "anything that does not carry out the company's mission." But I think that is too abstract to be really useful for most startups, since it's hard to realize that the mission changes as the company goes through the natural phase changes of growth.

Monday, October 20, 2008

The engineering manager's lament

I was inspired to write The product manager's lament while meeting with a startup struggling to figure out what had gone wrong with their product development process. When a process is not working, there's usually somebody who feels the pain most acutely, and in that company it was the product manager. Last week, I found myself in a similar situation, but this time talking to the engineering manager. I thought I'd share a little bit about his story.

This engineering manager is a smart guy, and very experienced. He has a good team, and they've shipped a working product to many customers. He's working harder than ever, and so is his team. Yet, they feel like they are falling further and further behind with every release. The more features they ship, the more things that can go wrong. Systems seem to randomly fail, and as soon as they are fixed, a new one falls over.

Even worse, when it comes time to "fix it right" the team gets pushback from the business leaders, who want more features. If engineers want more time to spend making their old code more pretty, they are invited to do so on the weekends.

Unfortunately, the weekends are already taken, because features that used to ship on a friday now routinely cause collateral damage somewhere else on the platform, and so the team is having to work nights and weekends cleaning up after each launch.

Every few months, the situation comes to a head, where the engineers finally scream "enough!" and force the whole company to accept a rewrite of some key system. The idea is that once we move to the new system (or coding standard, or API, or ...) then all the problems will be solved. The current code is spaghetti, but the new code will be elegant. Sometimes the engineers win the argument, and sometimes they are overruled. But it doesn't seem to matter. The rewrite seldom works, and even when it does, a few months later they are back in the same dilemna, finding their now-old code falling apart. It's become "legacy code" and part of the problem.

It's tempting to blame the business leaders ("MBA-types") for this mess. And in a mess this big, there is certainly blame to go around. But they are pushing for the things that matter to customers - features. And they are cognizant that their funding is limited, and if they don't find out which features are absolutely critical for their customers soon, they won't be able to survive. So they are legitimately suspicious when, instead of working on adding monetization to their product, some engineer wants to take a few weeks off to go polish code that is supposed to be already done.

What's wrong with this picture? And why is the engineering manager suffering so badly? I can relate to his experience all too well. When I was working my first programming jobs, I was introduced to the following maxim: "time, quality, money - pick two." That was the watchword of our profession, and I was taught to treat with disdain those "suits" who were constantly asking for all three. We treated them like they had some kind of brain defect. If they wanted a high-quality product done fast, why didn't they want to pay for it? And if money was so tight, why were they surprised when we cut corners to get the product out fast? Or went past the deadline to get it done right?

I really believed this mantra, for a time. But it started to smell bad. In one company, we had a QA team as large as our engineering team, dozens of people who worked all day every day to find and report bugs in our prodcut. And this was a huge product, which took years to develop. It was constantly slipping, because we had a hard time adding new features while trying to fix all the bugs that the QA team kept finding. And yet, it was incredibly expensive to have all these QA testers on staff, too. I couldn't see that we were managing to pick even one. Other, more veteran programmers told me they had seen this in many companies too. They just assumed it was the way software companies worked.

Suffice to say, I no longer believe this.

In teams that follow the "pick two" agenda, which two has to be resolved via a power play. In companies with a strong engineering culture, the engineers pick quality. It's their professional pride on the line, after all. So they insist on having the final say on when a feature is "done" enough to show to customers. Business people may want to speed things up by spending more money, but enough people have read the Mythical Man-Month to know that doesn't work.

In teams that have a business culture, the MBA's pick time. After all, our startup is on a fixed budget. They set deadlines, schedules, and launch plans, and expect the engineering team to do what it takes to hit them. If quality suffers, that's just the way it is. Or, if they care a lot about quality, they will replace anyone who ships without quality. Unfortunately, threats work a lot better at incentivizing people to CYA than getting them to write quality software.


A situation where one faction "wins" at another's expense is seldom conducive to business success. As I evolved my thinking, I started to frame the problem this way: How can we devise a product development process that allows the business leaders to take responsibility for the outcome by making conscious trade-offs?

When I first encountered agile software techniques, in the form of extreme programming, I thought I had found the answer. I explained it to people this way: agile lets you make the trade-offs visible to whole company, so that they can make informed choices. How? By shipping software early, you give them continuous feedback about how it well it's working. They can use the software themselves, since every iteration produces working (if incomplete) code. And if they want to invest in higher quality, they can. But, if they want to invest in more experiments (features), they can do that too. But in neither case should they be surprised by the result. Sound good?

It didn't work. The business leaders I've run this system with ran into the same traps as I had in previous jobs. I had just passed the burden on to them. But of course they didn't feel reponsible for the outcome - that was my job. So I wound up with the worst of both worlds: handing the steering wheel over to someone else, but then still being blamed for the bad results.

Even worse, agile wasn't really helping me ship higher quality software. We were using it to get features to market faster, and that was working well. But we were cutting corners in the development methodology as well as in the code, in the name of increased speed. But because we had to spend more and more time fixing things, we started slowing down, even as we tried to speed up. That's the same pain the engineering manager I met with was experiencing. As the situation deteriorates, he's got to work harder and harder just to keep the product from regressing.

It was my own failure to ship quality software in the early days of IMVU that really got me thinking about this problem in a new way. I now believe that the "pick two" concept is fundamentally flawed, and that lean startups can achieve all three simultaneously: quickly bring high-quality software to market at low cost. Here's why.

First of all, it's a myth that cutting corners saves time. When we ship software with defects, we wind up having to waste time later dealing with those defects. If each new feature contains a few recurring problems, then over time we'll become swamped with the overhead of fixing and won't be able to ship any new features.

So far, that sounds like just another argument for "doing things right" the first time, no matter how long they take. But that's problematic too. The biggest form of waste is building software nobody wants. The second biggest form of waste is fixing bugs in software nobody wants. If we defer fixing bugs in order to bring a product to market sooner, and this allows us to find out if we're on the right track or not, then it was worthwhile to ship with those bugs.

Here's how I've resolved the paradox in my own thinking. There are two kinds of bugs:
  • One kind are what I call defects: situations where the software doesn't behave in a predictable way. Examples: intermittently failing code, obfuscated code that's hard to use properly, code that's not under test coverage (and so you don't know what it does), bad failure handling, etc.

  • The second kind of bugs are the type your traditional QA tester will find: situations where the software doesn't do what the customer expects. For example, you click on a button labeled "Quit" and in response the software erases your hard drive. That's a bug. But if the software reliably erases your hard drive every time, that's not a defect.
The resolution to the paradox is to realize that only defects cause you future headaches, and cannot be deferred. That's why we need continuous integration and test-driven development. Whenever we add a feature or fix a bug, we need to make sure that code never goes bad, never mysteriously stops working. Those are the kinds of indefinite costs that make our team grind to a halt over time. Traditional bugs don't - you can choose to fix them or not, depending on what your team is trying to accomplish.

Defects are what make refactoring difficult. When you improve code, you always test it at least a little bit. If what you see is what will ultimately make it into production, it's pretty easy to make sure you did a good job. But code that is riddled with defects is a cameleon - one moment it works, the next it doesn't anymore. That leads to fear of refactoring, which leads to spaghetti code.

By shipping code without defects, the team actually speeds up over time. Why? Because we never have to revisit old code to see why it stopped working. We can add new team members, and they can get started right away (as an aside, new engineers at IMVU were always required to ship something to customers on their first or second day). And the whole team is gettng smarter and learning, so our effectiveness increases. Plus, we get the benefit of code reuse, and all the great libraries and tools we've built. Every iteration, we get a little more done.

So how can I help the engineering manager in pain? Here's my diagnosis of his problem:
  • He has some automated tests, but his team doesn't have a continuous integration server or practice TDD. Hence, the tests tend to go stale, or are themselves intermittent.
  • No amount of fixing is making any difference, because the fixes aren't pinned in place by tests, so they get dwarfed by the new defects being introduced with new features. It's a treadmill situation - they have to run faster and faster just to stay at the level of quality/features they're at today.
  • The team can't get permission from the business leaders to get "extra time" for fixing. This is because the are constantly telling them that features are done as soon as they can see them in the product. Because there are no tests for new features (or operational alerts for the production code), the code that supports those new features could go bad at any moment. If the business leaders were told "this feature is done, but only for an indeterminate amount of time, after which it may stop working suddenly" they would not be so eager to move on to the next new feature.
Here's what I've counseled him to try:
  • Get started with continuous integration. Start with just one test, a good one that runs reliably, and make sure it gets run on every checkin.
  • Tie the continuous integration server in with source control, so that nobody can check in while the tests are failing.
  • Practice five why's to get to the root cause of future problems. Use those opportunities to add tests or alerts that would have prevented that problem. Make the investment proportional to the problem caused, so that everyone (even the business leaders) feels good about it.
My prediction is that these three practices will quickly change the feeling around the office, because the most important code will wind up under test quite soon (after all, it's the code that people care the most about, and so when it fails, the team notices and fixes it right away). With the most important code under test, the level of panic when something goes wrong will start to decrease (because it will tend to be in less important parts of the product). And has the tension goes down, it will be easier to get the whole team (including the MBA's) to embrace TDD and other good practices as further refinements.

Good luck, engineering manager. May your team, one day soon, refactor with pride.
Reblog this post [with Zemanta]

Saturday, October 11, 2008

Three decisions to make on virtual goods

I was invited to the Virtual Goods Summit yesterday, and got to see quite a few interesting speakers and panels. It got me thinking about what decisions are essential when building a virtual goods product.

I'm really proud of the virtual goods system we built at IMVU, and a whole day of presentations left me feeling very confident in IMVU's durable competitive advantage. Most of the breakthroughs we had at IMVU were made possible because we had really good people grappling with really difficult problems. Those problems were the necessary consequences of some decisions we made early on. I'd love to take credit for the good work that came later, but that's not really fair. The credit is due to much smarter people than me, and to the incredible power of necessity, that mother of invention.

Here's what I mean. I believe there are only a few key decisions to make about virtual goods. These decisions cause problems that will challenge you for years, if you are lucky enough to be successful with your product. If you know what you're getting into, you really don't need to solve those problems in advance. As the size of your community and economy grows, you'll be able to solve them as they arise, as long as you know what to look for.

What makes a particular decision key? Those that require extreme focus and which are hard to reverse later. Of course, no choices in the real world are completely binary. At IMVU, where we are known for our incredible user-generated content, we also have some first-party content, that was important in seeding our catalog. But the work we've done to enable UGC dwarfs the effort we've put into first-party content, so much so that I think it has rightfully consumed almost all of our focus. Hence, this caveat. I recommend that you focus on one answer to each of these questions, even if you dabble in the other possibility.

User-generated content (UGC) or first-party content?
Some products, like Habbo Hotel or World of Warcraft, employ professional artists and designers to create compelling virtual goods that players want to buy. Others, like IMVU or Second Life, rely almost exclusively on third-party developers to create goods.

If you go the UGC route, be prepared to learn an awful lot about copyright and trademark law. Be prepared to deal with excruciating judgements about what kind of content is appropriate for your audience. Your challenge will be incentivizing your developers to create, and managing the volume of their output. How do you help customers find the best of the best? And what do you do with the rest? You may have to staff up a large customer service department to review and approve all of the items. And you may find yourself at the whim of some pretty unstable people. Yes, I have had to deal with developers on strike. Are you ready for that?

If you go the first-party route, you can try a different set of problems. You do get the beneift of total control of your content, so you can guarantee your customers a quality experience. However, virtual goods is a hits-driven business, which means you must constantly figure out what kind of content your customers want. Sales depend on being right (whereas in UGC we can pretty much try anything, and promote the winners). Your platform may require you to staff up a large production department to crank out enough content to satisfy your customers. You may have to take template-based shortcuts ("500 colors of the same basic shirt"). You may also struggle to keep up with changes in your audience. If your early adopters are interested in one kind of thing, but your mainstream audience has a different set of tastes, you may find yourself with a lot of rework on your hands. Can your platform really accomodate emo and anime? Princess and goth?

One last note on UGC vs first-party content, especially for you who are thinking of going the first-party route. Be careful not to get yourself accidentally into the UGC business. Humans are incredibly resourceful, and they really like to express themselves. Beware the trap that Google Lively fell into. They thought they had UGC under control, because they prohibited any user-generated 3D content or animations. They did allow people to create chat rooms and set the topic to whatever they wanted. Guess what they wanted to talk about. Lively's launch was marred by zillions of sex chat rooms, with people spending hours and hours trying to figure out how to use Lively's stock animations to make their avatars do things that looked kinky.

Subscription or a la carte payments?
I notice that people sometimes forget that one of the most profitable virtual goods businesses in the world is Blizzard's World of Warcraft, even though they don't allow customers to trade their hard-earned cash for virtual goods directly. Instead, they charge a flat-rate subscription, and then manage goods using a combination of gameplay, an internal currency and various trading systems. Most systems that come out of the video game world use subscriptions, as do products targeted primarily at kids. Why? Subscriptions help keep gameplay balanced between those with lots of money and those with less, which is key for products focused on fun. Subscriptions also constrain payment-method choices. In the US, it generally means getting parents involved, which is great for kid-focused products (that need parents anyway) but not necessarily great for products targeted at teenagers. On the other hand, they have a huge advantage when it comes to chargebacks, because regular subscribers provide a huge base of "safe" transactions that form a stable denominator for chargeback ratios.

Other sites, like IMVU, Mob Wars and many others allow customers to buy items a la carte, one at a time (or in bundles) as an upsell to the standard experience. This is especially important for teenagers, because it allows them to participate in products without having to go through the complex negotiation with their parents that a subscription usually requires. If you support mobile or prepaid cards for billing, they can even do it on their own, without approval, just using their discretionary income. If you go this route, be prepared to make money a dollar at a time, and to work with lots and lots of payment vendors. You're also going to have to deal with a lot more fraud and chargeback issues, because you'll have to work harder to forge the kind of long-term billing relationship subscription businesses enjoy. On the other hand, you can service customers that are hard to reach: the unbanked, international customers without credit cards, and teenagers everywhere.

Merchandising or gameplay?
In a game like World of Warcraft, Kart Racer, or Mob Wars, customers don't struggle to discover new items. It's obvious what they want to buy, because the objects confer direct benefits that are rooted in the gameplay mechanics. The challenge is to balance the benefits that paying customers get with the benefits that customers get by investing three other attributes: passion, time and skill. An unbalanced game can lose all of its customers (paying or otherwise) rapidly.

For other products, customers are generally shopping for virtual goods in some kind of catalog. If you go this route, be prepared to make a long-term investment in the skill of merchandising. If you have a non-trivial amount of goods for people to buy, they are going to have to find ways to sort through them to find the ones they want. You'll need bundles, promotions, category and segment managers, and algorithms for upsell, cross-sell, and conversion optimization. In other words, a marketing department.


You can use these three questions to analyze existing businesses. For example, IMVU is a user-generated, a la carte, merchandising product. Habbo is first-party, a la carte, merchandising. Mob Wars is first-party, a la carte, gameplay. WoW is first-party, subscription, gameplay. I hope it also proves useful to people thinking about creating a new virtual goods product. My belief is that many of the decisions you make later will flow from these early ones.

Here's how we thought about it in the early days of IMVU. We first tackled the issue of UGC vs first-party content. This one was easy for us, because our values committed us to try to make freedom of expression core to our product. It was part of our mission statement. Of course, it didn't hurt that we only had a small team and not a lot of money, and third-parties gave us a lot of leverage. As for subscriptions, we wanted to have them from the beginning, but decided against them. Here was our reasoning. Fundamentally, we wanted our third-party developers to make money from their time invested in IMVU. So we did this thought experiment: we imagined a customer who was willing to buy a virtual shirt from one of our developers, but couldn't afford to pay us the subscription fee. Would we really turn them away? We decided the answer was no - our developers interests had to come first. So we chose a la carte. Last, we tackled the gameplay question. We knew IMVU would be a socializing platform, and so we decided that we would let our customers interact using their own rules for assigning status, like we do in the physical world. The government doesn't assign each clothing brand its own level of benefit. The companies that own those brands work hard to differentiate their offering so that people who wear their clothes will be perceived in a certain way. We wanted to let our developers do the same thing, so we opted not to go the gameplay route.

That's how we formed our initial IMVU hypothesis. Very few of the decisions we made in those early days wound up, years later, affecting the outcome of the product. But these three did. I hope sharing them will help others who are thinking of working with virtual goods. If that's you, share your experience in a comment. Did you find these questions helpful? And for those of you already experienced with virtual goods, what do you think is missing?

Tuesday, October 7, 2008

The App Store after the gold rush

I wrote earlier about the issue of distribution advantage on the iPhone. As the gold rush drives thousands of apps onto the platform, it's getting harder and harder for new entrants to get oxygen. Take a look at The App Store after the gold rush - FierceDeveloper:
According to a recent BusinessWeek feature, the flood of new games, productivity tools and related iPhone software is making it difficult for the vast majority of apps to crack the consumer consciousness. A number of developers are slashing their prices to remain competitive, but it appears that the gold rush that followed on the heels of the App Store's July 10 grand opening is already over, and the get-rich-quick stories of developers like Steve Demeter--who reportedly raked in $250,000 in just two months for his iPhone game Trism--have already passed into coder lore.
The App Store is a channel for customer acquisition. As the channel gets more and more crowded, just launching an app in the store is getting worse and worse as a strategy for each new entrant. This is completely analogous to the situation elsewhere on the internet, where launching a new website, product, or service with PR is getting harder and harder. Customers and prospects are overwhelmed by the number of media and companies clamoring for their attention. If your launch is not immediately successful, you quickly fall into oblivion. On the App Store, the same dynamic is in play. If your app doesn't immediately make it into the Top 25 page, it's pretty hard to have any kind of durable growth.

So what can you do? I think it's helpful to think about two kinds of competition for distribution: acquisition competition and retention competition.

Acqusition competition is how new apps get new customers. On the web, we have many of these channels: SEM, SEO, world of mouth, PR and viral. On the iPhone, it seems that two are driving most of the installs: the "newest apps" RSS feed (which may be combined with PR) and a primitive form of SEO, when people search the App Store for a specific kind of app. Over time, we should get more channels that service the long tail of apps for which the current channels are not working. For example, if any of the mobile ad networks gets major traction, they may become a dominant way that people discover new apps.

Retention competition is how you get people to come back to your app. The primary place this competition is visible is on the home screen of the iPhone itself. But the real battle is in the mind of the people who have installed your application. What causes them to come back to your app, instead of spending their time doing something else? Are they turning on their phone specifically to get your app? Do they browse around looking for an app to pass time? Does your app solve a specific problem that they have? Do you have a way to notify them by SMS or email when something notable happens?

The reason I think it's important to think about retention competition when you are thinking about acquisition is that it strongly influences your acquisition options. If your app has incredibly strong retention, you will probably do very well with the current PR/new app system of acquisition. Why? Because you'll be able to leverage your strong retention to stay in the Top 25 list, which will lead to strong acquisition, in a nice positive feedback loop. If your app has strong word-of-mouth or viral components, your retention drives new acquisition, and it's not so important to have good placement in the store. If and when a good SEM solution shows up for iPhone, you may be able to use it to artificially drive your app into the Top 25, as a one-time event. Then, if your retention is good enough, you can stay there. Or if your lifetime value is high enough, you can just keep spending on SEM.

So if you have a new app that you are thinking of launching, what should you do? My advice: don't launch big. Don't do PR upfront, don't put out a press release. Figure out how to launch quietly, so you can find out what your retention and referral rates are going to be. If necessary, consider doing this under a different brand name than the one you are wedded to using. Having that data will let you pick an acquisition strategy that is appropriate for your app. It's like knowing the future.


Monday, October 6, 2008

When NOT to listen to your users; when NOT to rely on split-tests

There are three legs to the lean startup concept: agile product development, low-cost (fast to market) platforms, and rapid-iteration customer development. When I have the opportunity to meet startups, they usually have one of these aspects down, and need help with one or two of the others. The most common need is becoming more customer-centric. They need to incorporate customer feedback into the product development and business planning process. I usually recommend two things: try to get the whole team to start talking to customers ("just go meet a few") and get them to use split-testing in their feature release process ("try it, you'll like it").

However, that can't be the end of the story. If all we do is mechanically embrace these tactics, we can wind up with a disaster. Here are two specific ways it can go horribly wrong. Both are related to a common brain defect we engineers and entrepreneurs seem to be especially prone to. I call it "if some is good, more is better" and it can cause us to swing wildly from one extreme of belief to another.

What's needed is a disciplined methodology for understanding the needs of customers and how they combine to form a viable business model. In this post, I'll discuss two particular examples, but for a full treatment, I recommend Steve Blank's The Four Steps to the Epiphany.




Let's start with the "do whatever customers say, no matter what" problem. I'll borrow this example from randomwalker's journal - Lessons from the failure of Livejournal: when NOT to listen to your users.
The opportunity was just mind-bogglingly huge. But none of that happened. The site hung on to its design philosophy of being an island cut off from the rest of the Web, and paid the price. ... The site is now a sad footnote in the history of Social Networking Services. How did they do it? By listening to their users.
randomwalker identifies four specific ways in which LJ's listening caused them problems, and they are all variations on a theme: listening to the wrong users. The early adopters of LiveJournal didn't want to see the site become mainstream, and the team didn't find a way to stand up for their business or vision.

I remember having this problem when I first got the "listening to customers" religion. I felt we should just talk to as many customers as possible, and do whatever they say. But that is a bad idea. It confuses the tactic, which is listening, with the strategy, which is learning. Talking to customers is important because it helps us deal in facts about the world as it is today. If we're going to build a product, we need to have a sense of who will use it. If we're going to change a features, we need to know how our existing customers will react. If we're working on positioning for our product, we need to know what is in the mind of our prospects today.

If your team is struggling with customer feedback, you may find this mantra helpful. Seek out a synthesis that incorporates both the feedback you are hearing plus your own vision. Any path that leaves out one aspect or the other is probably wrong. Have faith that this synthesis is greater than the sum of its parts. If you can't find a synthesis position that works for your customers and for your business, it either means you're not trying hard enough or your business is in trouble. Figure out which one it is, have a heart-to-heart with your team, and make some serious changes.




Especially for us introverted engineering types, there is one major drawback to talking to customers: it's messy. Customers are living breathing complex people, with their own drama and issues. When they talk to you, it can be overwhelming to sort through all that irrelevant data to capture the nuggets of wisdom that are key to learning. In a perfect world, we'd all have the courage and stamina to perservere, and implement a complete Ideas-Code-Data rapid learning loop. But in reality, we sometimes fall back on inadequate shortcuts. One of those is an over-emphasis on split-testing.

Split-testing provides objective facts about our product and customers, and this has strong appeal to the science-oriented among us. But the thing to remember about split-testing is that it is always retrospective - it can only give you facts about the past. Split-testing is completely useless in telling you what to do next. Now, to make good decisions, it's helpful to have historical data about what has and hasn't worked in the past. If you take it too far, though, you can lose the creative spark that is also key to learning.

For example, I have often fallen into the trap of wanting to optimize the heck out of one single variable in our business. One time, I became completely enamored with Influence: The Psychology of Persuasion (which is a great book, but that's for another post). I managed to convince myself that the solution to all of our company's problems were contained in that book, and that if we just faithfully executed a marketing campaign around the principles therein, we'd solve everything. I convinced a team to give this a try, and they did tried dozens of split-test experiments, each around a different principle or combination of principles. We tried and tried to boost our conversion numbers, each time analyzing what worked and what didn't, and iterating. We were excited by each new discovery, and each iteration we managed to move the conversion needle a little bit more. Here was the problem: the total impact we were having was miniscule. It turns out that we were not really addressing the core problem (which had nothing to do with persuasion). So although we felt we were making progress, and even though we were moving numbers on a spreadsheet, it was all for nothing. Only when someone hit me over the head and said "this isn't working, let's try a radically new direction" did I realize what had happened. We'd forgotten to use the all the tools in our toolbox, and lost sight of our overarching goal.

It's important to be open to hearing new ideas, especially when the ideas you're working on are split-testing poorly. That's not to say you should give up right away, but always take a moment to step back and ask yourself if your current path is making progress. It might be time to reshuffle the deck and try again.

Just don't forget to subject the radical new idea to split-testing too. It might be even worse than what you're doing right now.




So, both split-testing and customer feedback have their drawbacks. What can you do about it? There are a few ideas I have found generally helpful:
  • Identify where the "learning block" is. For example, think of the phases of the synthesis framework: collecting feedback, processing and understanding it, choosing a new course of action. If you're not getting the results you want, probably it's because one of those phases is blocked. For example, I've had the opportunity to work with a brilliant product person who had an incredible talent at rationalization. Once he got the "customer feedback" religion, I noticed this pattern: "Guys! I've just conducted three customer focus groups, and, incredibly, the customers really want us to build the feature I've been telling you about for a month." No matter what the input, he'd come around to the same conclusion as before.

    Or maybe you have someone on your team that's just not processing: "Customers say they want X, so that's what we're building." Each new customer that walks in the door wants a different X, so we keep changing direction.

    Or consider my favorite of all: the "we have no choice but to stay the course" pessimist. For this person, there's always some reason why what we're learning about customers can't help. We're doomed! For example, we simply cannot make the changes we need because we've already promised something to partners. Or the press. Or to some passionate customers. Or to our team. Whoever it is, we just can't go back on our promise, it'd be too painful. So we have to roll the dice with what we're working on now, even if we all agree it's not our best shot at success.

    Wherever the blockage is happening, by identifying it you can work on fixing it.

  • Focus on "minimum feature set" whenever processing feedback. It's all too easy to put together a spec that contains every feature that every customer has ever asked for. That's not a challenge. The hard part is to figure out the fewest possible features that could possibly accomplish your company's goals. If you ever have the opportunity to remove a feature without impacting the customer experience or business metrics - do it. If you need help determining what features are truly essential, pay special attention to the Customer Validation phase of Customer Development.

  • Consider whether the company is experiencing a phase-change that might make what's made you successful in the past obsolete. The most famous of these phase-change theories is Crossing the Chasm, which gives very clear guidance about what to do in a situation where you can't seem to make any more progress with the early-adopter customers you have. That's a good time to change course. One possibility: try segmenting your customers into a few archetypes, and see if any of those sounds more promising than another. Even if one archetype currently dominates your customer base, would it be more promising to pursue a different one?
As much as we try to incorporate scientific product development into our work, the fact remains that business is not a science. I think Drucker said it best. It's pretty easy to deliver results in the short term or the long term. It's pretty easy to optimize our business to serve one of employees, customers or shareholders. But it's incredibly hard to balance the needs of all three stakeholders over both the short and long-term time horizon. That's what business is designed to do. By learning to find a synthesis between our customers and our vision, we can make a meaningful contribution to that goal.

Sunday, October 5, 2008

The product manager's lament

Life is not easy when you're working in an old-fashioned waterfall development process, no matter what role you play. But I have a special sympathy for the "product manager" in a startup that is bringing a new product to a new market, and doing their work in large batches. I met one recently that is working on a really innovative product, and the stories I heard from their development team made me want to cringe. The product manager was clearly struggling to get results from the rest of the team. These are smart people trying hard to all row in the same direction. So why are they having so much difficulty?

Let's start with what the product manager does. He's supposed to be the person who specifies what the product will do. He writes detailed specs which lay out exactly what features the team should build in its next iteration. These specs are handed to a designer, who builds layouts and mockups of all the salient points. Then the designs are handed to a team of programmers with various specialties. Each specialist takes up his part of the spec (UI, middleware, backend) and cranks out code. Last, the QA team builds a test plan based on the spec, and then tests the features to make sure they conform to the plan.

This system naturally lends itself to a pipeline approach, which the product manager organizes. While the programmers are off building the next major feature, he is busy writing specs so that, when they finish, there won't be any idle time before they can start the next iteration. If the programmers go idle, it's bad news for the product manager, because he's supposed to keep them busy building product. Otherwise, the company is wasting serious money.

When I met this team, some acrimony had built up. The last few features came out pretty different from what was origianlly spec'd, and took far too long, to boot. The programmers keep asking for more say in the designs and direction that they work on. So the team has been spending more and more time in the spec and design phases, trying to get team buy-in on what they are going to build. For some reason, though, despite the increased buy-in, the final product often doesn't look anything like the original spec. The VP Engineering spends all of his time trying to make sure the programmers understand and implement the spec. Each iteration takes longer than the previous one. Frustration is mounting.

It doesn't take long to discover that the product manager is being forced to write every spec five times. First, he writes it nice and clear. Next he works with the designer to build out the design spec, and with QA to build out the test plan. When the programmers get it, they often start negotiating with him about what's going to be built. They exchange countless emails, and he's being constantly interrupted and being asked to clarify exactly what the spec means. The fourth spec exists only in these emails, which are changing the design in an ad-hoc fashion. By the time QA gets the feature, their test plan is badly out of date. So the product manager winds up actually having to use the software, by hand, updating the spec and helping create a new test plan. Naturally, the deviations from the spec are so severe, that he has to rewrite the spec he's currently working on (for the next major feature) to take them into account.

Ironically, this system was designed to keep each functional group 100% utilized, so nobody goes idle, including the product manager. But as the iterations get longer, he's spending more and more of his time answering questions. The interruptions are so bad, he has to write his new specs at 3AM, just to keep the pipeline full.

What's wrong with this picture? Everyone is working at 110%, with full dedication. But the team is falling further and further behind.

Here are the changes I'm working with this team to implement
  • Work in cross-functional teams. Each team has a representative of each function. To start, we'll try a product manager, designer, programmer or two, and QA. The team owns a complete feature end-to-end.
  • Focus on speed of iteration rather than utilization of every function. Let people go idle, if they can't help the current iteration succeed. I'm contiuously amazed how many people have untapped creativity and resourcefulness. They don't want to be idle. By letting them focus on the success of their team exclusively, you empower them to do whatever it takes to make the team successful. Will that mean someone in design will jump in to help QA get the release certified? We'll see.
  • Switch the spec process from push to pull. Start with a one-page spec, no more. Then, let the team ask questions of the product manager whenever they need clarification. In exchange, the team agrees to show each piece of working code to the product manager for his approval. They'll find points of disagreement much faster, and resolve them in realtime. Plus, the team will get better and better at interpreting the concise specs that only have to be written once. (Eventually, they may abolish specs altogether)
There's much more this team can do to eliminate waste in the way that they work and thereby iterate faster. Eventually, I hope to get them on a full agile diet, with TDD, scrums, sprints, pair programming, and more. But first I think we need to save the product manager from that special form of torture only a waterfall product development team can create. Once the different parts of the team can trust each other again, we'll have the basis we need to start a true continuous improvement feedback loop.

Saturday, October 4, 2008

About the author

(Update February, 2011: This post originally dates from October, 2008 back when I first started writing this blog. I've updated the "official" conference bio below but otherwise the text remains unchanged from that original essay.)

The most common feedback I've heard from readers has been that I should provide details on my background. I didn't include much on the blog at first, because I want you to judge what I write based on what I say, rather than who I am. So if you're new, consider not paying any attention to the rest of this post, and just diving into the archives, if you haven't already. (Maybe you'd like to start with The lean startup, How to listen to customers, or What does a startup CTO actually do?)

For everyone else, here's the standard bio paragraph I use for conferences and other formal occasions:
Eric Ries is the creator of the Lean Startup methodology and the author of the popular entrepreneurship blog Startup Lessons Learned. He previously co-founded and served as Chief Technology Officer of IMVU. In 2007, BusinessWeek named Ries one of the Best Young Entrepreneurs of Tech and in 2009 he was honored with a TechFellow award in the category of Engineering Leadership. He serves on the advisory board of a number of technology startups, and has worked as a consultant to a number of startups, companies, and venture capital firms. In 2010, he became an Entrepreneur-in-Residence at Harvard Business School.

He is the co-author of several books including The Black Art of Java Game Programming (Waite Group Press, 1996). While an undergraduate at Yale Unviersity, he co-founded Catalyst Recruiting. Although Catalyst folded with the dot-com crash, Ries continued his entrepreneurial career as a Senior Software Engineer at There.com, leading efforts in agile software development and user-generated content. 
I'm one of those people who's been programming since they can remember. I got my start programming on an old IBM XT; it was thanks to MUDs that I first discovered the internet. Those early text-based games were programmed by their own users, and it was by far the best tutorial I could ever have received in the power of software. In a MUD, you could literally conjure new objects that never existed before, just by programming them. I know many people who think that software works like magic, but to me it actually was magic.

Later, I discovered you could get paid to program computers, and really never looked back. While I was still in high school, I became a Java "expert" during a time when there was no such thing. Thanks to Sun's amazing PR blitz, there was tremendous demand for experts on Java, and I did my best to convince people that I was one of that mythical breed. Thanks to the anonymity of the internet, I landed a few jobs, and did quite a bit of writing.

By the time the entrepreneurial bug hit me, the dot-com boom was in its waning days. So much for timing. But I managed a few "good learning experiences" before throwing myself full-bore into IMVU. For almost five years I had the opportunity to build and serve with one of the most talented team I have ever seen. It was by far the most intense and most rewarding experience of my professional life. Because of IMVU's reputation, I've also had the opportunity to serve as an advisor or board member for more than a dozen startups. Rolling up my sleeves and serving with them has enriched my understanding and provided many of the lessons I write about here.

In retrospect, there are some clear themes that stand out from across my career. I have always tried to be a consistent advocate for rapid iterations, fact-based decision making, free software, and values-centric organizations. Every startup has a chance to change the world, by bringing not just a new product, but an entirely new institution into existence. That institution will touch many people in its life: customers, investors, employees, and everyone they touch as well. I believe we have an obligation to ensure the resulting impact is worthy of the energies we invest in bringing it to life.

Eventually, I came to summarize these themes with the phrase "the lean startup." Lean is one of the major trends shaping our world, and its impact goes beyond just optimizing our supply chains. Lean startups can be the most capital efficient companies in the world, because they strive to prevent energy from being expended uselessly. Human talent, passion, and wisdom is too precious a commodity to allow it to be wasted.

So that's me, your author. I hope you take something of value from this blog. If you do, please share your story here in a comment.

Thanks for stopping by.