Embrace technical debt

Financial debt plays an important and positive role in our economy under normal conditions. Yet, especially in times like these, it’s easy to rail against the badness of being in debt; it’s a very human feeling. Remember Hamlet?

LORD POLONIUS:
Neither a borrower nor a lender be;
For loan oft loses both itself and friend,
And borrowing dulls the edge of husbandry.

Technical debt works the same way, and has the same perils. Here’s one of my favorite introductions to the subject, courtesy of Martin Fowler:

In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

The human tendency to moralize about debt affects engineers, too. Many conclude that technical debt is a bad thing, and that teams that incur technical debt are sloppy, irresponsible or stupid.

In this post, I want to challenge that idea, by talking about real-world situations where debt is highly valuable. I hope to show why lean and agile techniques actually reduce the negative impacts of technical debt and increase our ability to take advantage of its positive effects. As usual, this will require a little theory and a willingness to move beyond the false dichotomy of “all or nothing” thinking.

I won’t pretend that there aren’t teams that take on technical debt for bad reasons. Many legacy projects become completely swamped servicing the debt caused by past mistakes. But there is more to technical debt than just the interest payments that come due. Startups especially can benefit by using technical debt to experiment, invest in process, and increase their product development leverage.

In a startup, we should take full advantage of our options, even if they feel dirty or riddled with technical debt. Those moralizing feelings are not always reliable. In particular, try these three things:

Invest in technical debts that may never come due.
The biggest source of waste in new product development is building something that nobody wants. This is a sad outcome which we should work very hard to avoid. Yet there is one silver lining when it does happen: we wind up throwing out working code, debt-riddled and elegantly designed alike. This happened quite often in the early days of IMVU.

For example, I’ve talked often about our belief that an instant messaging add-on product would allow IMVU to take advantage of a network effects strategy. Unfortunately, customers hated that initial product. The thousands of lines of code that made that feature work were a mixed bag – some elegantly designed and under great test coverage, others a series of hacks. The failure of the feature had nothing to do with the quality of the code. As a result, many technical debts were summarily cancelled. Had we taken longer to get that feedback by insisting on writing cleaner code, the debt would have been much deeper.

Accept that good design sometimes leads to technical debt anyway.
Discussions of technical debt are usually framed this way (again from Martin Fowler):

The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline.

This framing takes for granted that the quick and dirty approach will incur significantly more technical debt than the slow and clean approach. Yet other agile principles suggest the opposite, as in YAGNI and DoTheSimplestThingThatCouldPossiblyWork. Reconciling these principles requires a little humility.

Most of us think we know a good design when we see it. Unfortunately, no matter how much up-front analysis we do, until the design is tested by actual practice, we can't really know. Outside the world of hypothetical examples, it's more important to make continual progress than to build the ultimate design.

For example, at a previous virtual world company, we spent years developing an architecture to cope with millions of simultaneous users. Unfortunately, we made two critically flawed assumptions: that customers would primarily consume first-party assets that we shipped to them on CD and that they would tend to congregate in a relatively uniform way. Neither assumption proved remotely accurate. The design failure meant that there was constant thrashing as the servers struggled to provision capacity according to the “elegant” algorithm we’d designed.

As in many scalability decisions, we’d have been much better off investing in agility, so that we could change the architecture in response to actual customer demand, rather than trying to predict the future. That’s what Just-in-time Scalability is all about. Sometimes quick and dirty actually incurs less debt.

Leverage product development with open source and third parties.
Financial leverage refers to investing that is supplemented by borrowed money. Similarly, product development leverage refers to situations in which our own work is fortified by the work of outsiders. For example, early on at IMVU, we incorporated in tons of open source projects. This was a huge win (and we were delighted to give credit where it was due), because it allowed our initial products to get to market much faster. The downside was that we had to combine dozens of projects whose internal architectures, coding styles, and general quality varied widely. It took us a long time to pay off all the debt that incurred – but it was worth it.

In addition, third-party services and API’s enabled us to do more with less, but at a cost: taking on the technical debt of products and teams outside our direct control. We’re not accustomed to accounting for technical debt that occurs in code that we don’t write, but this is short sighted. It’s important to learn to see the whole system that makes our product work: human as well as machine, internal as well as external.

For example, IMVU’s early business model was made possible by Paypal’s easy self-serve and open access payment system. However, we’ve often had to put up with unreliable service, caused by their inflexible internal architecture. We had to live with their technical debts without being able to repay them. It was still a good trade.

Not all debts are created equal.
Interest rates vary, so we should be selective about taking on new debts. Given the choice between incurring technical debt in a particular end-user-visible feature and incurring the same level of debt in a core system, I’d much prefer the former. Here’s why:

• There’s a chance that I’ll never have to pay for that particular debt, because the feature may have no value for customers.
  
  
• It’s possible that the feature, even with debt, might be good enough, and therefore not need revision for a long time. Technical debt manifests as rigidity or inflexibility. When modifying a part of the product afflicted by debt, the work requires a lot of extra – and unpredictable – clean up. But if a given feature is rarely modified, its debt is much less expensive.

The opposite is true with debt in a core system; it’s much more likely that this debt will slow down our ability to make changes later on. For example, an unreliable library deep in the core will manifest as intermittent defects all throughout the product, each of which is hard to localize and debug. Side-effects that reduce agility are the most damaging symptoms of technical debt.

Lean vs. debt
In the world of physical goods, the leaner a supply chain is, the less debt is required to operate it. This makes lean supply chains more robust in the face of the unexpected: if sales suddenly dry up, they are stuck with less unsold inventory and simultaneously have less debt to service. The just-in-time nature of the value chain reduces risk in the face of uncertainty and is also more capital efficient.

A similar relationship applies to technical debt. Teams that practice an agile or lean development process are able to minimize the accumulation of technical debt without sacrificing speed, because they work in smaller batches. They also take better advantage of debt, because they find out sooner if a particular investment has paid off. Traditional development teams, by contrast, often build and deploy large systems before learning if their early choices were sensible, and therefore wind up with a much larger debt to pay. In fact, by the time they become aware of it, they’ve already started to pay significant interest on that debt.

Invest in speed instead of features or debt
This relationship between lean and debt opens up new approaches for dealing with technical debt. The usual debate is phrased as an either-or choice between taking more time to “build it right” or taking a shortcut and incurring more debt. But those are not our only two options. Taking on technical debt does allow investing energy elsewhere, but other new features are not the only option.

We can trade technical debt for process improvement, too. If that improvement pays off (by reducing the batch size of our work, for example), it becomes easier to address all technical debt in the future – including the debt just incurred. And because any particular debt might never come due, this is a better trade. To take one concrete example, it’s often worthwhile to write test coverage for legacy code even without taking the time to refactor.

This reverses the standard intuition about what engineering activities add value, which usually concludes that test coverage is a form of necessary waste but a refactoring is value-added work. However, a refactoring (by itself) might go stale or introduce unintended side-effects. Adding test coverage will make it easier to refactor in the future and also reduce our fear of making changes elsewhere.

Investing in the dynamics of development is more valuable than investing in the static status quo. Startups are always moving, so invest in moving faster and better.

Technical debt in the real world
So far, all of these considerations have been framed in the form of abstract either-or tradeoffs. Real life seldom presents such comparable choices. Instead, we balance lots of unknowns. How much technical debt will a particular approach incur? How likely will customers ultimately use that feature? How painful will it be to refactor later? How much will it slow us down in the meantime? And how much more expensive would it be to do it right? Oh, and how likely is it that the “right” approach actually is?

Luckily, there are better options for these complex decisions than picking an easy extreme, like “never incur technical debt” or “anything goes.” Instead, we can choose a disciplined approach to making proportional investments in prevention and paying down debt, such as Five Whys. They work by focusing our energy on making process and technical changes in precisely those areas that are causing the biggest waste and slowdown.

This is better than making abstract choices about where to invest: better design, paying down old debts, or better process. Instead, techniques like Five Whys teach us to view the entire application and product development team as one integrated system. From this holistic viewpoint, we can optimize accordingly.

Once we can see opportunities for truly global efficiency gains, all that remains is to ensure our team actually makes room for those investments. To do that, we add specific speed regulators, like integrating source control with our continuous integration server or the more elaborate dance required for continuous deployment. This produces a powerful combination: the speed of just-in-time experimentation wedded to a discipline of rigorous waste-reduction.

One last thought. When I talk and write about the advanced product development process at IMVU today, like the cluster immune system or the disciplined approach we take to split-testing and interaction design, it may sound as if we had that capability from the start. Nothing could be further from the truth. The early IMVU was riddled with legacy code and technical debt. We spent endless hours arguing about whether we’d made the right choices in the past. And with the benefit of hindsight, it’s clear that we often made serious mistakes. As one engineer recently told me, “Once we had money in the bank and were near-profitable, I think we would have been well-served by increased up-front product and technology planning. As a culture, we hadn’t yet learned how to make long-term decisions.” He’s right.

In the end, what mattered wasn’t that we did everything right, but that our fundamental approach was flexible and resilient. At no point did we stop everything and do a ground-up rewrite. Instead, we incrementally improved our process, architecture, and infrastructure, always learning and adjusting. The blur you see today is the result of the beneficial compounding interest of that approach applied with discipline over many years. Trust me, it’s a lot of fun.

(This post was tremendously enhanced by a number of early readers from the Twitterverse. You know who you are. Thanks so much.)

The Principles of Product Development Flow

If you've ever wondered why agile or lean development techniques work,The Principles of Product Development Flow: Second Generation Lean Product Development by Donald G. Reinertsen is the book for you. It's quite simply the most advanced product development book you can buy.


For those who hunger for a rigorous approach to managing product development, Donald Reinertsen's book is epic. Myths are busted on practically every page, even myths that are associated with lean/agile. For example, take the lean dictum of working in small batches. I push this technique quite often, because traditional product development tends to work in batches that are much too large. Yet it's not correct to say that batch sizes should be as small as possible. Reinertsen explains how to calculate the optimal batch size from an economic point of view, math and all. It's wonderful to have an author take these sorts of questions seriously, instead of issuing yet another polemic.

The book is structured as a series of principles, logically laid out and briefly discussed - 175 in all. It moves at a rapid clip, each argument backed up with the relevant math and equations: marginal profit, Little's law, Markov processes, probability theory, you name it. This is not for the faint of heart.

The use of economic theory to justify decisions is a recurring theme of the book. Its goal is to help us recognize that every artifact of our product development process is really just a proxy variable. Everything: schedules, efficiency, throughput, even quality. In order to trade them off against each other, we have to convert their impact into economic terms. They are all proxies for our real goal, maximizing an economic variable like profit or revenue.Therefore, in order to maximize the true productivity (aka profitability) of our development efforts, we need to understand the relationships between these proxy variables.

Just for the economic explanations, this book would be worth the price of admission. But it goes beyond that, including techniques for improving the economics of product development. Reinertsen weaves together ideas from lean manufacturing, maneuver warfare, queuing theory, and even the architecture of computer operating systems and the Internet. It's refreshing to see ideas from these different domains brought together in a coherent way:

If we limit ourselves to the relatively simple WIP [work-in-progress] constraints used in lean manufacturing, we will underexploit the power of WIP constraints. Instead, we will explore some of the more advanced ideas used in the world of telecommunications.

If we showed the Toyota Production System to an Internet protocol engineer, it is likely that he would remark, "That is a tail-dropping FIFO queue. That is what we started with on the Internet 30 years ago. We are now four generations beyond that method." I mention this to encourage you to look beyond the manufacturing domain for approaches to control flow.

Reinertsen is keenly aware of what makes product development different from other business functions, like manufacturing, that we sometimes use as a metaphor. Product development deals in designs, which are fundamentally intangible. This is why product development routinely creates disruptive innovation, because our ability to invent new products is limited only (well, primarily) by our capacity for imagination. And yet it is this same ephemeral nature that gives rise to the most difficult problems of product development: how to tell if we're making progress, the high variability of most product development tasks (e.g. will this bug take 5 minutes or 5 weeks to fix?), and the resulting extreme uncertainty that is, incidentally, the environment where startups thrive.

To motivate you to buy this book, I want to walk you through some of Reinertsen's indictment of the status quo in product development, which is based on his extensive interviews, surveys, and consulting work. He starts the book with twelve cardinal sins. See if any of these sound familiar:

1. Failure to correctly quantify economics.
2. Blindness to queues.
3. Worship of efficiency.
4. Hostility to variability.
5. Worship of conformance.
6. Institutionalization of large batch sizes.
7. Underutilization of cadence.
8. Managing timelines instead of queues.
9. Absence of WIP constraints.
10. Inflexibility.
11. Noneconomic flow control.
12. Centralized control.

Reinertsen is not pulling punches. For example, here's him discussing our collective blindness to queues:

To understand the economic cost of queues, product developers must be able to answer two questions. First, how big are our queues? Today, only 2 percent of product developers measure queues. Second, what is the cost of these queues? To answer this second question, we must determine how queue size translates into delay cost, which requires knowing the cost of delay. Today, only 15 percent of product developers know the cost of delay. Since few companies can answer both questions, it should be no surprise that queues are managed poorly today.

Or take this indictment of our worship of efficiency:

But, what do you product developers pay attention to? Today's developers incorrectly try to maximize efficiency ... Any subprocess within product development can be viewed in economic terms. The total cost of the subprocess is composed of its cost of capacity and the delay cost associated with its cycle time. If we are blind to queues, we won't know the delay cost, and we will only be aware of the cost of capacity. Then, if we seek to minimize total cost, we will only focus on the portion we can see, the efficient use of capacity.

This explains why today's product developers assume that efficiency is desirable, and that inefficiency is an undesirable form of waste. This leads them to load their porcesses to dangerously high levels of utilization. How high? Executives coming to my product development classes report operating at 98.5 percent utilization in the precourse surveys. What will this do? [This book] will explain why large queues form when processes with variability are operated at high levels of capacity utilization.

Or consider principle B9: The Batch Size Death Spiral Principle: Large batches lead to even larger batches:

The damage done by large batches can become regenerative when a large batch project starts to acquire a life of its own. It becomes a death march where all participants know they are doomed, but no one has the power to stop. After all, when upper management has been told a project will succeed for 4 years, it is very hard for anyone in middle management to stand up and reverse this forecast...

Our problems grow even bigger when a large project attains the status of the project that cannot afford to fail. Under such conditions, management will almost automatically support anything that appears to help the "golden" project. After all, they want to do everything in their power to eliminate all excuses for failure.

Have you had trouble buying a new piece of test equipment? Just show it will benefit the "golden" project and you will get approval. Have a feature that nobody would let you implement? Find a way to get it into the requirements of the "golden" projec. These large projects act as magnets attracting additional cost, scope, and risk...

At the same time, large batches encourage even larger batches. For example, large test packages bundle many tests together and grow in importance with increasing size. As importance grows, such test packages get even higher priority. If engineers want their personal tests to get high priority, their best strategy is to add them to this large, high-priority test package. Of course, this then makes the package even larger and of higher priority.

This snippet is characteristic of Reinertsen's writing style and reasoning. He shows how the actions of people inside traditional systems are motivated by their rational assessment of their own economics. By setting up the wrong incentives, we are rewarding the very behaviors that we seek to prevent. Reinertsen has a visceral anger about all that waste, and his stories are crackling with disdain for the people who manage such systems - especially when their actions are motivated by intuition, voodoo, or blindness. Startups are frequently guilty as charged - the 4-year death march example above could be written about dozens of venture-backed companies slogging it out in the land of the living dead.

Reinertsen does not speak about startups specifically - his book is meant to speak broadly to product development teams across industries and sectors. Yet his analysis of the sources of waste in development and the remedies that allow us to iterate faster are especially useful for startups. There is an important caveat, however. Product development in established companies and markets has a clear economic rationale to judge effectiveness and productivity. The goal is to increase profitability by making high-ROI investments in new products. To give one example, Reinertsen emphasizes the power of measuring the cost of delay (COD) of a new product. That is, in order to make economically rational decisions about cycle time for a given process, we should understand what it costs the company if the products produced by that process are delayed by, say, one day. Armed with that information, we can make rational trade-offs. Take one of Reinertsen's example:

Unhappy with late deliveries, a project manager decides he can reduce variability by inserting a safety margin or buffer in his schedule. He reduces uncertainty in the schedule by committing to an 80 percent confidence schedule. But, what is the cost of this buffer? The project manager is actually trading cycle time for variability. We can only know if this is a good trade-off if we quantify both the value of the cycle time and the economic benefit of reduced variability.

Does this sound familiar? Many of the startups I talk to - and their boards - seem to equate ability to "hit the schedule" with competence and productivity. Yet timely delivery of new features often comes at the expense of agility, especially if cycle times are long. That is often a bad trade (although, as I'm sure Reinertsen would hasten to add, not always!). For example, many startups would do better by removing buffers from their schedules, embracing the variability of their delivery times, and reducing their cycle times.

Even worse, and unlike their established counterparts, startups often experience a non-quantifiable cost of delay. In a truly new market, we face no meaningful competition, there are no tradeshows to present at, and customers are not clamoring for our product. This means that there are no external factors that argue for shipping product on any given day. A day delay has almost no cost, as far as profitability is concerned. Remember that startups operate by a different unit of progress: what I call validated learning about customers. Any activity that promotes learning is progress, and productivity needs to be measured with respect to that. And that's also where we need to modify some of the specific practices Reinertsen recommends. If the product development team can be engaged in activities that promote business learning at the expense of shipping - or even selling - product, that's a good trade. Hence the need for partitioning our resources into a separate problem team and solution team. As with any methodology, applying the principles faithfully may require modifying the practices to fit a specific context.

Let me close with an excerpt of Reinertsen at his best, using an unexpected example to illustrate the power of fast feedback to make learning more efficient:

It should be obvious that fast feedback improves the speed of learning. What may be less obvious is that fast feedback also increases the efficiency with which we generate information and learn new things. It does this by compressing the time between cause and effect. When this time is short, there are fewer extraneous signals that can introduce noise into our experiment.

Team New Zealand designed the yacht that won the America's Cup. When they tested improvements in keel designs, they used two virtually identical boats. The unimproved boat would sail against the improved boat to determine the effect of a design change. By sailing one boat against the other, they were able to discriminate small changes in performance very quickly.

In contrast, the competing American design team used a single boat supplemented by computer models and NASA wind tunnels. The Americans could never do comparison runs under truly identical conditions because the runs were separated in time ...

Team New Zealand completed many more cycles of learning, and they generated more information in each cycle. This ultimately enabled them to triumph over a much better funded American team. It is worth noting that Team New Zealand explicitly invested in a second boat to create this superior test environment. It is common that we must invest in creating a superior development environment in order to extract the smaller signals that come with fast feedback.

There's far more material in this book that I would love to be able to excerpt. Unfortunately, each principle builds on the previous ones so tightly that it's hard to form coherent excerpts without quoting the whole thing. And that's exactly my point. When we're ready, this book has a tremendous amount to teach all of us. It's not a beginner's guide, and it doesn't hold your hand. Instead, it tackles the hard questions head-on. I've read it and re-read it; for a process junkie like me, I just can't put it down. I hope you'll enjoy it as much as I have.

The Principles of Product Development Flow: Second Generation Lean Product Development

10 years of entrepreneurship

I recently passed my ten-year anniversary of becoming an entrepreneur. I almost didn't realize it, but there's no escaping the math. In the summer of 1999, I started working on a startup from my college dorm. I'd eventually take a leave of absence from school and pursue it full-time, making just enough progress to catch the tail end of the dot-com bubble - and all of the dot-bomb crash. As we say, it was a "good learning experience."

Since then, entrepreneurship has occupied me in one form or another pretty much full-time. Much of what I've learned over the years has been invested in this blog, and that makes creating a sweeping summation challenging. I'm honored to be advancing the work of putting entrepreneurship on a more rigorous footing. But that's still in the future. Looking back, what strikes me the most is not how much I've learned - it's how much time I wasted on stuff that turned out to be utterly unimportant.

I pretty much missed all the trends. I'd been on the internet since I was playing MUDs as a kid, but by 1999 I felt I'd already missed the boat. All the good domain names were taken, all the good ideas were being implemented. If I'd bought just a handful of the "best of the rest" domain names that were available at the time (for a whopping $70 each), I probably could have just retired right then. And the great ideas that became today's successful tech startups dwarf anything that had been done to that point. I just couldn't imagine what the next ten years of innovation would look like. Yet my feeling of having missed out prevented me from experimenting with ideas that might have worked.

Nonetheless, my first startup was a tool for college students from elite colleges to create resumes and help them find jobs. Getting students to create their online profile was easy, but getting employers to pay for access proved difficult. Unfortunately, we were fixated on "building a real business" and never noticed that maybe students would want to share their profiles with each other. Could we have built the first college-based social network five years before Facebook? Maybe, but the thought never even entered our minds. (You can even see the humiliating evidence of my smug incompetence in this absurd article from 1999.) We were focused on revenue, but we didn't understand that revenue is not important for its own sake in an early stage company. Instead, we should have thought of it as an indication of validated learning.

And speaking of Facebook, I definitely didn't think it was a good idea when I first heard about it. Heck, I'd already seen Friendster flame out. What was different this time? And Google? No business model, either. I turned down two opportunities to interview at Google in its pre-AdWords days. It seemed like just a bunch of research-oriented PhD's. Oops. And then, at my first Silicon Valley startup, I watched friends get laid off in successive waves as it started to fail. Almost all of them got scooped up by pre-IPO Google this time. I was "lucky" to not be laid off, or so I thought. Yet, as it turns out, the earlier you got laid off, the earlier you got your Google options. That year, right before the IPO, those months mattered a great deal in terms of financial outcomes.

And while I'm confessing, let me add that I knew Matt Cohler way before he was famous. When he left his consulting job to join LinkedIn (whatever that was), I didn't think twice about it. In fact, I remember sending him and his obscure-to-me co-founder (aka Reid Hoffman) a bug report early-on, instead of taking them up on their offer of an in-person meeting. Doh! And when Matt left LinkedIn to take the reins at Facebook; once again, it didn't register. I was much more focused on other transient success stories of the day. I managed to be envious of dozens of other companies, founders, and colleagues who seemed to be having tremendous success but later turned out to be a mirage. If you'd asked me to rank the top most important people I'd met that year, I doubt it would look very impressive in retrospect.

I'm confident of that last statement, because I made the same mistake again a few years later, when I won an award in 2007. BusinessWeek named me one of the top young entrepreneurs in tech, based on my work at IMVU. Being called a techno-wonderboy in front of everyone I knew was pretty strange, and it felt stranger still to be taking credit for the hard work of the many people who really made IMVU a success. But they were very supportive, and the experience was a good one. Trying to take full advantage of the opportunity, I even reached out and met a few of the other award winners. But, looking back, was it obvious which of the other winners were destined to create world-changing companies? Nope. I completely missed Twitter, for example, which was just one more company to me. Oops.

And yet, missing out on these trends wasn't the end of the world. If I had joined Google early, I'd never have had the opportunity to be part of the founding of IMVU. Then I wouldn't have the chance to work with the incredible employees and mentors from whom I learned so much. And, as I've said many times on this blog, if it weren't for those colossal failures and embarrassing missteps, I'd never have learned anything of significance. So, looking back, I'm grateful for the failures and missed opportunities, embarrassing though they are.

One thing really stands out to me today. I wasted a lot of energy, time, and passion on trend-spotting and trying to compare my success with others. Is it really worthwhile spending time and money trying to impress each other with our supposed successes, especially in a business where real feedback can take five or ten years? We go to mixers, buy fancy offices, focus on PR, and try to one-up each other. I think it's wasteful. Instead, let's focus on building companies that matter, on creating real value for customers, and learning. In time, success will come. And if it doesn't, at least you'll have spent your time doing something intrinsically worthwhile.

In the meantime, don't worry if you can't spot the trends. Neither can the rest of us (well, except for Matt Cohler).

Lean Startup fbFund slides and video

As a follow-up to my previous post on my talk for fbFund at Facebook, there was enough interest in watching video of the talk that I have finally uploaded it using Apple's MobileMe. There are three sizes of video available here; I've embedded the medium quality below. Because the Flip was set at a bad angle, you can't see the slides too well. I've embedded them via slideshare below the video, so you can follow along at home, if you'd like.

Edit: I finally got YouTube upload working, which has better embedding options than MobileMe. If you want to see the original video, use the link above.




Slides are below:

The Lean Startup fbFund Edition

View more documents from Eric Ries.

Before I close, let me quote just one additional bit of twitter feedback from one of the video beta-testers:

bmeschke: @ericries This is such GREAT information. A lot of it i have heard before, but your presentation is so clear. http://bit.ly/IMorB

bmeschke: @ericries @KhuramMalik has been trying to convince me of ur pov and i saw the value, but ur 2 startups example really gives in perspective.

I really value feedback like this; it keeps me going when I have doubts. I worry sometimes about making this blog too much into a travelogue or journal of my public events. My goal is to make everything I post here a substantive contribution to the larger entrepreneurship discussion; I hope to live up to that more often than not. As always, I welcome your feedback. If you have thoughts about the right balance of news, events, and commentary, feel free to leave it as a comment or drop me an email. I truly appreciate your support.

Lean Startup fbFund wrap-up

Last week I had a real blast meeting with the companies at the fbFund incubator at Palo Alto. They were kind enough to open the event up to the public, which meant we were seriously stressing the building's HVAC - and I have pictures to prove it:









Thank you all so much!

As usual, I'd like to post the slides and then offer some additional commentary. Since I tried out some new material in this talk, it was especially educational to see the reaction. (Note: there is video from this event, but it's not online yet. I had a flip cam with me, and the organizers captured their own video with a tripod and everything. If you're interested in seeing it, leave a comment and I'll see if I can get it online.)

Without further ado...

The Lean Startup fbFund Edition

View more documents from Eric Ries.

Now for the feedback. Always have to start with a little praise; thank you all so much!

edzschau: Just returned from great prez by @ericries, don't miss him, great presenter, #leanstartup

mashagenzel: Eric Ries live blows the Eric Ries webcast out of the water, a delightful presence #leanstartup

davemcclure: wow: SRO full house @fbFund REV incubator for @EricRies talk on #LeanStartup (164 Hamilton, Palo Alto) http://yfrog.com/11of5mj

Next, here are the key concepts that people seem to have taken away:

manukumar: A startup is a human institution designed to deliver a new product or service under conditions of extreme uncertainty @ericries #leanstartup

It sometimes feels strange to talk to a group of startup founders and try to tell them what a startup is. Yet, I think this definition is too important to leave out. I think it makes explicable why so-called "best practices" that people bring to startups from other contexts fail so miserably. If your practice is not designed to cope with uncertainty, it has no place in a startup - even if your startup is located in government or enterprise.

alasaarela: Eric Ries: the most profitable business you can build is a Ponzi scheme :) LOL #leanstartup

This was a new meme that I'm trying out. We tend to equate startup success with making money, but that is a poor choice. I think we have to raise our sights a little higher; after all, Ponzi schemes make a lot of money, at least until they blow. They don't really create value. The same has been true of an unfortunate number of startups, they manage to generate a lot of hype, raise a lot of money, and sometimes make some of their investors, employees, or founders rich. But did they leave the world a better place than before they existed? Now, I don't think even most dot-com era founders were bad people, I just think traditional startup methods make it too easy to become confused about whether we're creating real progress or not.

bigs: @ericries says Stealth dev is a (undesirable, failure-presaging) customer-free zone. #LeanStartup

Of course, a big enabler of those kinds of mistakes is stealth-mode. Another recent meme that I hope more and more startups will take to heart: "stealth is a customer-free zone." There are rare times when stealth is a good strategy, but it amplifies risks without necessarily improving rewards. Danger, Will Robinson.

davemcclure: amazing concepts on Continuous Development => "Cluster Immune System" @EricRies #LeanStartup @fbFund http://yfrog.com/bf50cj

dalelarson: Because most features take longer to argue and prioritize than to build. -@ericries #leanstartup

Another new idea in the section on continuous deployment and the cluster immune system. Although there is cost and overhead associated with continuous deployment, the benefits are immense. One such benefit is that, when combined with A/B testing, you can try out small features in less than the amount of time it takes to argue or prioritize them. Nothing is more demoralizing to an engineering team. Prioritizing in a vacuum is a leading source of waste.

jellytheory: Ask WHY? 5X when something unexpected happens: behind every tech problem is a human one. #leanstartup

bmorrow: Not just for engineers: ask why 5x every time something unexpected happens. Find and fix the cause, not just symptoms #leanstartup #fbFund

There seemed to be a lot of resonance about Five Whys this time around, and I'm glad to see it becoming part of the discussion, hence yesterday's "How to conduct a Five Whys root cause analysis."

mashagenzel: Absolutely true, "Startups' competitive advantage is being able to go through customer validation loop faster" via @ericries #leanstartup

dalelarson: Progress == validated customer learning. Iterating at whiteboard != progress. #leanstartup

With so many startups in the room, I could see the stricken faces when I started to talk about the fact that most of us founders are pretty good at keeping people busy - but aren't necessarily sure if we're making progress. Getting clear about what constitutes progress is probably the biggest shift in mindset required to build a lean startup. See Validated learning about customers for more info.

biganderson: Visionary (paying) customers: more visionary than founders, bc they live with the problem that fdrs are trying to solve #leanstartup #fbfund

manukumar: Difference between a vision and a delusion is that a vision is grounded in reality/facts. :@ericries #leanstartup

jordym: "3 types of people have reality distortion fields: good startup founders, bad startup founders, and crazy people." - @ericries #leanstartup

jellytheory: At fbFund REV, listening to @ericries: Is your startup based on a delusion or vision? They often get blurred by entrepreneurs! #leanstartup

OK, let's talk about the vision thing. It's so important, and also so dangerous. Being able to convince other people around you (those within the "reality distortion field") is necessary to sustain the passion and energy that a startup needs. But it can also be used for evil - to convince people to abandon their senses and work on something that nobody will ever want. How can we tell the difference? I saw a lot of people stealing glances at someone else in the room while I was talking about this. I've been there: is it me or my cofounder that's crazy? What if it's both of you? Use some customer development to find out.

daphnecarmeli: "if you don't A/B test, features get PRETTIER over time; if u do A/B test they get more EFFECTIVE over time" @ericries #leanstartup @fbfund

And last but certainly not least, I wanted to try out a line that I haven't used in a long time. The power of A/B testing is so under-exploited in product development, that I'm trying new ways to explain its benefits. Remember that we can use split-testing for both the problem team and solution team, and that causes a lot of confusion. Split-testing is great for linear optimization; making our landing pages, conversion rates, and retention metrics incrementally better day-in day-out. But it's also amazing for testing big hypotheses, like what our customers really want to get out of our product. If you're not doing both, you're missing out.

Last, I was really struck by one section of the talk that seems to have been completely ignored: a new section on the power of Small Batches. Given that this is one of the most powerful concepts for building and iterating faster, I'm surprised that it didn't register. So, here's a request: how can I make this punchier, clearer, or just generally better? Any ideas?

Anyway, thanks so much to everyone who came out. I had an incredible time. See you next time.

How to conduct a Five Whys root cause analysis

In the lean startup workshops, we’ve spent a lot of time discussing the technique of Five Whys. It allows teams to diagnose sources of waste in their development process and continuously improve, reversing the usual trend of teams getting slower over time. With Five Whys, teams can accelerate, even as they scale.

In a previous post, I outlined the benefits of Five Whys: that it allows you to make large investments in infrastructure incrementally, takes advantage of the 80/20 rule to free up resources immediately, and helps organizations become built to learn. Today, I want to talk about the mechanics of Five Whys in greater detail.

First, a caveat. My intention is to describe a full working process, similar to what I’ve seen at IMVU and other lean startups. But as with all process changes, it should not be interpreted as a spec to be implemented right away. In fact, trying too much is just as dangerous at not doing enough. Just as the lean movement has taught us to build incrementally, it has also taught us to attempt process changes incrementally as well. You need to transition to a work flow of small batches – in small batches.

Five Whys involves holding meetings immediately following the resolution of problems the company is facing. These problems can be anything: development mistakes, site outages, marketing program failures, or even internal missed schedules. Any time something unexpected happens, we could do some root cause analysis. Yet it’s helpful to begin by tackling a specific class of problems. For example, a common place to get started is with this rule: any time we have a site outage of any duration, we will hold a post-mortem meeting immediately afterwards.

The first step is to identify three things about the meeting: what problem we are trying to solve, who will run the meeting, and who was affected by the problem. For the problem, it’s essential to hold the meeting immediately following a specific symptom. Five Why’s rarely works for general abstract problems like “our product is buggy” or “our team moves too slow.” Instead, we want to hold it for a specific symptom, like “we missed the Jan 6 deadline by two weeks” or “we had a site outage on Nov 10.” Have faith that if a large general problem is really severe, it will be generating many symptoms that we can use to achieve a general solution.

Always explicitly identify the person running the meeting. Some organizations like to appoint a “Five Whys master” for a specific area of responsibility. For example, at IMVU we had masters appointed for topics like site scalability or unit test failures. The advantage of having an expert run each meeting is that this person can get better and better at helping the team find good solutions. The downside is the extra coordination required to get that person in the room each time. Either way can work. In any event, nobody should hold a master position for too long. Rotation is key to avoid having a situation where one person becomes a bottleneck or single point of failure.

The person running the meeting does not have to be a manager or executive. They do need to have the authority necessary to assign tasks across the organization. That’s because Five Whys will often pierce the illusion of separate departments and discover the human problems that lurk beneath the surface of supposedly technical problems. In order to make Five Whys successful, the person running the meeting has to have the backing of an executive sponsor who has the requisite authority to back them up if they wind up stepping in political landmines. But this executive sponsor doesn’t need to be in the room – what matters is that everyone in the room understands that the person running the meeting has the authority to do so. This means that if you are trying to introduce Five Whys into an organization that is not yet bought-in, you have to start small.

In order to maximize the odds of success, we want to have everyone affected by the problem in the meeting. That means having a representative of every department or function that was affected. When customers are affected, try to have someone who experienced the customer problem first-hand, like the customer service rep who took the calls from angry customers. At a minimum, you have to have the person who discovered the problem. Otherwise, key details are likely to be missed. For example, I have seen many meetings analyzing a problem that took a long time to be diagnosed. In hindsight, the problem was obvious. If the people responsible for diagnosis aren’t in the post-mortem meeting, it’s too easy to conclude, “those people were just too stupid to find the problem” instead of focusing on how our tools could make problems more evident and easier to diagnose.

A root cause analysis meeting has a clear problem, leader, and stakeholders. The most important guideline for the meeting itself is that the purpose of the meeting is to learn and to improve, not to assign blame or to vent. Assume that any problem is preventable and is worth preventing. Problems are caused by insufficiently robust systems rather than individual incompetence. Even in the case of a person making a mistake, we have to ask “why do our tools make that mistake so easy to make?”

The heart of the meeting is the analysis itself. For each problem, we want to ask “why did that happen?” and “why wasn’t it prevented by our process?” We do that iteratively until we have at least five levels of analysis. Of course, the number five is not sacrosanct; it’s just a guideline. What’s critical is that we don’t do too few levels, and we don’t do too many. One hundred whys would be overwhelming. But if we stay stuck at the technical parts of the problem, and never uncover the human problems behind them, we’re not going far enough. So I would keep the meeting going until we’re talking about human problems, and preferably system-level problems. For example, a site outage may seem like it was caused by a bad piece of code, but: why was that code written? Why didn’t the person who wrote it know that it would be harmful? Why didn’t our tests/QA/immune system catch and prevent the problem? Why wasn’t it immediately obvious how to fix the problem?

Pay attention to whether people are comfortable “naming names” in the meeting. If people are afraid of blame, they’ll try to phrase statements in vague, generic terms or use the passive voice, as in “a mistake was made” rather than “So-and-so failed to push the right button.” There’s no easy fix to this problem. Trust takes time to build up, and my experience is that it may take months to establish enough trust that people are confident that there won’t be retribution for speaking up candidly. Stay patient, and be on alert for blame-type talk or for post-meeting revenge. I recommend a zero-tolerance policy for these behaviors – otherwise our Five Whys meetings can descend into Five Blames.

Another common issue is the tendency of root causes to sprout branches. Complex problems rarely have only one cause, and looking for the primary cause is easier in theory than in practice. The branching of causes is also a prime target for so-called “anchor draggers” – people who aren’t really on board with the exercise in the first place. An easy way to derail the meeting is to keep insisting that more and more lateral causes be considered, until the team is running around in circles. Even well intentioned people can wreak the same havoc by simply staying over-focused on technical or ancillary issues. Try to stay focused on just one line of inquiry. Remember, Five Whys is not about making an exhaustive survey of all the problems. It’s about quickly identifying the likely root cause. That’s why it’s more important to do Five Whys frequently than to get it exactly right. It’s a very forgiving practice, because the most wasteful problems will keep clamoring for attention. Have faith that you’ll have many more opportunities to tackle them, and don’t get hung up on any particular solution.

Once you’ve found approximately five levels of the problem, which includes at least one or two human-level issues, it’s time to turn to solutions. The overall plan is to make a proportional investment in each of the five levels. The two major guidelines are: don’t do too much, and don’t do nothing. Almost anything in between will work.

For example, I often cite a real example of a problem that has as its root cause a new employee who was not properly trained. I pick that example on purpose, for two reasons: 1) most of the companies I work with deal with this problem and yet 2) almost none of them have any kind of training program in place for new employees. The reason is simple: setting up a training program is seen as too much work to be justified by the problem. Yet in every situation where I have asked, nobody has been tasked with making a realistic estimate, either of the impact of this lack of training or the real costs of the solution. In fact, even the investigation itself is considered too much work. Five Whys is designed to avoid these nebulous arguments. If new employees are causing problems, that will be a routine topic. If those problems are minor, each time it happens we’ll budget a small amount of time to make progress on the solution.

Let’s imagine the ideal solution would be to spend six weeks setting up a training program for new employees. You can almost hear a manager now: “sure, if you want me to spend the next six weeks setting this up, just let me know. It’s just a matter of priorities. If you think it’s more important than everything else I’m working on, go right ahead and find someone else to take over my other responsibilities…” This logic is airtight, and has the effect of preventing any action. But Five Whys gives us an alternative. If we’ve just analyzed a minor problem that involved a new employee, we should make a minor investment in training. To take an extreme example, let’s say we’ve decided to invest no more than one hour in the solution. Even in that case, we can ask the manager involved to simply spend the first hour of the six-week ideal solution. The next time the problem comes up, we’ll do the next hour, and so on.

In fact, at IMVU, we did exactly that. We started with a simple wiki page with a few bullet points of things that new engineers had tripped over recently. As we kept doing root cause analysis, the list grew. In response to Five Whys that noticed that not all new engineers were reading the list, we expanded it into a new engineer curriculum. Soon, each new engineer was assigned a mentor, and we made it part of the mentor’s job to teach the curriculum. Over time, we also made investments in making it easier to get a new engineer set up with their private sandbox, and even dealt with how to make sure they’d have a machine on their desk when they started. The net effect of all this was to make new engineers incredibly productive right away – in most cases, we’d have them deliver code to production on their very first day. We never set out to build a world-class engineering-training process. Five Whys simply helped us eliminate tons of waste by building one.

Returning to the meeting itself, the person running the meeting should lead the team in brainstorming solutions for each of the problems selected. It’s important that the leader be empowered to pick one and only one solution for each problem, and then assign it to someone to get done. Remember that the cost of the solutions is proportional to the problem caused. This should make it easy to get buy-in from other managers or executives. After all, if it’s a severe problem like a site outage, do they really want to be seen as the person getting in the way of solving it? And if it’s a minor problem, are they really going to object to a few hours of extra work here and there, if it’s towards a good cause? My experience is: usually not.

There are no fixed rules for what constitutes a proportional investment. As teams get experience doing Five Whys, they start to develop rules of thumb for what is reasonable and what isn’t. To restate: the key is that all parties, including the non-technical departments, see the investments as reasonable. As long as we don’t veer to either extreme, the 80/20 rule will make sure that we don’t under-invest over the long term. Remember that if something is a serious problem, it will keep coming up over and over in these meetings. Each time, we’ll get to chip away at it, until it’s no longer a problem.

The last element of a good Five Whys process is to share the results of the analysis widely. I generally recommend sending out the results to the whole company, division, or business unit. This accomplishes two important things: it diffuses knowledge throughout the organization, and it provides evidence that the team in question is taking problems seriously. This latter point can eliminate a lot of waste. I have been amazed how many teams have severe inter-departmental trust issues caused by a lack of communication about problems. For example, engineering feels that they are constantly being pressured to take shortcuts that lower the quality of the product. At the same time, the very marketing people who are applying that pressure think the engineering team doesn’t take quality seriously, and doesn’t respond appropriately when their shoddy work leads to customer problems. Sharing Five Whys can alleviate this problem, by letting everyone know exactly how seriously problems are taken. I say exactly, because it may actually reveal that problems are not taken seriously. In fact, I have seen people in other departments sometimes catch sloppy thinking in a Five Whys report. By sharing the analysis widely, that feedback can flow on a peer-to-peer basis, quickly and easily.

Most organizations are unaware of how much time they spend firefighting, reworking old bugs, and generally expending energy on activities that their customers don’t care about. Yet getting a handle on these sources of waste is hard, especially because they are dynamic. By the time a tops-down company-wide review figured out the main problems, they’d have shifted to another location. Five Whys allows teams to react much faster and then constantly adapt. Without all that waste in their way, they simply go faster.

(If you’re new to Five Whys, I’m eager to hear your feedback. Does this help you get started? What questions or concerns do you have? Leave your thoughts in a comment. If you’ve tried Five Whys, please share your experiences so far. I’ll do my best to help.)

Join the lean startup discussion at Facebook on Thursday

I recently agreed to join Facebook's fbFund incubator as a mentor. Part of that includes giving a presentation for the fbFund companies on the lean startup methodology. I just got word that this is going to be happening this coming Thursday, June 25 in Palo Alto (in the former Facebook HQ). Best of all, I got word they are specially opening this event up to you, dear readers. Priority will be given to employees of companies participating in the fbFund REV program, but I am confident that there will be room for members of the general public who want to attend. If you'd like to reserve a spot, please RSVP.

fbFund REV featuring Eric Ries

The Lean Startup

Host:	fbFund
Type:	Education - Lecture
Network:	Global

Date:	Thursday, June 25, 2009
Time:	1:30pm - 3:30pm
Location:	fbFund REV Garage
Street:	164 Hamilton
City/Town:	Palo Alto, CA
RSVP Here

So, if you are in the Bay Area and want to swing by Palo Alto on Thursday afternoon, come join the conversation. As always, if you're a reader, please do say hello.

Pivot, don't jump to a new vision

In a lean startup, instead of being organized around traditional functional departments, we use a cross-functional problem team and solution team. Each has its own iterative process: customer development and agile development respectively. And the two teams are joined together into a company-wide feedback loop that allows the whole company to be built to learn. This combination allows startups to increase their odds of success by having more major iterations before they run out of resources. It increases the runway without additional cash.

Increasing iterations is a good thing - unless we're going in a circle. The hardest part of entrepreneurship is to develop the judgment to know when it's time to change direction and when it's time to stay the course. That's why so many lean startup practices are focused on learning to tell the difference between progress and wasted effort. One such practice is to pivot from one vision to the next.

Changing the vision is a hard thing to do, and should not be undertaken lightly. Some startups avoid getting customer feedback for precisely this reason: they are afraid that if early reactions are negative, they'll be "forced" to abandon their vision. That's not the goal of a lean startup. We collect feedback for one reason only: to find out whether our vision is compatible with reality or is a delusion. As Steve writes in the Four Steps to the Epiphany, we always seek to find a market for the product as currently specified, not conduct a focus group to tell us what the spec should be. If and only if we can't find any market for our current vision is it appropriate to change it.

So how do you know it's time to change direction? And how do you pick a new direction? These are challenging questions, among the hardest that an early startup team will have to grapple with. Some startups fail because the founders can't have this conversation - they either blow up when they try, or they fail to change because they are afraid of conflict. Both are lethal outcomes.

I want to introduce the concept of the pivot, the idea that successful startups change directions but stay grounded in what they've learned. They keep one foot in the past and place one foot in a new possible future. Over time, this pivoting may lead them far afield from their original vision, but if you look carefully, you'll be able to detect common threads that link each iteration. By contrast, many unsuccessful startups simply jump outright from one vision to something completely different. These jumps are extremely risky, because they don't leverage the validated learning about customers that came before.

I've spoken in some detail about a specific pivot that we went through at IMVU, when we decided to abandon the instant messaging add-on concept, and switch to a standalone instant messaging network. We went through another pivot when we switched again from instant messaging to social networking. Although I wish I could take credit for these pivots, the reality is that they were not caused by my singular insight or that of my other co-founders. Instead, they were made possible by a process-oriented approach that stimulated our thinking and encouraged us to take prudent risks. More than anything, it forced us to take advantage of necessity, the mother of invention. Here's what it looked like.

IMVU had a roughly two-month-long development cycle. Each cycle was punctuated by a meeting of our Business Advisory Board (BAB). At this meeting, we would present our goals for the cycle, all the raw results we'd managed to collect, and our conclusions about what was next. This created a forum for deep thinking and conflict over the direction of the company - a classic problem team activity. It gave the whole company license to go heads-down building product as fast as possible during the development cycle, acting as a solution team should. We knew we'd have the opportunity to think strategically at least once per cycle, so we could stay focused tactically in the meantime.

When it was time to pivot, there were usually certain signs that we'd look to. The most important one actually came from solution team activities. When your fundamental product hypothesis is wrong, the solution team is going to be chronically frustrated. You can try every kind of experiment, add new features, innovate like crazy, optimize the funnel - and get only modest results. One or two cycles of that kind of frustration and you might be able to blame the solution team for insufficient creativity. But eventually, as the company fails to find traction, you start to ask problem team questions: are we really solving an important problem for customers? Are our early adopters really adopting? And does our product really solve the problem we've promised them?

Ironically, although it's the solution team that is the early-warning system ("canary in the coal mine") for pivots, it's actually hard for the solution team to make the decision to pivot. That's why it's so essential to have a co-equal problem team. The more work you've sunk into a product or vision, the harder it is to let it go. As the CTO/VP Engineering, I was the worst offender. It was incredibly hard for me to throw out working code, especially when it was well-factored, unit tested, and generally brilliant (if I did say so myself). I was stuck between a rock and a hard place. Leading up to a pivot, each cycle, despite our best efforts, the metrics weren't good enough. We didn't believe the problem was that we weren't trying hard enough. But we also didn't want to believe that the work we'd expended so far was a waste. It was painful.

The problem team/solution team combined with the concept of the pivot provides a way out. First of all, remember that each team is cross-functional. That means that I (and other engineers) were able to participate in the problem team discussions. Just wearing a different hat made it easier to consider abandoning our work. Such discussions would have been impossible in our execution-oriented engineering team meetings. Context matters. Providing a full view of all the raw data helped, too. It allowed our advisors to help us see patterns we had missed, zooming out the viewpoint from the trees back to the forest. From that angle, it was easier to accept that our micro problems had macro causes.

The pivot helped even more. The hard part about abandoning work is the feeling of wasted effort, that we'd have been just as well-off if we had spent the past few months on vacation instead of working incredibly hard. By pivoting, we honor all the effort by recognizing that learning would have been impossible without the work of the solution team. And rather than just abandoning all that work, we look for ways to take advantage of it in our new direction.

That's the pattern we see in so many successful startups. They did everything they could to take advantage of what they'd built so far. Most engineers naturally think about repurposing the technology platform, and this is a common pattern. But there are a lot of other possibilities. I'd like to call out three in particular: pivot on customer segment, pivot on customer problem, or pivot on a specific feature.

In a segment pivot, we try to take our existing product and use it to solve a similar problem for a different set of customers. This happens commonly when consumer products get unexpectedly adopted in enterprise, as happened to my friends at PBworks. In those cases, the product may stay mostly the same, but the positioning, marketing, and - most importantly - prioritization of features changes dramatically.

In a customer problem pivot, we try to solve a different problem for the same customer segment. This is an exciting kind of change, usually. When doing intense customer development, the problem team can attain a high level of empathy with potential customers. If the results of that exercise is a realization that customers have a problem that our solution doesn't address, and that problem is more promising - it's time to pivot. Starbucks famously did this pivot when they went from selling coffee beans and espresso makers to brewing drinks in-house. They were still serving high-end coffee afficionados, but in a more convenient form. This paved the way for their crossing-the-chasm type breakthrough with mainstream customers.

In a feature pivot, we select out a specific feature from our current product and reorient the whole company around that. A good example is Paypal realizing that their customers were gravitating to the email-payments part of their original solution, and ignoring the complex PDA-based cryptography solution. In order to do this kind of pivot, you need to pay close attention to what customers are really doing, not what you think they should do. It also requires abandoning the extra features that make it hard for new customers to discover what's really valuable about the new, simplified solution.

Without the tools to pivot well, startups get stuck between two extremes: the living dead, still expending energy but not really making progress, always hoping the next new feature will cause traction to magically materialize, and the compulsive jumper, never picking a single direction long enough to find out if there's anything there. Instead of these dead-ends, use the problem and solution team framework and then: pivot, don't jump.

Why Continuous Deployment?

Of all the tactics I have advocated as part of the lean startup, none has provoked as many extreme reactions as continuous deployment, a process that allows companies to release software in minutes instead of days, weeks, or months. My previous startup, IMVU, has used this process to deploy new code as often as an average of fifty times a day. This has stirred up some controversy, with some claiming that this rapid release process contributes to low-quality software or prevents the company from innovating. If we accept the verdict of customers instead of pundits, I think these claims are easy to dismiss. Far more common, and far more difficult, is the range of questions from people who simply wonder if it's possible to apply continuous deployment to their business, industry, or team.

The particulars of IMVU’s history give rise to a lot of these concerns. As a consumer internet company with millions of customers, it may seem to have little relevancy for an enterprise software company with only a handful of potential customers, or a computer security company whose customers demand a rigorous audit before accepting a new release. I think these objections really miss the point of continuous deployment, because they focus on the specific implementations instead of general principles. So, while most of the writing on continuous deployment so far focuses on the how of it, I want to focus today on the why. (If you're looking for resources on getting started, see "Continuous deployment in 5 easy steps")

The goal of continuous deployment is to help development teams drive waste out of their process by simultaneously reducing the batch size and increasing the tempo of their work. This makes it possible for teams to get – and stay – in a condition of flow for sustained periods. This condition makes it much easier for teams to innovate, experiment, and achieve sustained productivity. And it nicely compliments other continuous improvement systems, such as Five Whys.

One large source of waste in development is “double-checking.” For example, imagine a team operating in a traditional waterfall development system, without continuous deployment, test-driven development, or continuous integration. When a developer wants to check-in code, this is a very scary moment. He or she has a choice: check-in now, or double-check to make sure everything still works and looks good. Both options have some attraction. If they check-in now, they can claim the rewards of being done sooner. On the other hand, if they cause a problem, their previous speed will be counted against them. Why didn't they spend just another five minutes making sure they didn't cause that problem? In practice, how developers respond to this dilemma is determined by their incentives, which are driven by the culture of their team. How severely is failure punished? Who will ultimately bear the cost of their mistakes? How important are schedules? Does the team value finishing early?

But the thing to notice in this situation is that there is really no right answer. People who agonize over the choice reap the worst of both worlds. As a result, developers will tend towards two extremes: those who believe in getting things done as fast as possible, and those who believe that work should be carefully checked. Any intermediate position is untenable over the long-term. When things go wrong, any nuanced explanation of the trade-offs involved is going to sound unsatisfying. After all, you could have acted a little sooner or a little more careful – if only you’d known what the problem was going to be in advance. Viewed through the lens of hindsight, most of those judgments look bad. On the other hand, an extreme position is much easier to defend. Both have built-in excuses: “sure there were a few bugs, but I consistently over-deliver on an intense schedule, and it’s well worth it” or “I know you wanted this done sooner, but you know I only ever deliver when it’s absolutely ready, and it’s well worth it.”

These two extreme positions lead to factional strife in development teams, which is extremely unpleasant. Managers start to make a note of who’s on which faction, and then assign projects accordingly. Got a crazy last-minute feature, get the Cowboys to take care of it – and then let the Quality Defenders clean it up in the next release. Both sides start to think of their point of view in moralistic terms: “those guys don’t see the economic value of fast action, they only care about their precious architecture diagrams” or “those guys are sloppy and have no professional pride.” Having been called upon to mediate these disagreements many times in my career, I can attest to just how wasteful they are.

However, they are completely logical outgrowths of a large-batch-size development process that forces developers to make trade-offs between time and quality, using the old “time-quality-money, pick two fallacy.” Because feedback is slow in coming, the damage caused by a mistake is felt long after the decisions that caused the mistake were made, making learning difficult. Because everyone gets ready to integrate with the release batch around the same time (there being no incentive to integrate early), conflicts are resolved under extreme time pressure. Features are chronically on the bubble, about to get deferred to the next release. But when they do get deferred, they tend to have their scope increased (“after all, we have a whole release cycle, and it’s almost done…”), which leads to yet another time crunch, and so on. And, of course, the code rarely performs in production the way it does in the testing or staging environment, which leads to a series of hot-fixes immediately following each release. These come at the expense of the next release batch, meaning that each release cycle starts off behind.

Many times when I interview a development team caught in the pincers of this situation, they want my help "fixing people." Thanks to a phenomenon called the Fundamental Attribution Error in psychology, humans tend to become convinced that other people’s behavior is due to their fundamental attributes, like their character, ethics, or morality – even while we excuse our own actions as being influenced by circumstances. So developers stuck in this world tend to think the other developers on their team are either, deep in their souls, plodding pedants or sloppy coders. Neither is true – they just have their incentives all messed up.

You can’t change the underlying incentives of this situation by getting better at any one activity. Better release planning, estimating, architecting, or integrating will only mitigate the symptoms. The only traditional technique for solving this problem is to add in massive queues in the forms of schedule padding, extra time for integration, code freezes and the like. In fact, most organizations don’t realize just how much of this padding is already going on in the estimates that individual developers learn to generate. But padding doesn’t help, because it serves to slow down the whole process. And as all development teams will tell you – time is always short. In fact, excess time pressure is exactly why they think they have these problems in the first place.

So we need to find solutions that operate at the systems level to break teams out of this pincer action. The agile software movement has made numerous contributions: continuous integration, which helps accelerate feedback about defects; story cards and kanban that reduce batch size; a daily stand-up that increases tempo. Continuous deployment is another such technique, one with a unique power to change development team dynamics for the better.

Why does it work?

First, continuous deployment separates out two different definitions of the terms “release.” One is used by engineers to refer to the process of getting code fully integrated into production. Another is used by marketing to refer to what customers see. In traditional batch-and-queue development, these two concepts are linked. All customers will see the new software as soon as it’s deployed. This requires that all of the testing of the release happen before it is deployed to production, in special staging or testing environments. And this leaves the release vulnerable to unanticipated problems during this window of time: after the code is written but before it's running in production. On top of that overhead, by conflating the marketing release with the technical release, the amount of coordination overhead required to ship something is also dramatically increased.

Under continuous deployment, as soon as code is written, it’s on its way to production. That means we are often deploying just 1% of a feature – long before customers would want to see it. In fact, most of the work involved with a new feature is not the user-visible parts of the feature itself. Instead, it’s the millions of tiny touch points that integrate the feature with all the other features that were built before. Think of the dozens of little API changes that are required when we want to pass new values through the system. These changes are generally supposed to be “side effect free” meaning they don’t affect the behavior of the system at the point of insertion – emphasis on supposed. In fact, many bugs are caused by unusual or unnoticed side effects of these deep changes. The same is true of small changes that only conflict with configuration parameters in the production environment. It’s much better to get this feedback as soon as possible, which continuous deployment offers.

Continuous deployment also acts as a speed regulator. Every time the deployment process encounters a problem, a human being needs to get involved to diagnose it. During this time, it’s intentionally impossible for anyone else to deploy. When teams are ready to deploy, but the process is locked, they become immediately available to help diagnose and fix the deployment problem (the alternative, that they continue to generate, but not deploy, new code just serves to increase batch sizes to everyone’s detriment). This speed regulation is a tricky adjustment for teams that are accustomed to measuring their progress via individual efficiency. In such a system, the primary goal of each engineer is to stay busy, using as close to 100% of his or her time for coding as possible. Unfortunately, this view ignores the overall throughput of the team. Even if you don’t adopt a radical definition of progress, like the “validated learning about customers” that I advocate, it’s still sub-optimal to keep everyone busy. When you’re in the midst of integration problems, any code that someone is writing is likely to have to be revised as a result of conflicts. Same with configuration mismatches or multiple teams stepping on each others’ toes. In such circumstances, it’s much better for overall productivity for people to stop coding and start talking. Once they figure out how to coordinate their actions so that the work they are doing doesn’t have to be reworked, it’s productive to start coding again.

Returning to our development team divided into Cowboy and Quality factions, let’s take a look at how continuous deployment can change the calculus of their situation. For one, continuous deployment fosters learning and professional development – on both sides of the divide. Instead of having to argue with each other about the right way to code, each individual has an opportunity to learn directly from the production environment. This is the meaning of the axiom to “let your defects be your teacher.”

If an engineer has a tendency to ship too soon, they will tend to find themselves grappling with the cluster immune system, continuous integration server, and five whys master more often. These encounters, far from being the high-stakes arguments inherent in traditional teams are actually low-risk, mostly private or small-group affairs. Because the feedback is rapid, Cowboys will start to learn what kinds of testing, preparation and checking really do let them work faster. They’ll be learning the key truth that there is such a thing as “too fast” – many quality problems actually slow you down.

But for engineers that have the tendency to wait too long before shipping, they too have lessons to learn. For one, the larger the batch size of their work, the harder it will be to get it integrated. At IMVU, we would occasionally hire someone from a more traditional organization who had a hard time letting go of their “best practices” and habits. Sometimes they’d advocate for doing their work on a separate branch, and only integrating at the end. Although I’d always do my best to convince them otherwise, if they were insistent I would encourage them to give it a try. Inevitably, a week or two later, I’d enjoy the spectacle of watching them engage in something I called “code bouncing.” It's like throwing a rubber ball against the wall. In a code bounce, someone tries to check in a huge batch. First they have integration conflicts, which require talking to various people on the team to know how to resolve them properly. Of course, while they are resolving, new changes are being checked in. So new conflicts appear. This cycle repeats for a while, until the team either catches up to all the conflicts or just asks the rest of the team for a general check-in freeze. Then the fun part begins. Getting a large batch through the continuous integration server, incremental deploy system, and real-time monitoring system almost never works on the first try. Thus the large batch gets reverted. While the problems are being fixed, more changes are being checked in. Unless we freeze the work of the whole team, this can go on for days. But if we do engage in a general check-in freeze, then we’re driving up the batch size of everyone else – which will lead to future episodes of code bouncing. In my experience, just one or two episodes are enough to cure anyone of their desire to work in large batches.

Because continuous deployment encourages learning, teams that practice it are able to get faster over time. That’s because each individual’s incentives are aligned with the goals of the whole team. Each person works to drive down waste in their own work, and this true efficiency gain more than offsets the incremental overhead of having to build and maintain the infrastructure required to do continuous deployment. In fact, if you practice Five Whys too, you can build all of this infrastructure in a completely incremental fashion. It’s really a lot of fun.

One last benefit: morale. At a recent talk, an audience member asked me about the impact of continuous deployment on morale. This manager was worried that moving their engineers to a more-rapid release cycle would stress them out, making them feel like they were always fire fighting and releasing, and never had time for “real work.” As luck would have it, one of IMVU’s engineers happened to be in the audience at the time. They provided a better answer than I ever could. They explained that by reducing the overhead of doing a release, each engineer gets to work to their own release schedule. That means, as soon as they are ready to deploy, they can. So even if it’s midnight, if your feature is ready to go, you can check-in, deploy, and start talking to customers about it right away. No extra approvals, meetings, or coordination required. Just you, your code, and your customers. It’s pretty satisfying.