Friday, July 31, 2009
So I used to wind up spending a lot of fruitless hours trying to play matchmaker, and generally having a hard time. Part of the problem is that I couldn't post a public link. Most of the jobs I come across are not listed in any directory, and most of the candidates are not officially looking - many don't even have a resume. Great startups are always hiring, if the fit is right. They may not have an open req for a specific job, but if the right amazing person walks in the door, they will always find something productive for them to do.
To solve this problem, I asked a friend to collaborate with me in building a new tool. It's incredibly simple, and it's called joblink.tw. Think of it as a bit.ly for jobs. Using it to recruit takes less than a minute.
joblink.tw lets you post a description of a job without having to reveal the name of the company who is offering it, or the contact information of the hiring manager. Who wants that stuff posted on the internet? Instead, each joblink has a form that allows anyone who's interested to contact the poster. The actual exchange of information occurs in private, in email.
The same process works for job candidates. Know someone who'd make a great engineer at a startup? Just write a few words about them, set their email address as the private recipient, and tweet about it. They can turn off these contacts at any time.
To see a list of the jobs that I've posted on joblink.tw so far, take a look at my joblink profile. joblink.tw also supports oauth, so you can log in with your twitter credentials. If you do, you can have your joblinks broadcast on the joblinktw twitter account. For updates about joblink.tw or to see joblinks that others have posted, follow on twitter.
So far, this tool has saved me countless hours, and made quite a few interesting connections. That means it's accomplished all of my goals for it. If anyone else finds it useful, too, that will prove a huge plus. And if anyone has feedback, please feel free to use our integrated uservoice forum or just post a comment on this post. I'd be happy to make it better (regular readers will recognize joblink as a minimum viable product).
Thursday, July 30, 2009
On August 19th, Eric will be speaking at a dinner in Boulder. The event will include a talk from Eric on The Lean Startup over dinner, followed by moderated table discussion and then final Q&A with Eric. Tickets are available now and include dinner. A discounted price is available for early stage entrepreneurs and students.Traveling to new startup hubs is one of my favorite things about being able to do this full-time. I want to thank Techstars for putting this event together and giving me a chance to experience the scene, even if it lacks a name. For what it's worth, I like Silicon Mountain.
On August 20th, Eric is leading a half day in-depth workshop on the Lean Startup. This is a great chance to really go deep on some of the concepts behind building Lean Startups. A very limited number of tickets are also available for this workshop. Early bird pricing expires on August 6th, so register early.
Brad Feld also had some thoughts on this event on his blog. I thought I'd share a little bit of that, too:
And for those of you who are in Colorado but don't know what my speaking events are like, please take a look at some previous posts. We've got slides, video, audio and twitter commentary. That way, you can know what you're in for in advance.
I’ve been interested in different approaches to software development going back to 1987 when – in my first company Feld Technologies – my partner Dave Jilk and I started talking about “semi-custom software development” (way ahead of its time). During the same period (1987 – 1990) and I did some work at MIT under Eric von Hippel on “user driven innovation with regard to software development” which today would probably fall under the heading of “open source software development approaches.”
In 2002 I became exposed to the idea of “agile software development” and subsequently was a first round investor in Rally Software which is now the market leader in Agile application lifecycle management software. Building on this, I’ve recently become fascinated with the notion of continuous deployment, a concept that has been popularized by Eric Ries and others.
As usual, if you're a reader and can attend, please come say hello. Thanks!
(Speaking of the fall tour, if you're interested in hosting or organizing a lean startup event near you, feel free to drop me a line. So far, I'm going to be on the east coast at least three times, and in Europe twice. Stay tuned for more details.)
Wednesday, July 29, 2009
Neither a borrower nor a lender be;
For loan oft loses both itself and friend,
And borrowing dulls the edge of husbandry.
Technical debt works the same way, and has the same perils. Here’s one of my favorite introductions to the subject, courtesy of Martin Fowler:
In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.
The human tendency to moralize about debt affects engineers, too. Many conclude that technical debt is a bad thing, and that teams that incur technical debt are sloppy, irresponsible or stupid.
In this post, I want to challenge that idea, by talking about real-world situations where debt is highly valuable. I hope to show why lean and agile techniques actually reduce the negative impacts of technical debt and increase our ability to take advantage of its positive effects. As usual, this will require a little theory and a willingness to move beyond the false dichotomy of “all or nothing” thinking.
I won’t pretend that there aren’t teams that take on technical debt for bad reasons. Many legacy projects become completely swamped servicing the debt caused by past mistakes. But there is more to technical debt than just the interest payments that come due. Startups especially can benefit by using technical debt to experiment, invest in process, and increase their product development leverage.
In a startup, we should take full advantage of our options, even if they feel dirty or riddled with technical debt. Those moralizing feelings are not always reliable. In particular, try these three things:
Invest in technical debts that may never come due.
The biggest source of waste in new product development is building something that nobody wants. This is a sad outcome which we should work very hard to avoid. Yet there is one silver lining when it does happen: we wind up throwing out working code, debt-riddled and elegantly designed alike. This happened quite often in the early days of IMVU.
For example, I’ve talked often about our belief that an instant messaging add-on product would allow IMVU to take advantage of a network effects strategy. Unfortunately, customers hated that initial product. The thousands of lines of code that made that feature work were a mixed bag – some elegantly designed and under great test coverage, others a series of hacks. The failure of the feature had nothing to do with the quality of the code. As a result, many technical debts were summarily cancelled. Had we taken longer to get that feedback by insisting on writing cleaner code, the debt would have been much deeper.
Accept that good design sometimes leads to technical debt anyway.
Discussions of technical debt are usually framed this way (again from Martin Fowler):
The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline.
This framing takes for granted that the quick and dirty approach will incur significantly more technical debt than the slow and clean approach. Yet other agile principles suggest the opposite, as in YAGNI and DoTheSimplestThingThatCouldPossiblyWork. Reconciling these principles requires a little humility.
Most of us think we know a good design when we see it. Unfortunately, no matter how much up-front analysis we do, until the design is tested by actual practice, we can't really know. Outside the world of hypothetical examples, it's more important to make continual progress than to build the ultimate design.
For example, at a previous virtual world company, we spent years developing an architecture to cope with millions of simultaneous users. Unfortunately, we made two critically flawed assumptions: that customers would primarily consume first-party assets that we shipped to them on CD and that they would tend to congregate in a relatively uniform way. Neither assumption proved remotely accurate. The design failure meant that there was constant thrashing as the servers struggled to provision capacity according to the “elegant” algorithm we’d designed.
As in many scalability decisions, we’d have been much better off investing in agility, so that we could change the architecture in response to actual customer demand, rather than trying to predict the future. That’s what Just-in-time Scalability is all about. Sometimes quick and dirty actually incurs less debt.
Leverage product development with open source and third parties.
Financial leverage refers to investing that is supplemented by borrowed money. Similarly, product development leverage refers to situations in which our own work is fortified by the work of outsiders. For example, early on at IMVU, we incorporated in tons of open source projects. This was a huge win (and we were delighted to give credit where it was due), because it allowed our initial products to get to market much faster. The downside was that we had to combine dozens of projects whose internal architectures, coding styles, and general quality varied widely. It took us a long time to pay off all the debt that incurred – but it was worth it.
In addition, third-party services and API’s enabled us to do more with less, but at a cost: taking on the technical debt of products and teams outside our direct control. We’re not accustomed to accounting for technical debt that occurs in code that we don’t write, but this is short sighted. It’s important to learn to see the whole system that makes our product work: human as well as machine, internal as well as external.
For example, IMVU’s early business model was made possible by Paypal’s easy self-serve and open access payment system. However, we’ve often had to put up with unreliable service, caused by their inflexible internal architecture. We had to live with their technical debts without being able to repay them. It was still a good trade.
Not all debts are created equal.
Interest rates vary, so we should be selective about taking on new debts. Given the choice between incurring technical debt in a particular end-user-visible feature and incurring the same level of debt in a core system, I’d much prefer the former. Here’s why:
- There’s a chance that I’ll never have to pay for that particular debt, because the feature may have no value for customers.
- It’s possible that the feature, even with debt, might be good enough, and therefore not need revision for a long time. Technical debt manifests as rigidity or inflexibility. When modifying a part of the product afflicted by debt, the work requires a lot of extra – and unpredictable – clean up. But if a given feature is rarely modified, its debt is much less expensive.
The opposite is true with debt in a core system; it’s much more likely that this debt will slow down our ability to make changes later on. For example, an unreliable library deep in the core will manifest as intermittent defects all throughout the product, each of which is hard to localize and debug. Side-effects that reduce agility are the most damaging symptoms of technical debt.
Lean vs. debt
In the world of physical goods, the leaner a supply chain is, the less debt is required to operate it. This makes lean supply chains more robust in the face of the unexpected: if sales suddenly dry up, they are stuck with less unsold inventory and simultaneously have less debt to service. The just-in-time nature of the value chain reduces risk in the face of uncertainty and is also more capital efficient.
A similar relationship applies to technical debt. Teams that practice an agile or lean development process are able to minimize the accumulation of technical debt without sacrificing speed, because they work in smaller batches. They also take better advantage of debt, because they find out sooner if a particular investment has paid off. Traditional development teams, by contrast, often build and deploy large systems before learning if their early choices were sensible, and therefore wind up with a much larger debt to pay. In fact, by the time they become aware of it, they’ve already started to pay significant interest on that debt.
Invest in speed instead of features or debt
This relationship between lean and debt opens up new approaches for dealing with technical debt. The usual debate is phrased as an either-or choice between taking more time to “build it right” or taking a shortcut and incurring more debt. But those are not our only two options. Taking on technical debt does allow investing energy elsewhere, but other new features are not the only option.
We can trade technical debt for process improvement, too. If that improvement pays off (by reducing the batch size of our work, for example), it becomes easier to address all technical debt in the future – including the debt just incurred. And because any particular debt might never come due, this is a better trade. To take one concrete example, it’s often worthwhile to write test coverage for legacy code even without taking the time to refactor.
This reverses the standard intuition about what engineering activities add value, which usually concludes that test coverage is a form of necessary waste but a refactoring is value-added work. However, a refactoring (by itself) might go stale or introduce unintended side-effects. Adding test coverage will make it easier to refactor in the future and also reduce our fear of making changes elsewhere.
Investing in the dynamics of development is more valuable than investing in the static status quo. Startups are always moving, so invest in moving faster and better.
Technical debt in the real world
So far, all of these considerations have been framed in the form of abstract either-or tradeoffs. Real life seldom presents such comparable choices. Instead, we balance lots of unknowns. How much technical debt will a particular approach incur? How likely will customers ultimately use that feature? How painful will it be to refactor later? How much will it slow us down in the meantime? And how much more expensive would it be to do it right? Oh, and how likely is it that the “right” approach actually is?
Luckily, there are better options for these complex decisions than picking an easy extreme, like “never incur technical debt” or “anything goes.” Instead, we can choose a disciplined approach to making proportional investments in prevention and paying down debt, such as Five Whys. They work by focusing our energy on making process and technical changes in precisely those areas that are causing the biggest waste and slowdown.
This is better than making abstract choices about where to invest: better design, paying down old debts, or better process. Instead, techniques like Five Whys teach us to view the entire application and product development team as one integrated system. From this holistic viewpoint, we can optimize accordingly.
Once we can see opportunities for truly global efficiency gains, all that remains is to ensure our team actually makes room for those investments. To do that, we add specific speed regulators, like integrating source control with our continuous integration server or the more elaborate dance required for continuous deployment. This produces a powerful combination: the speed of just-in-time experimentation wedded to a discipline of rigorous waste-reduction.
One last thought. When I talk and write about the advanced product development process at IMVU today, like the cluster immune system or the disciplined approach we take to split-testing and interaction design, it may sound as if we had that capability from the start. Nothing could be further from the truth. The early IMVU was riddled with legacy code and technical debt. We spent endless hours arguing about whether we’d made the right choices in the past. And with the benefit of hindsight, it’s clear that we often made serious mistakes. As one engineer recently told me, “Once we had money in the bank and were near-profitable, I think we would have been well-served by increased up-front product and technology planning. As a culture, we hadn’t yet learned how to make long-term decisions.” He’s right.
In the end, what mattered wasn’t that we did everything right, but that our fundamental approach was flexible and resilient. At no point did we stop everything and do a ground-up rewrite. Instead, we incrementally improved our process, architecture, and infrastructure, always learning and adjusting. The blur you see today is the result of the beneficial compounding interest of that approach applied with discipline over many years. Trust me, it’s a lot of fun.
(This post was tremendously enhanced by a number of early readers from the Twitterverse. You know who you are. Thanks so much.)
Monday, July 13, 2009
If you've ever wondered why agile or lean development techniques work,The Principles of Product Development Flow: Second Generation Lean Product Development by Donald G. Reinertsen is the book for you. It's quite simply the most advanced product development book you can buy.
For those who hunger for a rigorous approach to managing product development, Donald Reinertsen's book is epic. Myths are busted on practically every page, even myths that are associated with lean/agile. For example, take the lean dictum of working in small batches. I push this technique quite often, because traditional product development tends to work in batches that are much too large. Yet it's not correct to say that batch sizes should be as small as possible. Reinertsen explains how to calculate the optimal batch size from an economic point of view, math and all. It's wonderful to have an author take these sorts of questions seriously, instead of issuing yet another polemic.
The book is structured as a series of principles, logically laid out and briefly discussed - 175 in all. It moves at a rapid clip, each argument backed up with the relevant math and equations: marginal profit, Little's law, Markov processes, probability theory, you name it. This is not for the faint of heart.
The use of economic theory to justify decisions is a recurring theme of the book. Its goal is to help us recognize that every artifact of our product development process is really just a proxy variable. Everything: schedules, efficiency, throughput, even quality. In order to trade them off against each other, we have to convert their impact into economic terms. They are all proxies for our real goal, maximizing an economic variable like profit or revenue.Therefore, in order to maximize the true productivity (aka profitability) of our development efforts, we need to understand the relationships between these proxy variables.
Just for the economic explanations, this book would be worth the price of admission. But it goes beyond that, including techniques for improving the economics of product development. Reinertsen weaves together ideas from lean manufacturing, maneuver warfare, queuing theory, and even the architecture of computer operating systems and the Internet. It's refreshing to see ideas from these different domains brought together in a coherent way:
If we limit ourselves to the relatively simple WIP [work-in-progress] constraints used in lean manufacturing, we will underexploit the power of WIP constraints. Instead, we will explore some of the more advanced ideas used in the world of telecommunications.Reinertsen is keenly aware of what makes product development different from other business functions, like manufacturing, that we sometimes use as a metaphor. Product development deals in designs, which are fundamentally intangible. This is why product development routinely creates disruptive innovation, because our ability to invent new products is limited only (well, primarily) by our capacity for imagination. And yet it is this same ephemeral nature that gives rise to the most difficult problems of product development: how to tell if we're making progress, the high variability of most product development tasks (e.g. will this bug take 5 minutes or 5 weeks to fix?), and the resulting extreme uncertainty that is, incidentally, the environment where startups thrive.
If we showed the Toyota Production System to an Internet protocol engineer, it is likely that he would remark, "That is a tail-dropping FIFO queue. That is what we started with on the Internet 30 years ago. We are now four generations beyond that method." I mention this to encourage you to look beyond the manufacturing domain for approaches to control flow.
To motivate you to buy this book, I want to walk you through some of Reinertsen's indictment of the status quo in product development, which is based on his extensive interviews, surveys, and consulting work. He starts the book with twelve cardinal sins. See if any of these sound familiar:
- Failure to correctly quantify economics.
- Blindness to queues.
- Worship of efficiency.
- Hostility to variability.
- Worship of conformance.
- Institutionalization of large batch sizes.
- Underutilization of cadence.
- Managing timelines instead of queues.
- Absence of WIP constraints.
- Noneconomic flow control.
- Centralized control.
To understand the economic cost of queues, product developers must be able to answer two questions. First, how big are our queues? Today, only 2 percent of product developers measure queues. Second, what is the cost of these queues? To answer this second question, we must determine how queue size translates into delay cost, which requires knowing the cost of delay. Today, only 15 percent of product developers know the cost of delay. Since few companies can answer both questions, it should be no surprise that queues are managed poorly today.Or take this indictment of our worship of efficiency:
But, what do you product developers pay attention to? Today's developers incorrectly try to maximize efficiency ... Any subprocess within product development can be viewed in economic terms. The total cost of the subprocess is composed of its cost of capacity and the delay cost associated with its cycle time. If we are blind to queues, we won't know the delay cost, and we will only be aware of the cost of capacity. Then, if we seek to minimize total cost, we will only focus on the portion we can see, the efficient use of capacity.Or consider principle B9: The Batch Size Death Spiral Principle: Large batches lead to even larger batches:
This explains why today's product developers assume that efficiency is desirable, and that inefficiency is an undesirable form of waste. This leads them to load their porcesses to dangerously high levels of utilization. How high? Executives coming to my product development classes report operating at 98.5 percent utilization in the precourse surveys. What will this do? [This book] will explain why large queues form when processes with variability are operated at high levels of capacity utilization.
The damage done by large batches can become regenerative when a large batch project starts to acquire a life of its own. It becomes a death march where all participants know they are doomed, but no one has the power to stop. After all, when upper management has been told a project will succeed for 4 years, it is very hard for anyone in middle management to stand up and reverse this forecast...This snippet is characteristic of Reinertsen's writing style and reasoning. He shows how the actions of people inside traditional systems are motivated by their rational assessment of their own economics. By setting up the wrong incentives, we are rewarding the very behaviors that we seek to prevent. Reinertsen has a visceral anger about all that waste, and his stories are crackling with disdain for the people who manage such systems - especially when their actions are motivated by intuition, voodoo, or blindness. Startups are frequently guilty as charged - the 4-year death march example above could be written about dozens of venture-backed companies slogging it out in the land of the living dead.
Our problems grow even bigger when a large project attains the status of the project that cannot afford to fail. Under such conditions, management will almost automatically support anything that appears to help the "golden" project. After all, they want to do everything in their power to eliminate all excuses for failure.
Have you had trouble buying a new piece of test equipment? Just show it will benefit the "golden" project and you will get approval. Have a feature that nobody would let you implement? Find a way to get it into the requirements of the "golden" projec. These large projects act as magnets attracting additional cost, scope, and risk...
At the same time, large batches encourage even larger batches. For example, large test packages bundle many tests together and grow in importance with increasing size. As importance grows, such test packages get even higher priority. If engineers want their personal tests to get high priority, their best strategy is to add them to this large, high-priority test package. Of course, this then makes the package even larger and of higher priority.
Reinertsen does not speak about startups specifically - his book is meant to speak broadly to product development teams across industries and sectors. Yet his analysis of the sources of waste in development and the remedies that allow us to iterate faster are especially useful for startups. There is an important caveat, however. Product development in established companies and markets has a clear economic rationale to judge effectiveness and productivity. The goal is to increase profitability by making high-ROI investments in new products. To give one example, Reinertsen emphasizes the power of measuring the cost of delay (COD) of a new product. That is, in order to make economically rational decisions about cycle time for a given process, we should understand what it costs the company if the products produced by that process are delayed by, say, one day. Armed with that information, we can make rational trade-offs. Take one of Reinertsen's example:
Unhappy with late deliveries, a project manager decides he can reduce variability by inserting a safety margin or buffer in his schedule. He reduces uncertainty in the schedule by committing to an 80 percent confidence schedule. But, what is the cost of this buffer? The project manager is actually trading cycle time for variability. We can only know if this is a good trade-off if we quantify both the value of the cycle time and the economic benefit of reduced variability.Does this sound familiar? Many of the startups I talk to - and their boards - seem to equate ability to "hit the schedule" with competence and productivity. Yet timely delivery of new features often comes at the expense of agility, especially if cycle times are long. That is often a bad trade (although, as I'm sure Reinertsen would hasten to add, not always!). For example, many startups would do better by removing buffers from their schedules, embracing the variability of their delivery times, and reducing their cycle times.
Even worse, and unlike their established counterparts, startups often experience a non-quantifiable cost of delay. In a truly new market, we face no meaningful competition, there are no tradeshows to present at, and customers are not clamoring for our product. This means that there are no external factors that argue for shipping product on any given day. A day delay has almost no cost, as far as profitability is concerned. Remember that startups operate by a different unit of progress: what I call validated learning about customers. Any activity that promotes learning is progress, and productivity needs to be measured with respect to that. And that's also where we need to modify some of the specific practices Reinertsen recommends. If the product development team can be engaged in activities that promote business learning at the expense of shipping - or even selling - product, that's a good trade. Hence the need for partitioning our resources into a separate problem team and solution team. As with any methodology, applying the principles faithfully may require modifying the practices to fit a specific context.
Let me close with an excerpt of Reinertsen at his best, using an unexpected example to illustrate the power of fast feedback to make learning more efficient:
It should be obvious that fast feedback improves the speed of learning. What may be less obvious is that fast feedback also increases the efficiency with which we generate information and learn new things. It does this by compressing the time between cause and effect. When this time is short, there are fewer extraneous signals that can introduce noise into our experiment.There's far more material in this book that I would love to be able to excerpt. Unfortunately, each principle builds on the previous ones so tightly that it's hard to form coherent excerpts without quoting the whole thing. And that's exactly my point. When we're ready, this book has a tremendous amount to teach all of us. It's not a beginner's guide, and it doesn't hold your hand. Instead, it tackles the hard questions head-on. I've read it and re-read it; for a process junkie like me, I just can't put it down. I hope you'll enjoy it as much as I have.
Team New Zealand designed the yacht that won the America's Cup. When they tested improvements in keel designs, they used two virtually identical boats. The unimproved boat would sail against the improved boat to determine the effect of a design change. By sailing one boat against the other, they were able to discriminate small changes in performance very quickly.
In contrast, the competing American design team used a single boat supplemented by computer models and NASA wind tunnels. The Americans could never do comparison runs under truly identical conditions because the runs were separated in time ...
Team New Zealand completed many more cycles of learning, and they generated more information in each cycle. This ultimately enabled them to triumph over a much better funded American team. It is worth noting that Team New Zealand explicitly invested in a second boat to create this superior test environment. It is common that we must invest in creating a superior development environment in order to extract the smaller signals that come with fast feedback.
The Principles of Product Development Flow: Second Generation Lean Product Development
Thursday, July 9, 2009
Since then, entrepreneurship has occupied me in one form or another pretty much full-time. Much of what I've learned over the years has been invested in this blog, and that makes creating a sweeping summation challenging. I'm honored to be advancing the work of putting entrepreneurship on a more rigorous footing. But that's still in the future. Looking back, what strikes me the most is not how much I've learned - it's how much time I wasted on stuff that turned out to be utterly unimportant.
I pretty much missed all the trends. I'd been on the internet since I was playing MUDs as a kid, but by 1999 I felt I'd already missed the boat. All the good domain names were taken, all the good ideas were being implemented. If I'd bought just a handful of the "best of the rest" domain names that were available at the time (for a whopping $70 each), I probably could have just retired right then. And the great ideas that became today's successful tech startups dwarf anything that had been done to that point. I just couldn't imagine what the next ten years of innovation would look like. Yet my feeling of having missed out prevented me from experimenting with ideas that might have worked.
Nonetheless, my first startup was a tool for college students from elite colleges to create resumes and help them find jobs. Getting students to create their online profile was easy, but getting employers to pay for access proved difficult. Unfortunately, we were fixated on "building a real business" and never noticed that maybe students would want to share their profiles with each other. Could we have built the first college-based social network five years before Facebook? Maybe, but the thought never even entered our minds. (You can even see the humiliating evidence of my smug incompetence in this absurd article from 1999.) We were focused on revenue, but we didn't understand that revenue is not important for its own sake in an early stage company. Instead, we should have thought of it as an indication of validated learning.
And speaking of Facebook, I definitely didn't think it was a good idea when I first heard about it. Heck, I'd already seen Friendster flame out. What was different this time? And Google? No business model, either. I turned down two opportunities to interview at Google in its pre-AdWords days. It seemed like just a bunch of research-oriented PhD's. Oops. And then, at my first Silicon Valley startup, I watched friends get laid off in successive waves as it started to fail. Almost all of them got scooped up by pre-IPO Google this time. I was "lucky" to not be laid off, or so I thought. Yet, as it turns out, the earlier you got laid off, the earlier you got your Google options. That year, right before the IPO, those months mattered a great deal in terms of financial outcomes.
And while I'm confessing, let me add that I knew Matt Cohler way before he was famous. When he left his consulting job to join LinkedIn (whatever that was), I didn't think twice about it. In fact, I remember sending him and his obscure-to-me co-founder (aka Reid Hoffman) a bug report early-on, instead of taking them up on their offer of an in-person meeting. Doh! And when Matt left LinkedIn to take the reins at Facebook; once again, it didn't register. I was much more focused on other transient success stories of the day. I managed to be envious of dozens of other companies, founders, and colleagues who seemed to be having tremendous success but later turned out to be a mirage. If you'd asked me to rank the top most important people I'd met that year, I doubt it would look very impressive in retrospect.
I'm confident of that last statement, because I made the same mistake again a few years later, when I won an award in 2007. BusinessWeek named me one of the top young entrepreneurs in tech, based on my work at IMVU. Being called a techno-wonderboy in front of everyone I knew was pretty strange, and it felt stranger still to be taking credit for the hard work of the many people who really made IMVU a success. But they were very supportive, and the experience was a good one. Trying to take full advantage of the opportunity, I even reached out and met a few of the other award winners. But, looking back, was it obvious which of the other winners were destined to create world-changing companies? Nope. I completely missed Twitter, for example, which was just one more company to me. Oops.
And yet, missing out on these trends wasn't the end of the world. If I had joined Google early, I'd never have had the opportunity to be part of the founding of IMVU. Then I wouldn't have the chance to work with the incredible employees and mentors from whom I learned so much. And, as I've said many times on this blog, if it weren't for those colossal failures and embarrassing missteps, I'd never have learned anything of significance. So, looking back, I'm grateful for the failures and missed opportunities, embarrassing though they are.
One thing really stands out to me today. I wasted a lot of energy, time, and passion on trend-spotting and trying to compare my success with others. Is it really worthwhile spending time and money trying to impress each other with our supposed successes, especially in a business where real feedback can take five or ten years? We go to mixers, buy fancy offices, focus on PR, and try to one-up each other. I think it's wasteful. Instead, let's focus on building companies that matter, on creating real value for customers, and learning. In time, success will come. And if it doesn't, at least you'll have spent your time doing something intrinsically worthwhile.
In the meantime, don't worry if you can't spot the trends. Neither can the rest of us (well, except for Matt Cohler).
Monday, July 6, 2009
Edit: I finally got YouTube upload working, which has better embedding options than MobileMe. If you want to see the original video, use the link above.
Slides are below:
Before I close, let me quote just one additional bit of twitter feedback from one of the video beta-testers:
bmeschke: @ericries This is such GREAT information. A lot of it i have heard before, but your presentation is so clear. http://bit.ly/IMorBI really value feedback like this; it keeps me going when I have doubts. I worry sometimes about making this blog too much into a travelogue or journal of my public events. My goal is to make everything I post here a substantive contribution to the larger entrepreneurship discussion; I hope to live up to that more often than not. As always, I welcome your feedback. If you have thoughts about the right balance of news, events, and commentary, feel free to leave it as a comment or drop me an email. I truly appreciate your support.
Friday, July 3, 2009
Thank you all so much!
As usual, I'd like to post the slides and then offer some additional commentary. Since I tried out some new material in this talk, it was especially educational to see the reaction. (Note: there is video from this event, but it's not online yet. I had a flip cam with me, and the organizers captured their own video with a tripod and everything. If you're interested in seeing it, leave a comment and I'll see if I can get it online.)
Without further ado...
Now for the feedback. Always have to start with a little praise; thank you all so much!
edzschau: Just returned from great prez by @ericries, don't miss him, great presenter, #leanstartup
manukumar: A startup is a human institution designed to deliver a new product or service under conditions of extreme uncertainty @ericries #leanstartupIt sometimes feels strange to talk to a group of startup founders and try to tell them what a startup is. Yet, I think this definition is too important to leave out. I think it makes explicable why so-called "best practices" that people bring to startups from other contexts fail so miserably. If your practice is not designed to cope with uncertainty, it has no place in a startup - even if your startup is located in government or enterprise.
This was a new meme that I'm trying out. We tend to equate startup success with making money, but that is a poor choice. I think we have to raise our sights a little higher; after all, Ponzi schemes make a lot of money, at least until they blow. They don't really create value. The same has been true of an unfortunate number of startups, they manage to generate a lot of hype, raise a lot of money, and sometimes make some of their investors, employees, or founders rich. But did they leave the world a better place than before they existed? Now, I don't think even most dot-com era founders were bad people, I just think traditional startup methods make it too easy to become confused about whether we're creating real progress or not.
bigs: @ericries says Stealth dev is a (undesirable, failure-presaging) customer-free zone. #LeanStartupOf course, a big enabler of those kinds of mistakes is stealth-mode. Another recent meme that I hope more and more startups will take to heart: "stealth is a customer-free zone." There are rare times when stealth is a good strategy, but it amplifies risks without necessarily improving rewards. Danger, Will Robinson.
davemcclure: amazing concepts on Continuous Development => "Cluster Immune System" @EricRies #LeanStartup @fbFund http://yfrog.com/bf50cjAnother new idea in the section on continuous deployment and the cluster immune system. Although there is cost and overhead associated with continuous deployment, the benefits are immense. One such benefit is that, when combined with A/B testing, you can try out small features in less than the amount of time it takes to argue or prioritize them. Nothing is more demoralizing to an engineering team. Prioritizing in a vacuum is a leading source of waste.
dalelarson: Because most features take longer to argue and prioritize than to build. -@ericries #leanstartup
jellytheory: Ask WHY? 5X when something unexpected happens: behind every tech problem is a human one. #leanstartup
mashagenzel: Absolutely true, "Startups' competitive advantage is being able to go through customer validation loop faster" via @ericries #leanstartup
biganderson: Visionary (paying) customers: more visionary than founders, bc they live with the problem that fdrs are trying to solve #leanstartup #fbfund
And last but certainly not least, I wanted to try out a line that I haven't used in a long time. The power of A/B testing is so under-exploited in product development, that I'm trying new ways to explain its benefits. Remember that we can use split-testing for both the problem team and solution team, and that causes a lot of confusion. Split-testing is great for linear optimization; making our landing pages, conversion rates, and retention metrics incrementally better day-in day-out. But it's also amazing for testing big hypotheses, like what our customers really want to get out of our product. If you're not doing both, you're missing out.
Last, I was really struck by one section of the talk that seems to have been completely ignored: a new section on the power of Small Batches. Given that this is one of the most powerful concepts for building and iterating faster, I'm surprised that it didn't register. So, here's a request: how can I make this punchier, clearer, or just generally better? Any ideas?
Anyway, thanks so much to everyone who came out. I had an incredible time. See you next time.
Thursday, July 2, 2009
In a previous post, I outlined the benefits of Five Whys: that it allows you to make large investments in infrastructure incrementally, takes advantage of the 80/20 rule to free up resources immediately, and helps organizations become built to learn. Today, I want to talk about the mechanics of Five Whys in greater detail.
First, a caveat. My intention is to describe a full working process, similar to what I’ve seen at IMVU and other lean startups. But as with all process changes, it should not be interpreted as a spec to be implemented right away. In fact, trying too much is just as dangerous at not doing enough. Just as the lean movement has taught us to build incrementally, it has also taught us to attempt process changes incrementally as well. You need to transition to a work flow of small batches – in small batches.
Five Whys involves holding meetings immediately following the resolution of problems the company is facing. These problems can be anything: development mistakes, site outages, marketing program failures, or even internal missed schedules. Any time something unexpected happens, we could do some root cause analysis. Yet it’s helpful to begin by tackling a specific class of problems. For example, a common place to get started is with this rule: any time we have a site outage of any duration, we will hold a post-mortem meeting immediately afterwards.
The first step is to identify three things about the meeting: what problem we are trying to solve, who will run the meeting, and who was affected by the problem. For the problem, it’s essential to hold the meeting immediately following a specific symptom. Five Why’s rarely works for general abstract problems like “our product is buggy” or “our team moves too slow.” Instead, we want to hold it for a specific symptom, like “we missed the Jan 6 deadline by two weeks” or “we had a site outage on Nov 10.” Have faith that if a large general problem is really severe, it will be generating many symptoms that we can use to achieve a general solution.
Always explicitly identify the person running the meeting. Some organizations like to appoint a “Five Whys master” for a specific area of responsibility. For example, at IMVU we had masters appointed for topics like site scalability or unit test failures. The advantage of having an expert run each meeting is that this person can get better and better at helping the team find good solutions. The downside is the extra coordination required to get that person in the room each time. Either way can work. In any event, nobody should hold a master position for too long. Rotation is key to avoid having a situation where one person becomes a bottleneck or single point of failure.
The person running the meeting does not have to be a manager or executive. They do need to have the authority necessary to assign tasks across the organization. That’s because Five Whys will often pierce the illusion of separate departments and discover the human problems that lurk beneath the surface of supposedly technical problems. In order to make Five Whys successful, the person running the meeting has to have the backing of an executive sponsor who has the requisite authority to back them up if they wind up stepping in political landmines. But this executive sponsor doesn’t need to be in the room – what matters is that everyone in the room understands that the person running the meeting has the authority to do so. This means that if you are trying to introduce Five Whys into an organization that is not yet bought-in, you have to start small.
In order to maximize the odds of success, we want to have everyone affected by the problem in the meeting. That means having a representative of every department or function that was affected. When customers are affected, try to have someone who experienced the customer problem first-hand, like the customer service rep who took the calls from angry customers. At a minimum, you have to have the person who discovered the problem. Otherwise, key details are likely to be missed. For example, I have seen many meetings analyzing a problem that took a long time to be diagnosed. In hindsight, the problem was obvious. If the people responsible for diagnosis aren’t in the post-mortem meeting, it’s too easy to conclude, “those people were just too stupid to find the problem” instead of focusing on how our tools could make problems more evident and easier to diagnose.
A root cause analysis meeting has a clear problem, leader, and stakeholders. The most important guideline for the meeting itself is that the purpose of the meeting is to learn and to improve, not to assign blame or to vent. Assume that any problem is preventable and is worth preventing. Problems are caused by insufficiently robust systems rather than individual incompetence. Even in the case of a person making a mistake, we have to ask “why do our tools make that mistake so easy to make?”
The heart of the meeting is the analysis itself. For each problem, we want to ask “why did that happen?” and “why wasn’t it prevented by our process?” We do that iteratively until we have at least five levels of analysis. Of course, the number five is not sacrosanct; it’s just a guideline. What’s critical is that we don’t do too few levels, and we don’t do too many. One hundred whys would be overwhelming. But if we stay stuck at the technical parts of the problem, and never uncover the human problems behind them, we’re not going far enough. So I would keep the meeting going until we’re talking about human problems, and preferably system-level problems. For example, a site outage may seem like it was caused by a bad piece of code, but: why was that code written? Why didn’t the person who wrote it know that it would be harmful? Why didn’t our tests/QA/immune system catch and prevent the problem? Why wasn’t it immediately obvious how to fix the problem?
Pay attention to whether people are comfortable “naming names” in the meeting. If people are afraid of blame, they’ll try to phrase statements in vague, generic terms or use the passive voice, as in “a mistake was made” rather than “So-and-so failed to push the right button.” There’s no easy fix to this problem. Trust takes time to build up, and my experience is that it may take months to establish enough trust that people are confident that there won’t be retribution for speaking up candidly. Stay patient, and be on alert for blame-type talk or for post-meeting revenge. I recommend a zero-tolerance policy for these behaviors – otherwise our Five Whys meetings can descend into Five Blames.
Another common issue is the tendency of root causes to sprout branches. Complex problems rarely have only one cause, and looking for the primary cause is easier in theory than in practice. The branching of causes is also a prime target for so-called “anchor draggers” – people who aren’t really on board with the exercise in the first place. An easy way to derail the meeting is to keep insisting that more and more lateral causes be considered, until the team is running around in circles. Even well intentioned people can wreak the same havoc by simply staying over-focused on technical or ancillary issues. Try to stay focused on just one line of inquiry. Remember, Five Whys is not about making an exhaustive survey of all the problems. It’s about quickly identifying the likely root cause. That’s why it’s more important to do Five Whys frequently than to get it exactly right. It’s a very forgiving practice, because the most wasteful problems will keep clamoring for attention. Have faith that you’ll have many more opportunities to tackle them, and don’t get hung up on any particular solution.
Once you’ve found approximately five levels of the problem, which includes at least one or two human-level issues, it’s time to turn to solutions. The overall plan is to make a proportional investment in each of the five levels. The two major guidelines are: don’t do too much, and don’t do nothing. Almost anything in between will work.
For example, I often cite a real example of a problem that has as its root cause a new employee who was not properly trained. I pick that example on purpose, for two reasons: 1) most of the companies I work with deal with this problem and yet 2) almost none of them have any kind of training program in place for new employees. The reason is simple: setting up a training program is seen as too much work to be justified by the problem. Yet in every situation where I have asked, nobody has been tasked with making a realistic estimate, either of the impact of this lack of training or the real costs of the solution. In fact, even the investigation itself is considered too much work. Five Whys is designed to avoid these nebulous arguments. If new employees are causing problems, that will be a routine topic. If those problems are minor, each time it happens we’ll budget a small amount of time to make progress on the solution.
Let’s imagine the ideal solution would be to spend six weeks setting up a training program for new employees. You can almost hear a manager now: “sure, if you want me to spend the next six weeks setting this up, just let me know. It’s just a matter of priorities. If you think it’s more important than everything else I’m working on, go right ahead and find someone else to take over my other responsibilities…” This logic is airtight, and has the effect of preventing any action. But Five Whys gives us an alternative. If we’ve just analyzed a minor problem that involved a new employee, we should make a minor investment in training. To take an extreme example, let’s say we’ve decided to invest no more than one hour in the solution. Even in that case, we can ask the manager involved to simply spend the first hour of the six-week ideal solution. The next time the problem comes up, we’ll do the next hour, and so on.
In fact, at IMVU, we did exactly that. We started with a simple wiki page with a few bullet points of things that new engineers had tripped over recently. As we kept doing root cause analysis, the list grew. In response to Five Whys that noticed that not all new engineers were reading the list, we expanded it into a new engineer curriculum. Soon, each new engineer was assigned a mentor, and we made it part of the mentor’s job to teach the curriculum. Over time, we also made investments in making it easier to get a new engineer set up with their private sandbox, and even dealt with how to make sure they’d have a machine on their desk when they started. The net effect of all this was to make new engineers incredibly productive right away – in most cases, we’d have them deliver code to production on their very first day. We never set out to build a world-class engineering-training process. Five Whys simply helped us eliminate tons of waste by building one.
Returning to the meeting itself, the person running the meeting should lead the team in brainstorming solutions for each of the problems selected. It’s important that the leader be empowered to pick one and only one solution for each problem, and then assign it to someone to get done. Remember that the cost of the solutions is proportional to the problem caused. This should make it easy to get buy-in from other managers or executives. After all, if it’s a severe problem like a site outage, do they really want to be seen as the person getting in the way of solving it? And if it’s a minor problem, are they really going to object to a few hours of extra work here and there, if it’s towards a good cause? My experience is: usually not.
There are no fixed rules for what constitutes a proportional investment. As teams get experience doing Five Whys, they start to develop rules of thumb for what is reasonable and what isn’t. To restate: the key is that all parties, including the non-technical departments, see the investments as reasonable. As long as we don’t veer to either extreme, the 80/20 rule will make sure that we don’t under-invest over the long term. Remember that if something is a serious problem, it will keep coming up over and over in these meetings. Each time, we’ll get to chip away at it, until it’s no longer a problem.
The last element of a good Five Whys process is to share the results of the analysis widely. I generally recommend sending out the results to the whole company, division, or business unit. This accomplishes two important things: it diffuses knowledge throughout the organization, and it provides evidence that the team in question is taking problems seriously. This latter point can eliminate a lot of waste. I have been amazed how many teams have severe inter-departmental trust issues caused by a lack of communication about problems. For example, engineering feels that they are constantly being pressured to take shortcuts that lower the quality of the product. At the same time, the very marketing people who are applying that pressure think the engineering team doesn’t take quality seriously, and doesn’t respond appropriately when their shoddy work leads to customer problems. Sharing Five Whys can alleviate this problem, by letting everyone know exactly how seriously problems are taken. I say exactly, because it may actually reveal that problems are not taken seriously. In fact, I have seen people in other departments sometimes catch sloppy thinking in a Five Whys report. By sharing the analysis widely, that feedback can flow on a peer-to-peer basis, quickly and easily.
Most organizations are unaware of how much time they spend firefighting, reworking old bugs, and generally expending energy on activities that their customers don’t care about. Yet getting a handle on these sources of waste is hard, especially because they are dynamic. By the time a tops-down company-wide review figured out the main problems, they’d have shifted to another location. Five Whys allows teams to react much faster and then constantly adapt. Without all that waste in their way, they simply go faster.
(If you’re new to Five Whys, I’m eager to hear your feedback. Does this help you get started? What questions or concerns do you have? Leave your thoughts in a comment. If you’ve tried Five Whys, please share your experiences so far. I’ll do my best to help.)