Please teach kids programming, Mr. President

Of course, what I really mean is: let them teach themselves. I'll explain in a moment.

After volunteering for the Obama campaign last year, a friend of mine insisted that I write a letter to our new President telling him what I thought he should do. This post is the result. Now, I don't pretend to be an expert on macroeconomics, international grand strategy or even enough of a policy wonk to make serious recommendations on how best to implement Issue X Reform. I can only speak to my little corner of the American experience. Here's what I do know:

1. The future strength of our economy depends on its ability to create, support, and sustain entrepreneurs. (If you are somehow not convinced of this point, I'll let Fareed Zakaria explain)
   
2. We know who the next generation of entrepreneurs are going to be. They are in school, right now, all across this country.
   
3. They are nerds.

I'm not offering extensive studies or research to support this conclusion; the evidence from my peers right here in the innovation capital of America, Silicon Valley, is absolutely overwhelming. Almost to a person, we learned the key skills that would enable us to compete in this new economy in between shifts of highly regimented classes and turns of humiliation at the hands of our more popular peers. (See Paul Graham's Why Nerds are Unpopular to learn more)

Take a look at this article on a programming Q&A site: How old are you, and how old were you when you started coding? There are over forty pages of responses from programmers of all ages, and if you just read the stories at random, you'll see a clear pattern. (Or, if you prefer a more quantitative analysis, one of the commenters has helpfully summarized them in graph form. We are nerds, after all. Here's the most striking thing about the statistics of this post: the average "age when started programming" is 13. Think of how many 10-year-olds there must be in the data to balance out the occasional person who started mid-career.

That data is completely consonant with the people I know who are successful technologists today, and similar patterns are documented in each recent wave of technology innovation. I am especially grateful to Malcolm Gladwell for reintroducing the stories of people like Bill Gates and Bill Joy into the mainstream discourse. What's striking about these stories, if you get past the PR hype, are two very important themes:

1. These prodigies were self-taught, and had a fundamental fascination with technology from a very young age.
   
   
2. Their stories would not have been possible without access to sympathetic adults with the necessary equipment and knowledge to get started.

It's this second point I want to emphasize. I've seen it first-hand in my own story. I learned programming on my own, because I thought it was fun. It was only years later that I discovered the shocking truth that you could get paid to do it for a living. But I was also very lucky. I went to excellent public schools (so-called science magnets) that had computer classes, which meant they had networked computers that I could use. My parents had computers in the house for as long as I can remember, and tolerated my obsession with them despite the fact that it seemed like a strange hobby with no obvious benefits. Without their support, there is no chance I would be where I am today. They didn't teach me to program, but they allowed me to teach myself. That's what I'd like every child to have the opportunity to do.

My belief is that, right now, even in the worst and most under-served schools in the country, there are kids with the same potential as Bill Joy. They are probably bored. They are getting beat up by their peers, getting into trouble with their teachers, and generally having a pretty bad time. Those are the kids I think we have an obligation and an opportunity to reach. I don't think we can rescue them from humiliation (that would require a seriousness about education reform I don't see any evidence we're ready for), but I do think we can offer them an escape. And it just so happens that escape is to an activity essential for the future of our civilization. I think it's a pretty good deal.

I didn't learn to program from school, although it sometimes happened at school. In fact, it often got me in trouble. We were supposed to learn how to use computers via a carefully structured curriculum that taught us basic concepts one at a time, slowly advancing the whole class through a regimented program. You've probably read accounts like this, from other arrogant nerds, but bear with mine: in the first week, my nerdy friends and I had already mastered the whole curriculum. We spent the rest of our time pretending to work on the assigned homework, but really trying to do interesting side projects, like sending juvenile messages across the school network or building primitive video games. We did our best not to get caught by our teachers or noticed by our peers. Our fear was well substantiated: both had severe consequences.

Later, I discovered the incredible world of online role-playing games, called MUDs. These were primitive open-ended video games created by the players themselves, using simple programming languages. I spent endless hours getting the world's best introduction to object-oriented programming, and I didn't even know I was doing work. MUDs made the essential truth about software into a powerful metaphor: that code is magic, giving those who wield it the ability to create new forms of value literally out of thin air. We also learned that law is code, and that leadership was needed to build thriving communities in a digital age. You can find the origins of many successful companies in these early lessons.

So all I'm asking, on behalf of the thousands of nerds who could one day change the world for the better, is that we give them access to simple, open, programmable devices; a little time to work on them; and a safe space to work in. They'll take it from there. They don't need adult supervision, or a certified curriculum. If we network them together, they'll answer each others' questions and collaborate on projects we can hardly imagine.

Those of us who made it stand ready to do our part. Given the opportunity, we will build the systems these kids need. We will answer their questions. We will mentor them to get them started, and give them jobs and internships when they are ready. Asked to help, I am confident that Silicon Valley and every other innovation center will step up.

But I do think this requires participation from the public sector, too. There are three threats that are limiting the opportunity to unlock these kids' creativity:

1. Inequity of access. Too many kids today don't have access to computers, cell phones, video games or other programmable devices. We need to leverage every part of our public inf, including public schools and libraries, to make access for those that want to learn programming universal. This doesn't have to be expensive - in fact, many of the physical devices are in place. But we need to open up access to kids so that they can use, program, and remix them on their own terms.
   
   
2. DRM and other restrictions. Increasingly, today's computers and video games are not programmable, they are locked to their users. There would be no Microsoft, Sun Microsystems, or countless other job-creating tech companies today if early computers required corporate authorization to use.
   
   When I was a kid, the way I logged onto the internet for the first time (to play MUDs, naturally) was through an open dial-up console at San Diego State University. When I say open, it's hard to believe how open it was: just dial the number, and you were dropped directly at a UNIX prompt. No logins, no codes, just raw uncensored internet access.
   
   
3. School hostility to phones, nerds, and other things they don't understand. An awful lot of kids have cell phones, and schools are busy banning them from classrooms. What a lost opportunity! Kids are voluntarily bringing a portable networked supercomputer to class, and we want to restrict them to pencil and paper?
   
   A modern phone like the iPhone is a miraculous device. But it's not very open, and not very programmable, unless you have an expensive Mac and an approved developer license. We need to think about how to make these devices programmable by their users, so that they can grow and share as soon as the innovation bug bites them. You might not enjoy typing in code on such a small device, but kids don't mind. I know; in class I used to write video games for the TI-82, a graphing calculator provided to me by my school. Sure it was tedious, but compared to the alternatives, I thought it was great.
   

Each of these trends will need to be countered by sensible public policy, and that's what I am hoping for from our new administration.

So that's my plea on behalf of nerds everywhere. If you're interested in helping them out, leave a note in the comments.

Work in small batches

Software should be designed, written, and deployed in small batches.

Of all of the insights I've contributed to the companies I've worked at over the years, the one I am most proud of is the importance of working in small batches. It's had tremendous impact in many areas: continuous deployment, just-in-time scalability, and even search engine marketing, to name a few. I owe it originally to lean manufacturing books like Lean Thinking and Toyota Production System.

The batch size is the unit at which work-products move between stages in a development process. For software, the easiest batch to see is code. Every time an engineer checks in code, they are batching up a certain amount of work. There are many techniques for controlling these batches, ranging from the tiny batches needed for continuous deployment to more traditional branch-based development, where all of the code from multiple developers working for weeks or months is batched up and integrated together.

It turns out that there are tremendous benefits from working with a batch size radically smaller than traditional practice suggests. In my experience, a few hours of coding is enough to produce a viable batch and is worth checking in and deploying. Similar results apply in product management, design, testing, and even operations. Normally I focus on the techniques you need to reduce batch size, like continuous integration. Today, I want to talk about the reasons smaller batches are better. This is actually a hard case to make, because most of the benefits of small batches are counter-intuitive.

Small batches mean faster feedback. The sooner you pass your work on to a later stage, the sooner you can find out how they will receive it. If you're not used to working in this way, it may seem annoying to get interrupted so soon after you were "done" with something, instead of just working it all out by yourself. But these interruptions are actually much more efficient when you get them soon, because you're that much more likely to remember what you were working on. And, as we'll see in a moment, you may also be busy buidling subsequent parts that depend on mistakes you made in earlier steps. The sooner you find out about these dependencies, the less time you'll waste having to unwind them.

Take the example of a design team prepping mock-ups for their development team. Should they spend a month doing an in-depth set of specifications and then hand them off? I don't think so. Give the dev team your very first sketches and let them get started. Immediately they'll have questions about what you meant, and you'll have to answer them. You may surface assumptions you had about how the project was going to go that are way off. If so, you can immediately evolve the design to take the new facts into account. Every day, give them the updated drawings, always with the proviso that everything is subject to change. Sometimes that will require the team to build something over again, but that's rarely very expensive, because the second time is so much more efficient, thanks to the knowledge gained the first time through. And over time, the development team may be able to start anticipating your needs. Imagine not having to finish the spec at all, because the team has already found an acceptable solution. I've witnessed that dozens of times, and it's a huge source of time-savings.

Small batches mean problems are instantly localized. This is easiest to see in deployment. When something goes wrong with production software, it's almost always because of an unintended side-effect of some piece of code. Think about the last time you were called upon to debug a problem like that. How much of the time you spent debugging was actually dedicated to fixing the problem, compared to the time it took to track down where the bug originated?

Small batches reduce risk. An example of this is integration risk, which we use continuous integration to mitigate. Integration problems happen when two people make incompatible changes to some part of the system. This comes in all shapes and sizes. You can have code that depends on a certain configuration that's deployed on production. If that configuration changes before your code is deployed, the person who changes it won't know they've introduced a problem. Your code is now a ticking time bomb, waiting to cause trouble when it's deployed.

Or consider the case of code that changes the signature of a commonly-called function. It's easy to find collisions if you make a drastic change, but harder when we do things like add new default parameters. Imagine a branch-based development system with two different who each added a new, but different, default-value argument to the end of the signature, and then gone through and updated all its callers. Anyone who has had to spend hours late at night resolving one of these conflicts knows how painful they are. The smaller the batch size, the sooner these kinds of errors are caught, and the easier the integration is. When operating with continuous deployment, it's almost impossible to have integration conflicts.

Small batches reduce overhead. In my experience, this is the most counter-intuitive of its effects. Most organizations have their batch size tuned so as to reduce their overhead. For example, if QA takes a week to certify a release, it's likely that the company does releases no more than once every 30 or 60 days. Telling a company like that they should work in a two-week batch size will sounds absurd - they'd spend 50% of their time waiting for QA to certify the release! But this argument is not quite right. This is something so surprising that I didn't really believe it the first few times I saw it in action. It turns out that organizations get better at those things that they do very often. So when we start checking in code more often, release more often, or conduct more frequent design reviews, we can actually do a lot to make those steps dramatically more efficient.

Of course, that doesn't necessarily mean we will make those steps more efficient. A common line of argument is: if we have the power to make a step more efficient, why don't we invest in that infrastructure first, and then reduce batch size as we lower the overhead? This makes sense, and yet it rarely works. The bottlenecks that large batches cause are often hidden, and it takes work to make them evident, and even more work to invest in fixing them. When the existing system is working "good enough" these projects inevitably languish and get deprioritized.

Take the example of the team that needs a week to certify a new release. Imagine moving to a two-week release cycle, with the rule that no additional work can take place on the next iteration until the current iteration is certified. The first time through, this is going to be painful. But very quickly, probably even by the second iteration, the weeklong certification process will be shorter. The development team that is now clearly bottlenecked will have the incentive needed to get involved and help with the certification process. They'll be able to observe, for example, that most of the certification steps are completely automatic (and horribly boring for the QA staff) and start automating them with software. But because they are blocked from being able to get their normal work done, they'll have a strong incentive to invest quickly in the highest ROI tests, rather than overdesigning a massive new testing system which might take ages to make a difference.

These changes pay increasing dividends, because each improvement now direclty frees up somebody in QA at the same time as reducing the total time of the certification step. Those freed up QA resources might be able to spend some of that time helping the development team actually prevent bugs in the first place, or just take on some of their routine work. That frees up even more development resources, and so on. Pretty soon, the team can be developing and testing in a continuous feedback loop, addressing micro-bottlenecks the moment they appear. If you've never had the chance to work in an environment like this, I highly recommend you try it. I doubt you'll go back.

If you're interested in getting started with the transition to small batches, I'd recommend beginning with Five Whys.

(I have infuriated many coworkers by advocating for smaller batch sizes without always being able to articulate why they work. Usually, I have to resort to some form of "try it, you'll like it," and that's often sufficient. Luckily, I now have the benefit of a forthcoming book, The Principles of Product Development Flow. It's really helped me articulate my thinking on this topic, and includes an entire chapter on the topic of reducing batch size.)

Continuous deployment with downloads

One of my goals in writing posts about topics like continuous deployment is the hope that people will take those ideas and apply them to new situations - and then share what they learn with the rest of us. So I was excited to read a recent post about applying the concept of continuous deployment to that thickest-of-all-clients, the MMOG. Joe Ludwig goes through his current release process and determines that the current time to make and deploy a release is about seven and half hours, which is why his product is released about once a month. While that's actually quite speedy for a MMOG, Joe goes through the thought experiment of what it would take to do it much faster:

Programmer Joe � Continuous Deployment with Thick Clients
If it takes seven and a half hours to deploy a new build you obviously aren’t going to get more than one of them out in an 8 hour work day. Let’s forget for a moment that IMVU is able to do this in 15 minutes and pretend that our target is an hour. For now let’s assume that we will spend 10 minutes on building, 10 minutes on automated testing, 30 minutes on manual testing ...

In fact, if those 30 minutes of manual testing are your bottleneck and you can keep the pipeline full, you can push a fresh build every 30 minutes or 16 times a day. Forget entirely about pushing to live for a moment and consider what kind of impact that would have on your test server. Your team could focus on fixing issues that players on the test server find while those players are still online. Assuming a small change that you can make in half an hour, it would take only an hour from the start of the work on that fix to when it is visible to players. That pace is fast enough that it would be possible to run experiments with tuning values, prices of items, or even algorithms.Of course for any of this to work the entire organization needs to be arranged around responding to player feedback multiple times per day. The real advantage of a rapid deployment system is to make your change -> test -> respond loop faster.

This is a great example of lean startup thinking. Joe is getting clear about what steps in the current process actually deliver value to the company, the imagining a world in which those steps were emphasized and others minimized. Of course, as soon as you do that, you start to reap other benefits, too.

I'd like to add one extra thought to Joe's thought experiment. Let's start with a distinction between shipping new software to the customer, and changing the customer's experience. The idea is that often you can change the customer's experience without shipping them new software at all. This is one of the most powerful aspects of web architecture, and it often gets lost in other client-server programming paradigms.

From one point of view, web browsers are a horribly inefficient platform. We often send down complete instructions for rendering a whole page (or even a series of pages) in response to every single click. Worse, those instructions often kick off additional requests back and forth. It would be a lot more efficient to send down a compressed packet with the entire site's data and presentation in an optimized format. Then we could render the whole site with much less latency, bandwidth usage, and server cost.

Of course, the web doesn't work this way for good reasons. Its design goals aren't geared towards efficiency in terms of technical costs. Instead, it's focused on flexibility, readability, and ease of interoperability. For example, it's quite common that we don't know the exact set of assets a given customer is going to want to use. By deferring their selection until later in the process, we can give up a lot of bookkeeping (again trading off for considerable costs). As a nice side-effect, it's also an ideal platform for rapid changes, because you can "update the software" in real time without the end-user even needing to be aware of the changes.

Some conclude that this phenomenon is made possible because the web browser is a fully general-purpose rendering platform, and assume that it'd be impossible to do this in their app without creating that same level of generality. But I think it's more productive to think of this as a spectrum. You can always move logic changes a little further "upstream" closer to the source of the code that is flowing to customers. Incidentally, this is especially important for iPhone developers, who are barred by Apple Decree from embedding a programming language or interpreter in their app (but who are allowed to request structured data from the server).

For example, at IMVU we would often run split-test experiments that affected the behavior of our downloadable client. Although we had the ability to do new releases of the client on a daily basis (more on this in a moment), this was actually too slow for most of the experiments we wanted to run. Plus, having the customer be aware that a new feature is part of a new release actually affects the validity of the experiment. So we would often ship a client that had multiple versions of a feature baked into it, and have the client call home to find out which version to show to any given customer. This added to the code complexity, latency, and server cost of a given release, but it was more than paid back by our ability to tune and tweak the experimental branches in near-real-time.

Further upstream on the spectrum are features that can be parameterized. Common examples are random events which have some numeric weighting associated with them (and which can be tuned) or user interface elements that are composed of text or graphics. We tacked on a feature to the IMVU client that worked like this: whenever the client called home to report a data-warehousing event, we used to have a useless return field (the client doesn't care if the event was successfully recorded). We repurposed that field to optionally include some XML describing an on-screen dialog box. That meant we could notify some percentage of customers of something at any time, which was great for split-testing. A new feature would often be first "implemented" by a dialog box shown to a few percent of the userbase. We'd pay attention to how many clicked the embedded link to get an early read on how much they cared about the feature at all. Often, we'd do this before the feature existed at all, apologizing all the way.

There are plenty more techniques even further upstream. Eventually, you wind up with specialized state machines, interpeters or a full-fledged embedded platform. We eventually embedded the Flash interpreter into our process, so we could experiment with our UI more quickly.

In fact, we considered releases themselves to be a special case of this more general system. We had a structured automated release process. After all, the release itself was just a static file checked into our website source control. Every new candidate release was automatically shown to a small number of volunteers, who would be prompted to upgrade. The system would monitor their data and if it looked within norms gradually offer the release to a small number of new users (who had no prior expectation of how the product should work). It would carefully monitor their behavior, and especially their technical metrics, like crashes and freezes. If their data looked OK, we'd have the option to ramp up the number of customers bit by bit until finally all new users were given the new release and all existing users were promtped to upgrade. Although we'd generally do a prerelease every day, we wouldn't pull the trigger on a full release that often, because our upgrade path for existing users wasn't (yet) without cost. It also gave our manual QA team a chance to inspect the client before it was widely deployed, due to the lower level of test coverage we have on that part of the product (it's much harder).

In effect, every time we check in code to our client code base, we are kicking off another split-test experiment that asks: "is the business better off with this change in it than without it?" Because of the mechanics of our download process, this is answered a little slower than on the web. But that doesn't make it any less important to answer.

To return to the case of the thick-client MMOG, there are some additional design constraints. It's more risky to have different players using different versions of the software, because that might introduce gameplay fairness issues. I think Joe's idea of deploying to the test server is a great way around this, especially if there is a regular crew of players subjecting the test server to near-normal behavior. But I also think this could work in a lot of production scenarios. Take the case of determining the optimal spawn rate for a certain monster or treasure. Since this is a parameterized scenario, it should be possible to do time-based experiments. Change the value periodically, and have a process for measuring the subsequent behavior of customers who were subjected to each unique value. If you want to be really fancy, you can segregate the data for customers who saw multiple variations. I bet you'd be able to write a simple linear optimizer to answer a lot of design questions that way.

My experience is that once you have a tool like this, you start to use it more and more. Even better, you start to migrate your designs to be able to take ever-increasing advantage of it. At IMVU, we found ourselves constantly migrating functionality upstream, in order to get faster iteration. It was a nearly uncoscious process; we just felt that much more productive with rapid feedback.

So thanks for sharing Joe! Good luck with your thought experiment, and let us know if you ever decide to make those changes a reality.

What is a market? (a guide for hackers)

(This post was inspired by a conversation with Nivi from Venture Hacks, but is otherwise not his fault)

There has been a proliferation of frameworks and metaphors lately that are designed to help startups avoid the all-too-common fatal mistake of failing to find a market. To wit: achieving product/market fit, getting customer validation, making something people want, things that matter, and of course the many excellent books on the topic, of which I'll mention just two of the best, Crossing the Chasm and The Innovator's Dilemma.

Having worked with dozens of founders over the past few months, I think I can safely say, without naming any names, that most of us are not too clear on what we're talking about when it comes to markets. Is a market a set of paying customers? Are there different types, or are they all similar? If so, how do we think about successful companies that don't charge money for their product? Are advertisers customers, and are they part of a market? Founders are constantly being barraged by incoherent and contradictory advice. For example, focus on finding a big market, but also, don't try to compete against large incumbents who already have a large market. But also, remember that eBay started out with pez dispensers but also remember that Google never advertised and I've heard that Facebook got it right from the start.

Few people want to admit that they don't understand what other people are talking about. That's especially true when jargon is flying and the stakes are high. For people who went to business school, I don't really know what to say. But I work with a lot of hackers-turned-founders who I do relate to. There's no earthly reason we should expect a programmer will have picked up a good understanding of market dynamics along the way, while they were busy figuring out how to grok partial template specialization.

Here's my attempt to explain market types using a metaphor most of us should be able to understand. It takes advantage of the idea, which I owe to Clayton Christensen, that customers buying a product are really "hiring" it to do a specific "job" for them. It's as if every customer, whether they are an enterprise, a small business, or an individual consumer, is actually an employer who wants to get things done. "People don't want to buy a quarter-inch drill. They want a quarter-inch hole!"

So imagine you're applying for a programming job. You're smart and have l33t skills. And yet, depending on who you talk to and what jobs you apply for, you may find an easier or harder time getting the job. Those differences are actually predictable, and fall into categories, and those categories are called market types. There are four:

1. Existing market. This is applying for an open job req. An employer is trying to hire someone for a specific job, and they think they know what that job is. You might be smarter than the other applicants, but there are zillions of them. Because the employer has so many choices, they can afford to be very picky. This is the world of incomprehensible job postings, jargon, and HR keyword-based screening. It's important to understand how these companies see you: your resume looks like crap, your references are poor or nonexistent, and you don't have the mandated 10 years of experience in J2EE ERP CRM WTF. Tough sell. In the world of startups, this is like trying to sell a product to a very demanding customer who needs to see a lot of features before buying.
   
   
2. Resegmented market (low cost variety). Here's a situation where you offer to do the job for 1/10th the cost. Instead of being paid $120k/yr, you are willing to do it for $12k/yr. This is how outsourcers get jobs they aren't otherwise "qualified" for. Keep in mind that a moderate price savings won't get it done; you can't call up traditional HR and say "I'll do the job for $105k" because that's not a meaningful sum to most companies, especially compared to the cost of making the wrong hire. In fact, you can't go through traditional channels at all. Recruiters and HR departments rarely recommend outsourcing - more likely, someone inside the company needs to get work done cheaply and circumvents the established hiring process to use an outside (cheap) vendor.
   
   Here your challenge is to convince that manager that you really can do the job at such low cost. Now your crappy resume and low-grade references become a strength: you're obviously not wasting money on marketing. Why do you think outsources have been happy to generate all this PR the past few years about people losing their jobs to India? Although it generates some political backlash, it establishes tremendous credibility among people who are desperately trying to save money. They must be thinking "if it's costing people their jobs, it must really work. Maybe I should try that..."
   
   
3. Resegmented market (niche variety). This involves changing the job description. If the company is looking for programmers, you convince them they absolutely need Ruby programmers. Now, there's nobody with 10 years experience in doing ETL QVC in Ruby, so now your 3 years is starting to look pretty good. And now all those "qualified" candidates who you were competing against in scenario #1 are starting to look unqualified, because although they may be experts in something, they don't know Ruby, and you've convinced the client that Ruby is the be-all-end-all of programming languages. Remember Java?
   
   
4. New market - this is like applying to a company that does not have an open req for programmers. Your challenge is to convince them to hire you anyway, even though they don't know what they need. Now even 10 years of experience is probably not good enough, because they have no idea how many years experience you ought to have. Without a basis for comparison, you first have to drive home the need, then you will have an easy time making the sale (after all, you're the person who they trust to bring it to their attention). A possible approach is to "steal" another job category. For example, "If you hire me you can free up 10 of your staff in department X, because I will write software that ..." Toughest part of the sale is to get agreement that you really can deliver benefits that they didn't previously know were possible.
   
   Here's the good and the bad news about new markets: you don't have any competition. When you call on a customer to try and get th job, it's unlikely that somebody else got there first. Unfortunately, that also means the csutomer is unlikely to know what you're talking about. Be prepared for an extended slog.
   

Think back to the confusing and contradictory advice I mentioned earlier. The #1 best way to cope with advice like that is to know your market type. That way, when someone say something like "you only have a few niche customers, so you can't be in a very big market" you can reply and (as a bonus) know what you're talking about. How about something like "don't worry! We're busy resegmenting the XYZ market with a disruptive low-price offering. Although that currently means we only can serve these few low-end customers, as our product improves, we'll eventually move up-market and kick the incumbents out."

They may not understand what you're talking about, but - don't worry! - they'll probably be too embarrassed to say anything.

You buy virtual goods

Jeremy Liew has a great piece in the WSJ about the central mystery of businesses that make money selling virtual goods:

Why do People Buy Virtual Goods? - WSJ.com: "My theory is that people buy digital goods for the same reason that they buy goods in the real world; (i) to be able to do more, (ii) to build relationships, and (iii) to establish identity."

Jeremy's argument is a good one, and I'm glad to see it being advanced in such a high-profile way. Having had a number of years to try and answer this question at cocktail parties and social gatherings of all kinds, I thought I'd try and expand on his framework a little, and then try and use these answers to help make suggestions for those who are trying to get people to buy virtual goods. Let me start by trying to convince you that, no matter who you are, you already buy virtual goods.

My goal, when talking to people who are new to the virtual worlds concept, is to convince them that they already buy virtual goods. This is always true, because modern economies have become increasingly virtual over the years. Why do Citizens of Humanity jeans cost $200, when physically similar pants can be had for one-tenth the price? The same is true of brand-name products in almost every category (and it can be measured). Brands are so pervasive that even those people who want to make a statement against it (do you build all your own software from source?) have to invest substantial time avoiding it, which is just another kind of premium. (Ironically, one of the best sources for insight on this phenomenon is Naomi Klein's anti-corporate manifesto No Logo.)

My point is not just that brands are a form of virtual goods, although that's true. Beyond their brand, I want to argue that every product contains both tangible and intangible sources of value, and that everything you buy has at least some "virtual goods" component. By recognizing these components, we can make better sense of what's going on in the online virtual goods market, and craft strategies that leverage people's pre-existing experience with virtual goods. This is why brand-based virtual goods have worked so well online. Paying a premium for a branded pair of virtual jeans is actually a pretty similar experience for most customers.

I dissect the value that a product gives to a customer into four sources:

• Practical utility - This is the tangible benefit that a product enables, whether that's transportation, warmth, cleanliness, or entertainment. Many products derive all of their value from this utility, online as well as off: generic drugs are pretty similar to a number of non-epic consumables in your average MMOG.
  
  
• Perceived value - This is the extra value a customer perceives as a result of good marketing, product design, product quality, or exception product/market fit. For example, many customers derive satisfaction from feeling like they bought the "best" product in a given category, even if that product has no objective performance difference from its nearest competitor. This is true even in cases where the customer derives no status benefit from the product (which we'll cover in a second). For example, home electronics brands like Sony and Bose work very hard to create an impression of exceptional performance even in products that are used primarily in private.
  
  
• Social value - When I can use a product to my benefit in a social situation, it can be transformed in value. All gifting-type products are influenced by this source, as Hallmark has long understood. But plenty of other product categories depend on social factors: status purchases, beauty products, fashion products, and (at least here in San Francisco) food and produce. For a non-brand example, look no further than De Beer's successful, if pernicious, marketing of diamonds.
  
  
• Identity value - This is the strongest source of value of all, and it's a little tricky to differentiate from the preceding two sources. This is the benefit you get from incorporating a product into your self-conception. For example, take your average Mac fanatic. When they buy an Apple laptop, they are doing more than enjoying a premium product and showing off. They are saying to the world and - more importantly - to themselves: I am the kind of person that buys Apple products. Apple has done a phenomenal job of convincing us that we, too, can be a little like Steve Jobs, if only we had one more iFoo in our lives. Many fashion and beauty products create this kind of affinity, especially in products that are not visible to others (don't make me spell it out). Identity products are not easily displaced, because the emotional investment is very high. This is every bit as true for online goods - just try and trade your friend's level 80 warlock for your "equivalent" level 80 rogue. Good luck.
  

I've tried to arrange these sources of value in a hierarchy, and although I don't have a lot of evidence that I got it right, this is my gut sense of how they stack up. The nice thing about understanding these sources is that consumers generally have separate budgets for each category. One of the biggest lessons I learned from my time at IMVU was that if you can move spending out of the entertainment budget (which is often constrained) and into the identity budget, you can make a lot more money per customer. Even in tough times (actually, especially in hard times) people spend significant sums to bolster their sense of who they are. For better or worse, products physical and online are parts of that formula for most people in our society.

So when thinking of selling virtual goods, consider combining sources of value into one product. This is true for online goods, for example in supporting item gifting. It's just as true for physical or even hybrid goods, as Webkinz has demonstrated by adding an online source of utility to a physical stuffed animal. Many virtual goods generate only mediocre returns when they draw from only one source of value, as Facebook has been finding out with their only-social-utility gifts, or dozens of startups have found out with their just-the-functionality technology offerings.

And when given the choice, try and move up the hierarchy of value. If given the opportunity to work with two customer segments, one of which sees your product as a basic utility and another of which sees it as a lifestyle statement, choose the latter. IMVU made that choice early on, when we abandoned some profitable customers who wanted to use our product as a regular-IM substitute. There was no way to service them while still engaging with the goths, emos and anime fans who were rapidly becoming IMVU's top evangelists. We doubled-down on identity value, and it worked out well.

The free software hiring advantage

This is one of those startup tips I'm a little reluctant to share, because it's been such a powerful source of competitive advantage in the companies I've worked with. But I'm going to share it anyway, because it feels like the right thing to do. Here's the short version: hire people from the online communities that develop free software. (Yes, you may be more familiar with the term open source, but let's give credit where credit is due, at least for today).

Especially for a startup, not taking maximum advantage of free software is crazy. The benefits are many and much-discussed, and so I'll mention only one in passing. It's one of the easiest ways to get leverage in your development process, amplifying the power of your team by letting you take advantage of code written by thousands of others. It's obvious that can lower your development costs, but I think it's even more important that it can reduce your time to market. It can benefit your team in other, more surprising ways as well.

This approach gives you an edge in hiring. Most of the best programmers I've known are been active in at least one free software project. It's a wonderful filter for people who are intrinsically motivated by the art of programming. Beyond the quality of the candidates themselves, I've noticed three big effects of hiring out of free software communities:

1. You can hire an expert in your own code base. I've had the good fortune to see this first-hand. I hired someone who was a key contributor to a library that was heavily used in our application. Although he didn't know much about our app when he started, he was able to be productive from day one, because we immediately put him to work extending our application's use of the library in question. He saw opportunities none of us could, because his point-of-view about what constituted "legacy code that I know well" and "third party code that some other strange people wrote" were exactly inverted from ours.
   
   
2. You can hire people who have worked together. Another unexpected benefit comes when you hire people who are part of the same online coding community. They share a common language, culture, and coding style. They have a certain amount of trust based on having been part of a common mission, and they share a passion for the project's goals. That's all helpful for recruiting, retaining, and ramping-up a new employee. You don't have to guess who their mentor will be.
   
   
3. You're not competing with every other company in town. This is especially true in Silicon Valley. The great programmers are already being headhunted by 10 other startups, and the cognitive overhead of trying to figure out which are actually unique opportunities is high. Free software contributors tend to be geographically dispersed, and so aren't part of the echo chamber. When you call, it's a more unusual occurrence, and you're more likely to get their attention.
   

If you want to successfully hire from a coding community, don't just barge in. Job posts almost never work; they're considered a form of spam. Instead, engage with the project. Submit patches. Make suggestions for how the project could be improved. Publish examples of how you use it. A surprising number of contributors to these projects have no idea how their work is used; I've often found myself in the position of being the largest single user of the software on the planet, without having ever talked to the people who wrote it. Talk about a reliable, low-maintenance vendor. This effort takes time, so this approach is not for the impatient. Still, the engagement itself is worthwhile, even if you never hire anyone. You're helping people who spend part of their lives helping you. It's basic self-interest.

Once you're part of the community, a big question is who to try and hire. As you get to know the contributors, it's evident who the real leaders are. Here's my heuristic for deciding who to approach. Ignore the famous people who are busy giving lots of speeches about how technology X will change the world. Find the person on the mailing list who patiently corrects newbies' mistakes. When someone writes in with a bad (but earnest) idea, who has the combination of in-depth knowledge and communications skills necessary to correct them without alienating them from the project? Communities that don't attract these kinds of leaders don't scale, so any successful project is bound to have some. It's worth the effort to bring them on board.

Once I've identified one of these superstars, here's what has worked for me. Approach them directly and privately. State your case plainly and be honest. I've found almost anyone will give you the time of day, if you tell them directly: "I am working on a company whose mission I profoundly believe in. We are heavy users of Project X, and are grateful to you for making that possible. Can I have a few minutes of your time?" Build up a relationship over time, find out what makes them tick, and try to make the case that your company is a great place to pursue that passion. If you're telling the truth, they'll come to see it your way eventually.

Given how many startups complain bitterly about how hard it is to find qualified programmers, I'm surprised more don't engage more fully with the people who make their technology stack possible. Try it, you just might like it. If you've never been a contributor to a free software project before, take a look at Contributing to Open Source Without Committing a Line Of Code.

Of course, if you're the target of one of these hiring calls, and not the company doing it, the perspective is pretty different. How do you evaluate a startup as a potential employer? As I promised over at Hacker News, I'll try and tackle that question in a future post. If you're interested in that topic, let me know in a comment.

Continuous deployment and continuous learning

At long last, some of the actual implementers of the advanced systems we built at IMVU for rapid deployment and rapid response are starting to write about it. I find these on-the-ground descriptions of the system and how they work so much more credible than just theory-type posts that I am excited to share them with you. I can personally attest that these guys know what they are talking about; I saw them do it first-hand. I will always be full of awe and gratitude for what they accomplished.

Continuous Deployment at IMVU: Doing the impossible fifty times a day by Timothy Fitz
Continuous Deployment isn’t just an abstract theory. At IMVU it’s a core part of our culture to ship. It’s also not a new technique here, we’ve been practicing continuous deployment for years; far longer than I’ve been a member of this startup.

It’s important to note that system I’m about to explain evolved organically in response to new demands on the system and in response to post-mortems of failures. Nobody gets here overnight, but every step along the way has made us better developers.

The high level of our process is dead simple: Continuously integrate (commit early and often). On commit automatically run all tests. If the tests pass deploy to the cluster. If the deploy succeeds, repeat.

Our tests suite takes nine minutes to run (distributed across 30-40 machines). Our code pushes take another six minutes. Since these two steps are pipelined that means at peak we’re pushing a new revision of the code to the website every nine minutes. That’s 6 deploys an hour. Even at that pace we’re often batching multiple commits into a single test/push cycle. On average we deploy new code fifty times a day.

We call this process continuous deployment because it seemed to us like a natural extension of the continuous integration we were already doing. Our eventual conclusion was that there was no reason to have code that had passed the integration step but was not yet deployed. Every batch of software for which that is true is an opportunity for defects to creep in: maybe someone is changing the production environment in ways that are incompatible with code-in-progress; maybe someone in customer support is writing up a bug report about something that's just being fixed (or worse, the symptom is now changing); and no matter what else is happening, any problems that arise due to the code-in-progress require that the person who wrote it still remember how it works. The longer you wait to find out about the problem, the more likely it is to have fallen out of the human-memory cache.

Now, continuous deployment is not the only possible way to solve these kinds of problems. In another post I really enjoyed, Timothy explains five other non-solutions that seem like they will help, but really won't.

1. More manual testing.

This obviously doesn’t scale with complexity. This also literally can’t catch every problem, because your test sandboxes or test clusters will never be exactly like the production system.

2. More up-front planning

Up-front planning is like spices in a cooking recipe. I can’t tell you how much is too little and I can’t tell you how much is too much. But I will tell you not to have too little or too much, because those definitely ruin the food or product. The natural tendency of over planning is to concentrate on non-real issues. Now you’ll be making more stupid mistakes, but they’ll be for requirements that won’t ever matter.

3. More automated testing.

Automated testing is great. More automated testing is even better. No amount of automated testing ensures that a feature given to real humans will survive, because no automated tests are as brutal, random, malicious, ignorant or aggressive as the sum of all your users will be.

4. Code reviews and pairing

Great practices. They’ll increase code quality, prevent defects and educate your developers. While they can go a long way to mitigating defects, ultimately they’re limited by the fact that while two humans are better than one, they’re still both human. These techniques only catch the failures your organization as a whole already was capable of discovering.

5. Ship more infrequently

While this may decrease downtime (things break and you roll back), the cost on development time from work and rework will be large, and mistakes will continue to slip through. The natural tendency will be to ship even more infrequently, until you aren’t shipping at all. Then you’ve gone and forced yourself into a total rewrite. Which will also be doomed.

What all of these non-solutions have in common is that they treat only one aspect of the problem, but at the expense of another aspect. This is a common form of sub-optimization, where you gain efficiency in one of the sub-parts at the expense of the efficiency of the overall process. You can't make these global efficiency improvements until you get clear about the goal of your development process.

That leads to a seemingly-obvious question: what is progress in software development? It seems like it should be the amount of correctly-working code we've written. Heck, that's what it says right there in the agile manifesto. But, unfortunately, startups can't afford to adopt that standard. As I've argued elsewhere, my belief is that startups (and anyone else trying to find an unknown solution to an unknown problem) have to measure progress with validated learning about customers. In a lot of cases, that's just a fancy name for revenue or profit, but not always. Either way, we have to recognize that the biggest form of waste is building something that nobody wants, and continuous deployment is an optimization that tries to shorten this code-data-learning feedback loop.

Assuming you're with me so far, what will that mean in practice? Throwing out a lot of code. That's because as you get better at continuous deployment, you learn more and more about what works and what doesn't. If you're serious about learning, you'll continuously learn to prune the dead weight that doesn't work. That's not entirely without risk, which is a lesson we learned all-too-well at IMVU. Luckily, Chad Austin has recently weighed in with an excellent piece called 10 Pitfalls of Dirty Code.

IMVU was started with a particular philosophy: We don't know what customers will like, so let's rapidly build a lot of different stuff and throw away what doesn't work. This was an effective approach to discovering a business by using a sequence of product prototypes to get early customer feedback. The first version of the 3D IMVU client took about six months to build, and as the founders iterated towards a compelling user experience, the user base grew monthly thereafter.

This development philosophy created a culture around rapid prototyping of features, followed by testing them against large numbers of actual customers. If a feature worked, we'd keep it. If it didn't, we'd trash it.

It would be hard to argue against this product development strategy, in general. However, hindsight indicates we forgot to do something important when developing IMVU: When the product changed, we did not update the code to reflect the new product, leaving us with piles of dirty code.

So that you can learn from our mistakes, Chad has helpfully listed ten reasons why you want to manage this dirty-code (sometimes called "technical debt") problem proactively. If we could do it over again, I would have started a full continuous integration, deployment, and refactoring process from day one, complete with five why's for root cause analysis. But, to me anyway, one of the most inspiring parts of the IMVU story is that we didn't start with all these processes. We hadn't even heard of half of them. Slowly, painfully, incrementally, we were able to build them up over time (and without ever having a full-stop-let's-start-over timeout). If you read these pieces by the guys who were there, you'll get a visceral sense for just how painful it was.

But it worked. We made it. So can you.

The lean startup @ Web 2.0 Expo (and a call for help)

I've been asked to speak this year at the Web 2.0 Expo to explain the lean startup concept to a larger audience.

The Lean Startup: a Disciplined Approach to Imagining, Designing, and Building New Products.: Web 2.0 Expo San Francisco 2009

The current macroeconomic climate presents unparalleled opportunities for those that can thrive with constrained resources. The Lean Startup is a practical approach for creating and managing a new breed of company that excels in low-cost experimentation, rapid iteration, and true customer insight. It uses principles of agile software development, open source and web 2.0, and lean manufacturing to guide the creation of technology businesses that create disruptive innovation.

1:30pm Wednesday, 04/01/2009 Location: 2022

I'd like to extend my thanks to the people who've come to my recent "office hours" talks and given such valuable feedback. I know some folks weren't able to get in to the most recent one (sorry!), and so I'm looking forward to having a bigger venue to work with. The trade-off, though, is a more structured presentation format. And that's where the call for help comes in.

I have to practice what I preach, right? If you're interested in being part of my "customer advisory board" for this presentation, please get in touch. I'm looking for a few people to run ideas by, give feedback, and generally help craft ideas so that they can have the biggest impact on this audience. I'm especially interested in hearing from those of you who are planning to attend the Web 2.0 Expo or have attended one in the past. If you want to do it, please start with the presentation Steve and I did for Maples Investments a few months ago. That was for a different audience and in a much smaller venue, but I'd like to use it as a starting point. If you'd like to give feedback, feel free to post comments here, or get in touch on LinkedIn, Facebook, or try to beat the spam filter on email.

Lastly for those of you who decide to stop by the Expo, please come say hello and let me know you're a reader, so I can say thanks in person. As a group you've given me such incredible support and feedback these past few months, and I truly appreciate it.

Achieving a failure

We spend a lot of time planning. We even make contingency plans for what to do if the main plan goes wrong. But what if the plan goes right, and we still fail? This is the my most dreaded kind of failure, because it tricks you into thinking that you're in control and that you're succeeding. In other words, it inhibits learning. My worst failures have all been of this kind, and learning to avoid them has been a constant struggle.

See if this plan sounds like a good one to you:

• Start a company with a compelling long-term vision. Don't get distracted by trying to flip it. Instead, try and build a company that will matter on the scale of the next century. Aim to become the "next AOL or Microsoft" not a niche player.
• Raise sufficient capital to have an extended runway from experienced smart money investors with deep pockets who are prepared to make follow-on investments.
• Hire the absolute best and the brightest, true experts in their fields, who in turn can hire the smartest people possible to staff their departments. Insist on the incredibly high-IQ employees and hold them to incredibly high standards.
  
• Bring in an expert CEO with outstanding business credentials and startup experience to focus on relentless execution.
• Build a truly mainstream product. Focus on quality. Ship it when it's done, not a moment before. Insist on high levels of usability, UI design, and polish. Conduct constant focus groups and usability tests.
• Build a world-class technology platform, with patent-pending algorithms and the ability to scale to millions of simultaneous users.
  
• Launch with a PR blitz, including mentions in major mainstream publications. Build the product in stealth mode to build buzz for the eventual launch.

I had the privilege, and the misfortune, to be involved with a startup that executed this plan flawlessly. It took years, tens of millions of dollars, and the efforts of hundreds of talented people to pull it off. And here's the amazing thing about this plan: it actually worked. I think we accomplished every one of those bullet points. Check. Mission accomplished.

Only this company was a colossal failure. It never generated positive returns for its investors, and most of its employees walked away dejected. What went wrong?

This company was shackled by shadow beliefs that turned all those good intentions, and all that successful execution, into a huge pile of wasted effort. Here are a few:

• We know what customers want. By hiring experts, conducting lots of focus groups, and executing to a detailed plan, the company became deluded that it knew what customers wanted. I remember vividly a scene at a board meeting, where the company was celebrating a major milestone. The whole company and board play-tested the product to see its new features first hand. Everyone had fun; the product worked. But that was two full years before any customers were allowed to use it. Nobody even asked the question: why not ship this now? It was considered naive that the "next AOL" would ship a product that wasn't ready for prime time. Stealth is a customer-free zone. All of the efforts to create buzz, keep competitors in the dark, and launch with a bang had the direct effect of starving the company for much-needed feedback.
  
  
• We can accurately predict the future. Even though some aspects of the product were eventually vindicated as good ones, the underlying architecture suffered from hard-to-change assumptions. After years of engineering effort, changing these assumptions was incredibly hard. Without conscious process design, product development teams turn lines of code written into momentum in a certain direction. Even a great architecture becomes inflexible. This is why agility is such a prized quality in product development.
  
  
• We can skip the chasm. As far as I know, there are no products that are immune from the technology life cycle adoption curve. By insisting on building a product for mainstream customers, the company guaranteed that they would be unhappy with the number and type of customers they got for the first version. Worse was the large staff in departments appropriate to a mainstream-scale product, especially in customer service and QA. The passionate early adopters who flocked to the product at its launch could not sustain this outsized burn rate.
  
  
• We can capitalize on new customers. As with many Silicon Valley failures, a flawless PR launch turned into a flawed customer acquisition strategy. The first version product wasn't easy enough to use, install, and pay for. It also had hardware requirements that excluded lots of normal people. Millions of people flocked to the website, but the company could only successfully monetize early adopters. As a result, the incredible launch was mostly wasted.
  
  
• We know what quality means. All of the effort invested in quality, polish, stability and usability turned out to be for nothing. Although the product was superior to its competitors in many ways, it was missing key features that were critical for the kinds of customers who never got to participate in the company's focus groups (or were represented on its massive QA staff). Worse, many of the wrong assumptions built into the technical architecture meant that, in the real world outside the testing lab, the product's stability was nothing to write home about. So despite the millions invested in quality, the end result for most customers was no better than the sloppy beta-type offerings of competitors.
  
  
• Advancing the plan is progress. This is the most devastating thing about achieving a failure: while in the midst of it, you think you're making progress. This company had disciplined schedules, milestones, employee evaluations, and a culture of execution. When schedules were missed, people were held accountable. Unfortunately, there was no corresponding discipline of evaluating the quality of the plan itself. As the company built infrastructure and added features, the team celebrated these accomplishments. Departments were built and were even metrics-driven. But there was no feedback loop to help the company find the right metrics to focus on.

These shadow beliefs have a common theme: a lack of reality checks. In my experience, great startups require humility, not in the personal sense, but in the organizational capacity to emphasize learning. A good learning feedback loop trumps even the best execution of a linear plan. And what happened with this ill-fated company? Although it failed, many of the smart people involved have accomplished great things. I know of at least five former employees that went on to become startup founders. They all got a tremendous first-hand lesson in achieving a failure, all on someone else's dime (well, millions of dimes). As we move into a new economic climate, it's my hope that our industry will stop this expensive kind of learning and start building lean startups instead.

The interesting thing about an analysis like this is that it seems obvious in retrospect. A lot of people say that they know that they don't know what customers want. And yet, if you go back and look at the key elements of the plan, many of them are in subtle conflict with the shadow beliefs. Understanding that tension requires a lot of reflection and a good teacher. I myself didn't understand it until I had the opportunity to view that failure through the lens of the customer development theory. I had a big advantage, because Steve Blank, the father of customer development, got to see the failure up close and personal too as an investor and board member. A much less painful way to learn the lesson is read his book: The Four Steps to the Epiphany.

Refactoring yourself out of business

Let me start out by saying I am a big fan of refactoring, the ongoing process of changing code so that it performs the same behaviors but has more elegant structure. It's an essential discipline of good software development, especially in startups. Nonetheless, I want to talk about a dysfunction I've seen in several startups: they are literally refactoring themselves to death.

Here's a company I met with recently. They have a product that has a moderate number of customers. It's well done, looks professional, and the customers that use it like it a lot. Still, they're not really hitting it out of the park, because their product isn't growing new users as fast as they'd like, and they aren't making any money from the current users. They asked for my advice, and we went through a number of recommendations that readers of this blog will already be able to guess: adding revenue opportunities, engagement loop optimization, and some immediate split-testing to figure out what's working and what's not. Most of all, I encouraged them to start talking to their most passionate customers and running some big experiments based on that feedback.

I thought we were having a successful conversation. Towards the end, I asked when they'd be able to make these changes, so that we could meet again and have data to look at together. I was told they weren't sure, because all of their engineers were currently busy refactoring. You see, the code is a giant mess, has bugs, isn't expandable, and is generally hard to modify without introducing collateral damage. In other words, it is dreaded legacy code. The engineering team has decided it's reached a breaking point, and is taking several weeks to bring it up to modern standards, including unit tests, getting started with continuous integration, and a new MVC architecture. Doesn't that sound good?

I asked, "how much money does the company have left?" And it was this answer that really floored me. They only have enough money to last another three months.

I have no doubt that the changes the team is currently working on are good, technically sound, and will deliver the benefits they've claimed. Still, I think it is a very bad idea to take a large chunk of time (weeks or months) to focus exclusively on refactoring. The fact that this time is probably a third of the remaining life of the company (these projects inevitably slip) only makes this case more exaggerated.

The problem with this approach is that it effectively suspends the company's learning feedback loop for the entire duration of the refactoring. Even if the refactoring is successful, it means time invested in features that may prove irrelevant once the company starts learning again. Add to that the risk that the refactoring never completes (because it becomes a dreaded rewrite).

Nobody likes working with legacy code, but even the best engineers constantly add to the world's store of legacy code. Why don't they just learn to do it right the first time? Because, unless you are working in an extremely static environment, your product development team is learning and getting better all the time. This is especially true in startups; even modest improvements in our understanding of the customer lead to massive improvements in developer productivity, because we have a lot less waste of overproduction. On top of that, we have the normal productivity gains we get from: trying new approaches on our chosen platform to learn what works and what doesn't; investments in tools and learning how to use them; and the ever-increasing library of code we are able to reuse. That means that, looking back at code we wrote a year or two ago, even if we wrote it using all of our best practices from that time, we are likely to cringe. Everybody writes legacy code.

We're always going to have to live with legacy code. And yet it's always dangerous to engage in large refactoring projects. In my experience, the way to resolve this tension is to follow these Rules for Refactoring:

• Insist on incremental improvements. When sitting in the midst of a huge pile of legacy code, it's easy to despair of your ability to make it better. I think this is why we naturally assume we need giant clean-up projects, even though at some level we admit they rarely work. My most important lesson in refactoring is that small changes, if applied continuously and with discipline, actually add up to huge improvements. It's a version of the law of compounding interest. Compounding is not a process that most people find intuitive, and that's as true in engineering as it is in finance, so it requires a lot of encouragement in the early days to stay the course. Stick to some kind of absolute-cost rule, like "no one is allowed to spend more time on the refactoring for a given feature than the feature itself, but also no one is allowed to spend zero time refactoring." That means you'll often have to do a refactoring that's less thorough than you'd like. If you follow the suggestions below, you'll be back to that bit of code soon enough (if it's important).
  
  
• Only pay for benefits to customers. Once you start making lots of little refactorings, it can be tempting to do as many as possible, trying to accelerate the compounding with as much refactoring as you can. Resist the temptation. There's an infinite amount of improvement you can make to any piece of code, no matter how well written. And every day, your company is adding new code, that also could be refactored as soon as it's created. In order to make progress that is meaningful to your business, you need to focus on the most critical pieces of code. To figure out which parts those are, you should only ever do refactoring to a piece of code that you are trying to change anyway.
  
  For example, let's say you have a subsystem that is buggy and hard to change. So you want to refactor it. Ask yourself how customers will benefit from having that refactoring done. If the answer is they are complaining about bugs, then schedule time to fix the specific bugs that they are suffering from. Only allow yourself to do those refactoring that are in the areas of code that cause the bug you're fixing. If the problem is that code is hard to change, wait until the next new feature that trips over that clunky code. Refactor then, but resist the urge to do more. At first, these refactoring will have the effect of making everything you do a little slower. But don't just pad all your estimates with "extra refactoring time" and make them all longer. Pretty soon, all these little refactorings actually cause you to work faster, because you are cleaning up the most-touched areas of your code base. It's the 80/20 rule at work.
  
  
• Only make durable changes (under test coverage). There's no point refactoring code if it's just going to go back to the way it was before, or if it's going to break something else while you're doing it. The only way to make sure refactorings are actually making progress (as opposed to just making work) is to ensure they are durable. What I mean is that they are somehow protected from inadvertent damage in the future.
  
  The most common form of protection is good unit-test coverage with continuous integration, becaus that makes it almost impossible for someone to undo your good work without knowing about it right away. But there are other ways that are equally important. For example, if you're cleaning up an issue that only shows up in your production deployment, make sure you have sufficient alerting/monitoring so that it would trigger an immediate alarm if your fix became undone. Similarly, if you have members of your team that are not on the same page as you about what the right way to structure a certain module is, it's pointless to just unilaterally "fix" it if they are going to "fix" it right back. Perhaps you need to hash out your differences and get some team-wide guidelines in place first?
  
  
• Share what you learn. As you refactor, you get smarter. If that's not a team-wide phenomenon, then it's still a form of waste, because everyone has to learn every lesson before it starts paying dividends. Instead of waiting for that to happen, make sure there is a mechanism for sharing refactoring lessons with the rest of the team. Often, a simple mailing list or wiki is good enough, but judge based on the results. If you see the same mistakes being made over again, intervene.
  
  
• Practice five whys. As always with process advice, I think it's essential that you do root cause analysis whenever it's not working. I won't recap the five-why's process here (you can read a previous post to find out more); the key idea is to refine all rules based on the actual problems you experience. Symptoms that deserve analysis include: refactorings that never complete, making incremental improvements but still feeling stuck in the mud, making the same mistakes over and over again, schedules slipping by increasing amounts, and, of course, month-long refactoring projects when you only have three months of cash burn left.

Back to my conversation with the company I met recently. Given their precarious situation, I really struggled with what advice to give. On the one hand, I think they have an urgent problem, and need to invest 100% of their energy into finding a business model (or another form of traction). On the other, they already have a team fully engaged on making their product architecture better. In my experience, it's very hard to be an effective advocate for "not refactoring" because you can come across as anti-quality or even anti-goodness. In any event, it's enormously disruptive to suddenly rearrange what people are working on, no matter how well-intentioned you are.

I did my best to help, offering some suggestions of ways they could incorporate a few of these ideas into their refactoring-in-progress. At a minimum, they could ensure their changes are durable, and they can always become a little more incremental. Most importantly, I encouraged the leaders of the company to bring awareness of the business challenges to the product development team. Necessity is the mother of invention, after all, and that gives me confidence that they will find answers in time.

Three freemium strategies

I'm excited to see so much attention being paid to freemium businesses lately. These are companies that generate revenue by offering a free product with an upsell or premium version. Their economics blends elements of the free, advertising-supported, "eyeballs" business with more traditional e-commerce and subscription businesses. For founders, I think it also has another big attraction: the ability to avoid a lot of "free vs paid" arguments. You can reach for all the scale of a free service and still make money. Pretty good deal, right?

Andrew Chen recently did a great article on the economic model underlying freemium (it includes a detailed spreadsheet, too!). In it, he mentions the core dilemma that all freemium companies face:

the key is to create the right mix of features to segment out the people who are willing to pay, but without alienating the users who make up your free audience. Do it right, and your conversion rates might be as high as 20%. Do it wrong, and your LTV gets very close to zero. This is why premium features have to be built into the core of a freemium business, rather than added in at the end. You want to be right at the balance between free and ‘mium!

Having worked in the freemium business for a number of years, I can safely report that struggling with these questions never goes away. I have sat in hundreds of meetings about what to give away and what to charge for, and they are not easy. Things get a lot worse when we don't have a coherent model of why we're using freemium in the first place. This is a common outcome I've observed among founding teams that take the "split the difference" route to freemium.

The way to resolve this tension is to agree on a fundamental freemium strategy, that leverages the free part of your business to add real value. In my experience, there are three basic choices:

1. Free serves paid. In this model, your free users trade their time for the benefit of your paying customers. For example, Puzzle Pirates does a great job of allowing trading between two different currencies: one that is earned with time, and another that is earned by paying. Both currencies are valuable, and free users can trade theirs to the paying customers, who are allowed to access benefits that would otherwise take a long time to achieve. Other examples are user-generated content sites, where free users create content that is valuable to paying customers. The key to this model is to make sure that customers who create value are heavily rewarded, and those that don't create value are marginalized, so that they have a strong incentive to pay.
   
   
2. Free trial. The original freemium model, where customers are given a certain amount of time before being forced to either quit the service or pay. The big question is when to pull the plug - too soon, and you risk customers not being sufficiently addicted to say yes; too late and you risk giving all the value away for free. Luckily, once you realize you are in this business, you can answer these questions empirically with good split-testing and linear optimization.
   
   
3. Free as inventory. Some businesses actually sell access to their free users. I think this is the right way to think about many advertising businesses, like Google AdWords. Some dating sites work this way too, where you can post your profile for free, and people pay only when they want to contact you. This ensures that the most popular people get lots of value without having to pay. Here the key is for your free customers to get value from the site that is greater than the costs they perceive by the fact that you're selling access to them. Google has shown their awareness of this issue by carefully managing the annoyance-factor of ads that are shown during search. What differentiates this model from "free serves paid" is that the free users don't need to consciously do anything special to be valuable. For network-effects businesses, like Skype, just being on the network is enough to create value.

Strategy is all about what you're not going to do; for a freemium business, it's about which users you're willing to turn away. Knowing which model you're in can make these decisions a little less excruciating. It's not that you shouldn't experiment - on the contrary, this is one of the most fundamental hypotheses you want to test early on. However, it's not very useful to mix and match features from different models. Better to set up complete experiments that are themselves internally coherent. Take a cohort of new customers and expose them to a completely different approach, and measure their behavior over time. Use that data to make an informed choice about whether you want to change strategies.

Lean hiring tips

In preparing for the strategy series panel this week, I have been doing some thinking about costs. Fundamentally, lean startups do more with less, because they systematically find and eliminate waste that slows down value creation. Sounds a little abstract, though, doesn't it? I want to talk specifics, and when you come right down to it, most technology startups don't have a very interesting cost structure. Almost all of it is in one giant fixed-cost bucket: salaries. And all of that cost was caused by one activity: hiring.

Hiring is no different from any other company process. I've written previously about how to conduct a good interview, how to build a process to assess fit, and, most importantly, how to do root cause analysis to continuously improve. But hiring does have some special challenges, and today I want to talk about the one that drives almost all of the cost (and most of the mistakes): deciding when and how to hire new people.

In the startups I've been involved with, there are few mistakes I regret more than hiring too many people too soon. It's one of the hardest things to see while it's happening, and it's one of the hardest mistakes to undo, because of the human and emotional costs of layoffs. Much of what I have found effective is counter-intuitive, because it requires investing more effort in each hire - all in the name of efficiency. This can be especially challenging for HR professionals who measure their success based on the number of jobs they fill. This is another sub-optimization caused by incentivizing the wrong metric.

We should begin by clearing up one common misconception: that hiring more people means you can create more value faster. This is a variant of the time-quality-money fallacy, and it is not necessarily true. There are two complicating factors. One, which is described in great detail in the Mythical Man-Month (in finding that link, I discovered that today is coincidentally the exact 11th anniversary of my first purchasing that book: Jan 19, 1998), is that as you add people to a team or project, there is an increase in communications overhead that makes everyone slightly less productive. Second, a lot of work that you do is actually not value-creating - it's waste. Hiring people to do more of that kind of work can actually slow down your whole company. In fact, unless you are adding capacity to a bottleneck in your development process, you probably should not be hiring at all. For more detail on this second point, I recommend The Goal: A Process of Ongoing Improvement.

In any event, in order to hire effectively, you have to have a strong understanding if what the unit of progress is for your company. For additional thoughts on that topic, see
Lean startups vs lean companies.

When to hire
Let's start with deciding when to add a new job to the payroll. Here's my suggestion: don't hire for any job unless you have tried, and failed, to do it yourself. There are lots of bottlenecks in any company, but only a few of them actually are on the critical path of a startup's success. For example, let's say you don't have an operations team, and so you're still getting paged late at night when servers crash. Is it time to make your first operations hire? That depends. If you're coming in to work groggy and therefore are not spending time with customers, I'm willing to bet it is. But if it's just an inconvenience, then you should focus on figuring out how to mitigate the pain rather than hire to alleviate it.

The main drawback of this approach is that most people in most startups are already too busy. As a mentor of mine always says, "Startups don't starve, they drown." If you're already too busy, how can you afford to take on yet another job? Wouldn't it be faster to just hire a new person and delegate to them? Not in my experience. If you hire a smart, competent person, they will inevitably find ways to be productive and keep busy. But you still won't know if the work they are doing is on the critical path. They may be dong good work, but you must be able to evaluate its contribution to the company's overall success to know if you've made a hiring mistake or not. If you've done the job yourself, you'll know that it's important and you'll know what success looks like. Oftentimes, in my experience, you'll realize that you don't need to hire for it at all.

Insisting that someone inside the company take on every new job has other benefits, too. It's a way of making necessity the mother of invention, and taking advantage of your limited resources to help the company decide not to do things it doesn't need to do. This tool is often hidden in plain sight. I didn't truly appreciate its power until I saw how larger startups can lose it if they're not careful. Having too few people forces companies to focus on working smarter and eliminating unnecessary projects.

Hiring experts
In many situations, the goal of hiring is to bring in expertise that the team doesn't already have. This might seem like another time where delegating makes sense, but again I recommend trying to have an existing person take on the job. In fact, I recommend putting one of your best people on the new job, even though that's especially painful. In my experience, having a generalist try to emulate an expert's job will result in three possible outcomes:

1. The person will fail. This may seem wasteful, but it's actually very valuable. Having a star performer attempt it will give the expert you eventually hire built-in credibility. You'll also have someone on staff who understands the problem, and can help the newcomer get integrated into the team faster.
   
   
2. The person will succeed, and it will matter. In this case, you now have an expert on staff already. This gives you the option to still hire an true expert, or to back-fill the person who has taken on the new job. In fact, in many situations, you'll actually find that the job is not as big as you thought, and you don't need to hire anybody. Or the person you get to do it will know how to invest in infrastructure that makes one of their many jobs automatic - this is one of the advantages of hiring smart people and building a culture of continuous improvement.
   
   
3. The person will succeed, and it won't matter. This is surprisingly common, and is the greatest savings of all. In many situations, people invest experts with wished-for magic powers, thinking they will "solve all our problems" even when that's not realistic. By having an in-house person succeed, you can evaluate whether having an expert on staff is really shortening the critical path of your organization's success. If not, you can avoid making the hire with confidence.

Returning to our example of the beleaguered founder who still has the pager, before hiring an operations guy, try promoting someone from within to take on the job. If you find yourself now spending time with customers, learning and guiding the learning of your whole company, you were probably right that the pager was getting in the way. But, if you find yourself getting caught up in some other distraction, and not really adding value, then the pager was a red herring. You need to get your priorities straight and, until you do, you probably may as well carry the pager too.


Relentlessly pursue passive candidates
Once they've decided to make a hire, startups should specialize in hiring candidates who are not actively looking for a job. This requires dramatically more work than hiring active candidates, but it usually generates positive returns.

The best candidates are those you (or someone on your team) have worked with before. Hiring them has substantial benefits over strangers: lower risk (because you know their strengths and weaknesses ahead of time), faster team integration (because they know someone already and that person can mentor them), and lower communication overhead (because they are more likely to speak and undersatnd a common language).

Most companies I work with think they are already pursuing these already-known passive candidates. I disagree. I am willing to bet that, right now, you can think of at least one person who is not working at your company who is an absolute star at what they do. Even if you have a referral bonus program, even if you've talked to that person about your company, and even if you've sent a recruiter to call her, I bet you could do more. Ask yourself this question: have you really made that person an offer they can't refuse? No Godfather-type violence is required. Telling someone "I really believe you can change the course of my company's life, and I'm willing to do whatever it takes to get you on board" is surprisingly powerful. Every person who you could make that offer to but haven't is a lost opportunity. Multiply that across everyone in your organization, and every superstar they know, and that's a huge potential pool of talent.

Even passive candidates that you don't already know have advantages that make them worth pursuing. You learn much more from trying to convince them to join, because they are more likely to ask you tough questions. You want people to turn you down if your pitch is bad, so you have more opportunities to learn how to get better (and is it possible that the problem isn't just with your pitch?). Given that most recruiting channels are pay-for-performance, it's usually cheaper, too, because it has a lower conversion rate. If you want to expand your pool of available passive candidates, give a site like NotchUp a try. They maintain a huge database of passive candidates, by offering to pay them when they interview. Again, by requiring you to invest more in each candidate you see, you learn to be much more careful about talking to the right people, and more focused on closing those candidates. (They also have a pay-for-performance model that's much cheaper than your typical contingency recruiter.)



Hiring well requires a lot of introspection in order to do it well. You have to really understand your company and what it needs. You need to be able to communicate the value your company creates to others. And you need to be able to ask yourself the tough questions to continue to refine your hiring process as you scale. But in lean times like these, I don't see any real alternative.