Tuesday, December 30, 2008

Assessing fit with the Wisdom of Crowds

Cover of When I wrote earlier about how to conduct a good technical interview, I had only a few things to say about how to assess if the candidate fits in with the team, including this:
This responsibility falls squarely to the hiring manager. You need to have a point of view about how to put together a coherent team, and how a potential candidate fits into that plan. Does the candidate have enough of a common language with the existing team (and with you) that you'll be able to learn from each other? Do they have a background that provides some novel approaches? Does their personality bring something new?
A few commenters have taken issue with the idea that it's solely the hiring manager's responsibility to assess fit, arguing correctly that the whole team should participate in the evaluation and decision. I completely agree. Still, I do think fit is a quality that requires special treatment, because it is the hardest attribute to evaluate.

Unlike the other attributes we look for in an interview candidate (like drive, brains or empathy) fit is not an individual quality. It's caught up in group dynamics. Worse, it has a self-referential quality to it. The very team that is making the assessment is being asked to assess itself at the same time as the candidate. How else can they tell whether the new team that will be created by the addition of this person will be superior to the team as it is presently constituted?

I have found James Surowiecki's book The Wisdom of Crowds particularly helpful in thinking through these issues. This book is Tipping Point-esque, full of anecdotes and interesting social science citations. Its central thesis is that, under the right circumstances, groups of people can be smarter than even their smartest single member. I find his observations compelling, and I feel good recommending the book to you, even though I know there are many among us who find "argument-by-anecdote" irritating. You don't have to buy the argument, but the facts and citations are worth the price of admission.

Let me briefly summarize the part of the book I find most helpful (leaving a few out, which aren't germane to today's topic). Not all crowds are wise. In order to get optimal results from a group-based effort, you need three things: diversity, Independence, and an objective method for aggregating results. For example, we conduct elections with a secret ballot, which ensures Independence (since nobody can directly influence your vote); we let everyone vote, which ensures diversity (since even extreme opinions can be heard); and we use a numerical count of the results (which, recent experiences notwithstanding, is supposed to mean that everyone's vote counts equally according to an objective formula). Similar mechanisms are at work in the stock market, Google PageRank, and guessing an ox's weight at the state fair.

Remove any of these essential ingredients, and you can find examples of groups gone bad: the Bay of Pigs invasion fiasco, market bubbles, and pretty much every one of Dilbert's team meetings.

Anyway, back to fit. Surowiecki has helped me in two ways:
  1. Assessing fit in the context of what makes a good team. In order to improve the performance of a team, it's not enough just to keep adding smart people. You actually need to find a diverse set of people whose strengths and weaknesses complement each other. The problem most teams have with fit, in my experience, is they confuse it with liking. Many great engineers I've worked with (especially the type I talked about in the hacker's lament) seem to think that if they get along well with someone in an interview, they'll be a good addition to the team. This kind of homogeneity can lead to groupthink.

  2. Putting together a process for helping a team assess fit. Another dangerous group dynamic in fit questions is that people are very sensitive to the opinions of their peers when talking about the group itself. In order to make an informed decision, the team needs a process of gathering and combining their opinions without having anyone's voice stifled.
For example, let's say you are on a team that prides itself in its all-hands-on-deck-at-all-hours style of working. There are a lot of great teams that work this way, preferring a series of sprints and lulls to a steady pace. Now let's say you're interviewing someone who doesn't seem to like to work that way. They work steady hours, but not for lack of drive; their references say they have exceptional output. Now picture the group meeting to talk about this prospective candidate. First up, the alpha-hacker on the team says something like "we should pass, this candidate is lazy."

Now it's your turn. Even if you are convinced that this candidate would make a great addition to the team, how much courage does it take to say so? You might convince the group, but you might not. And if you don't, will it raise suspicion about how dedicated you are? Will you be implicitly calling into question the validity of the team's values? Will they then be watching you for signs of laziness in the future? What about that vacation you've been planning to take... and so on. In my experience, there are plenty of situations where dissenting voices simply opt out. There's a clear danger to speaking up, but a pretty murky benefit.

If you find this line of reasoning confusing (nobody on my team feels that way) or paranoid (let's just all be rational), you may be surprised what the other people in the room are thinking. Take a look at some of the social science research in this area, like the Asch conformity experiments. In those, a group of people are asked to answer a simple question about their observations, one at a time. The first few people are actually actors, and they all give the same patently false answer. The experiment measures the likelihood that the last person in the sequence, the real experimental subject, will conform to what the previous people have said, or dissent. Even though the answer is obvious, and the other people in the room are all strangers, a surprising number of people choose to conform. I have found the pressure is much higher in situations where the answer is unclear, and the other group members are coworkers.

Combating these tendencies is the real job of the hiring manager. If that's you (or you are on a team whose hiring manager abdicates that responsibility), here are three suggestions that have worked for me to take advantage of the wisdom of crowds in hiring. Each of these are based in changes I've made to my hiring process in response to five whys analysis of previous hiring mistakes.
  • Before you meet the candidate, spend time thinking about the strengths and weaknesses of your team. Try to brainstorm some archetypes that probably would interview badly but actually be successful in filling out your team. Having been through this exercise in advance can help you listen carefully to what team members say about the candidate, and see if their objections are simply fearing what is different, or if they have a more serious concern.

    One experience I had was with a candidate who was absolutely convinced that our development methodology wouldn't work. He spent an inordinate amount of time grilling us on exactly how we work and why, asking smart but tough questions. He made us nervous, but we took a risk and hired him anyway. After we hired him, he spent weeks driving the team crazy with his critical (but, we had to admit, accurate) eye. Then, all of the sudden, a remarkable thing happened. Another new hire started to complain about the way we worked. Our former critic promptly set him straight, shooting down his complaints with the same ruthless efficiency he had previously devoted to analyzing our work. He had been converted, and from that point on acted as "defender of the faith" far better than I ever could.
  • Maintain strict Independence for each interviewer. Our rule was always that each interviewer was not allowed to talk any other interviewer once their session was concluded. The first time we'd exchange any words at all was during the end of the day assessment meeting. This prevents a previous interview from biasing a later interview. For example, I've seen situations where even a positive comment, like "wow, that candidate is smart!" cause disaster. The next interviewer, armed with an expectation of brilliance, chooses harder questions or becomes disappointed by an "only above average" performance.

  • Aggregate results carefully. When sitting in a room talking about the candidate, I have had success with two precautions. First, we would always share our experiences in reverse-seniority order. That meant that the most junior person would be forced to speak first, without knowing anything about what his or her manager thought. As the hiring manager, I would speak last, and I'd do my best to avoid giving any indication in advance of what I thought.

    We'd also have a structured discussion in two parts: in the first part, each person talks only objectively about what happened in the interview, without giving opinions or passing judgment. Others are allowed to ask "clarifying questions" only - no leading questions or comments. Only in the second round does each person give their opinion about whether to hire the person or not. This helps get the facts on the table in an objective way. If our team had ever struggled to do it, I would have insisted on written comments shared anonymously. In other words: do whatever you have to do to get an objective discussion going.
If you've been the victim of groupthink in hiring, or have suggestions for ways to avoid it, please share. All of us are team members, family members, coworkers and leaders. Tell us what you've learned in those different contexts. Maybe it'll help someone else avoid the same mistakes.

Tuesday, December 16, 2008

Engagement loops: beyond viral

There's a great and growing corpus of writing about viral loops, the step-by-step optimizations you can use to encourage maximum growth of online products by having customers invite each other to join. Today, I was comparing notes with Ed Baker (one of the gurus of viral growth). We were trying to broaden the conversation beyond just viral customer acquisition. Many viral products have flamed out over the years, able to capture large numbers of users, but proving transient in their value because they failed to engage customers for the long-term. Our goal is to understand the metrics, mechanics, and levers of engagement.

Levers of engagement
Let's start with the levers of engagement. What can you do to your product and marketing message to increase engagement?
  1. Synthetic notifications. The most blunt instrument is to simply reach out and contact your customers on a regular basis. This is such an obvious tactic that a surprising number of companies overlook it. For example, IMVU runs frequent promotional campaigns that offer discounts, special events, and other goodies to its customers. From a strictly "promotional marketing" point of view, they probably run those campaigns more than is optimal (there's always fatigue that diminishes the ROI on promotions the more you use them). But there is a secondary benefit from these activities: to remind customers that IMVU exists, and encourage them to come back to the site. The true ROI of a synthetic notification has to balance ROI, customer fatigue, and the engagement effects of the campaign itself.

    When you live with your own product every day, it's easy to lose sight of just how busy your customers are, and just how many things they are juggling in their own lives. A lot of engagement problems are caused by the customer completely forgetting about the provider of the service. Direct notifications can help ameliorate that problem.

  2. Organic notifications. Facebook, LinkedIn, and other successful social networks have elevated this technique to a high art. They do everything in their power to encourage customers to take actions that have a side-effect of causing other customers to re-engage. For example, from an engagement standpoint, it's a pretty good thing to automatically notify a person's friends whenever they upload pictures. But it's exponentially more engaging to have each person tag their friends in each picture, because the notification is so much more interesting: "you've been tagged in a photo, click to find out which one!" Similarly, the mechanics of sending users notifications when new friends of theirs join the site is a great organic re-engagement tactic. From the point of view of the existing customer, it goes beyond reminding them that the site exists; it also provides social validation of their choice to become a member in the first place.

    As with synthetic notifications, organic notifications are subject to fatigue, if they are not used judiciously. On Facebook, "poking" seems to have fairly high fatigue, whereas "photos" has low (close to zero?) fatigue. Ed adds this account: "When I first joined Facebook, I used to poke my friends and get poked back for the first few weeks, but now I rarely, if ever, poke people. Photos, on the other hand, is probably the primary reason I go to Facebook every day. Because they are constantly new and changing, I doubt I will ever get tired of looking at my friends photos, and I will probably always get especially excited to see a new photo that I have been tagged in."

  3. Positioning (the battle for your mind). The ultimate form of engagement is when the company doesn't have to do anything explicit to make it happen. For example, World of Warcraft never needs to send you an email reminding you to log in. And they don't need to prompt you to tell your guild-mates about the new epic loot you just won. The underlying dynamics of the product, your guild, and the fun you anticipate takes care of those impulses. This is true, to a greater or lesser extent, for every product. After you've acquired a customer, why would they bother to come back to your service? What do they get out of it? What is going on in their head when that happens?

    I wrote about this challenge for iPhone developers, in an essay on retention competition: the battle over what icon the user will click when they go to the home screen. At that point, there's no opportunity for marketing or sales; the battle is already won or lost in the person's mind. It's analogous to walking down the aisle in a supermarket. Just because you're already a Tide customer, doesn't necessarily mean you'll always buy Tide again. However, if you've come to believe that Tide is simply the only detergent in the world that can solve your cleaning problems, you're pretty unlikely to even notice the other competitors sitting on the shelf. Great iPhone apps work the same way.

    Marketing has a discipline about how to create those effects in the minds of customers; it's called positioning. The best introduction to the topic is Positioning (I highly recommend it, it's a very entertaining classic). But you don't have to be a marketing expert to use this tactic; you just need to think clearly about the key use cases for your product. Who is using it? What were they doing right before? And what causes them to choose one product over another? For example, a common use case for teenagers is: "I just got home from school, I'm bored, and I want to kill some time." If your product and its messaging is all about passing time while having fun, you might be able to get to the point where that is an automatic association, and they stop seriously considering other alternatives. That's exactly what the world's best video games do.

Seeing the engagement loop
We're just starting to weave these techniques into a broad-based theory of engagement, that would complement the work that has been done to date on viral marketing and viral loops. Notice that all of these techniques are attempting to affect one of a handful of specific behaviors that have to happen for a product to have high engagement. Do these sound at all familiar?
  1. A customer decides to return to your product, as a result of either natural interest, or a notification (organic or synthetic).
  2. They decide to take some action, perhaps influenced by the way in which they came back.
  3. This action may have side effects, such as sending out notifications or changing content on a website.
  4. These side effects affect other customers, and some side effects are more effective than others.
  5. Some of those affected customers decide to return to your product...
This is essentially a version of the viral loop. Let's look at a specific example, and start to think through what the metrics might look like if we attempted to measure it:
  1. Customer gets a synthetic message saying: "upload some photos!" Some percentage of customers click through.
  2. Some percentage of those actually upload.
  3. Those customers get prompted to tag their friends in their photos. Some percentage of them do (A), and these result in a certain number of emails sent (B).
  4. Each friend that's tagged gets an email that lets them know they've been tagged. Some percentage of them click through. (C)
  5. Of those, some percentage are themselves convinced to upload and photos. (D)
Calculating the "engagement ratio"
If we combine the quantities A-D using the same kinds of formulas we use for viral loop optimization, and the result is greater than one, we should see ever-increasing engagement notifications being sent. This will lead to some reactivation of dormant customers as well as some fatigue, as existing customers get many notification. Our theory is that the key to long-term retention is creating an engagement loop where the reactivation rate exceeds the rate of fatigue. This will yield a true "engagement ratio" that is akin to the viral ratio.

This makes intuitive sense, since the key to minimizing fatigue is to keep things new, exciting, and relevant. For example, user-generated content that includes of friends, especially if it includes you ("Joe tagged you in a photo. Click here to find out which one!") is usually going to be newer, more exciting, and more relevant than synthetic notifications ("Did you know you can know upload multiple photos at a time with our new photo uploader?"), or even than more generic organic notifications ("You've been poked by Joe."). High "engagement growth" with low fatigue is how you get the stickiness of a product to near 100%. You can try to churn out, but your friends keep pulling you back in. That's an engagement loop at work.

Seeing the whole
Engagement loops are a powerful concept all by themselves, and they can help you to make improvements to your product or service in order to optimize the drivers of growth for your business. But I think the value in this framework is that it can help make overall business decisions that require thinking about the whole rather than just one of the parts.

For example, let's say you have a viral ratio of 1.4. Your site is growing like wildfire, but your engagement isn't too good. You decide to do some research into why customers don't stay involved. When asked to describe your product, customers say something like "Product X is a place to connect with my friends online." Turns out, when optimizing your viral loop, this was the winning overall marketing message. It's stamped on your emails, landing pages, UI elements - everywhere. Removing a single instance of that message would make your viral ratio go down, and you know that for a fact, because you've split-tested every single possible variation.

As you talk to customers, you notice the following dilemma. Customers have a lot of options of places to connect with their friends online. And, compared to market leaders like Facebook and Myspace, you discover that your product isn't really that much better. Consequently, you are losing the positioning battle for your customers when they get home from school and ask themselves, "how can I connect with my friends right now?" Worse, your product isn't really about connecting with friends; that's just the messaging that worked best for the viral loop, where customers aren't that familiar your product anyway.

To win the positioning battle, you could try and make your product better than the competition, or find a different positioning that allows you to be the best at something else. Let's assume for the sake of argument that your competitors offerings are "good enough" and that you cant' figure out how to beat them at their own game. So you decide to try to reposition around a different value proposition, one that more closely matches what your product is best at. You could try and drive home that positioning with an expensive PR campaign, superbowl ads, and whatnot. But you don't have to - you have a perfectly good viral loop that is slowly but surely exposing the entire world to your positioning messages.

Here's what this long example is all about. When you go to change your messaging, imagine that your viral ration drops from 1.4 to 1.2. Disaster, right? Not necessarily. Since your viral ratio is still above one, it's still getting your message out, albeit a little slower. But if your new positioning message improves your engagement loop by more than the cost to your viral loop, you have a net win on your hands. Without measuring your engagement loop, can your business actually make tradeoff decisions like this one?

Connecting engagement and viral loops
The two loops are intimately connected, in a figure-eight pattern. Customers exit the viral loop and become part of the engagement loop. As your engagement improves, it becomes easier and easier to get customers to reenter the viral loop process and bring even more friends in. And as in all dynamic systems, there's no way to optimize a sub-part without sub-optimizing the whole. If you're focused on viral loops without measuring the effect of your changes on other parts of your business (of which engagement is just one), you're at risk of missing the truly big opportunities.

Hopefully, this theory will prompt some interesting responses. We'd love to hear your feedback and hear your stories. Have you struggled with engagement and retention? What's worked (and not worked) for you? Share your stories, and we'll incorporate them as we continue to flesh out this theory. Thanks for being part of the conversation.

Monday, December 8, 2008

Continuous integration step-by-step

Let's start with the basics: Martin Fowler's original article lays out the mechanics of how to set up a CI server and the essential rules to follow while doing it. In this post I want to talk about the nuts and bolts of how to integrate continuous integration into your team, and how to use it to create two important feedback loops.

First, a word about why continuous integration is so important. Integration risk is the term I use to describe the costs of having code sitting on some, but not all, developers' machines. It happens whenever you're writing code on your own machine, or you have a team working on a branch. It also happens whenever you have code that is checked-in, but not yet deployed anywhere. The reason it's a risk is that, until you integrate, you don't know if the code is going to work. Maybe two different developers made changes to the same underlying subsystem, but in incompatible ways. Maybe operations has changed the OS configuration in production in a way that is incompatible with some developer's change.

In many traditional software organizations, branches can be extremely long-lived, and integrations can take weeks or months. Here's how Fowler describes it:
I vividly remember one of my first sightings of a large software project. I was taking a summer internship at a large English electronics company. My manager, part of the QA group, gave me a tour of a site and we entered a huge depressing warehouse stacked full with cubes. I was told that this project had been in development for a couple of years and was currently integrating, and had been integrating for several months. My guide told me that nobody really knew how long it would take to finish integrating.
For those of you with some background in lean manufacturing, you may notice that integration risk sounds a lot like work-in-progress inventory. I think they are the same thing. Whenever you have code that is un-deployed or un-integrated, it's helpful to think of it as a huge stack of not-yet-installed parts in a widget factory. The more code, the bigger the pile. Continuous integration is a technique for reducing those piles of code.

Step 1: get a continuous integration server.
If you've never practiced CI before, let me describe what it looks like briefly. Whenever you check-in code to your source control repository, an automated server notices, and kicks off a complete "build and test" cycle. It runs all the automated tests you've written, and keeps track of the results. Generally, if all tests pass, it's happy (a green build) and if any tests fail, it will notify you by email. Most CI servers also maintain a waterfall display that shows a timeline of every past build. (To see what this looks like, take a look at the CI server BuildBot's own waterfall).

Continuous integration works to reduce integration risk by encouraging all developers to check in early and often. Ideally, they'll do it ever day or even multiple times per day. That's the first key feedback loop of continuous integration: each developer gets rapid feedback about the quality of their code. As they introduce more bugs, they have slower integrations, which signals to them (and others) that they need help. As they get better, they can go faster. In order for that to work, the CI process has to be seamless, fast, and reliable. As with many lean startup practices, it's getting started that's the hard part.

Step 2: start with just one test
You may already have some unit or acceptance tests that get run occaisionally. Don't use those, at least not right away. The reason is that if your tests are only being run by some people or in some situations, they probably are not very reliable. Startng with crappy tests will undermine the team's confidence in CI right from the start. Instead, I recommend you set up a CI server like BuildBot, and then have it run just a single test. Pick something extremely simple, that you are convinced could never fail (unless there's a real problem). As you gain confidence, you can start to add in additional tests, and eventually make it part of your team-wide TDD practice.

Step 3: integrate with your source control system
Most of the times I've tried to introduce TDD, I've run into this problem: some people write and run tests religiously, while others tend to ignore them. That means that when a test fails, it's one of the testing evangelists who inevitably winds up investigating and fixing it - even if the problem was caused by a testing skeptic. That's counter-productive: the whole point of CI is to give each developer rapid feedback about the quality of their own work.

So, to solve that problem, add a commit hook to your source control system, with this simple rule: nobody can check in code while the build is red. This forces everyone to learn to pay attention to the waterfall display, and makes a failed test automatically a big deal for the whole team. At first, it can be frustrating, especially if there are any intermittent or unreliable tests in the system. But you already started with just one test, right?

The astute among you may have noticed that, since you can't check in when the build is red, you can't actually fix a failing test. There are two ways to modify the commit hook to solve that problem. The first, which we adopted at IMVU, was to allow any developer to add a structured phrase to their check-in comment that would override the commit hook (we used the very creative "fixing buildbot"). Because commits are mailed out to the whole team, anyone who was using this for nefarious purposes would be embarrassed. The alternative is to insist that the build be fixed on the CI server itself. In that case, you'd allow only the CI account to check in during a red build.

Either way, attaching consequences to the status of the build makes it easier to get everyone on the team to adopt it at once. Naturally, you should not just impose this rule from on high; you have to get the team to buy-in to trying it. Once it's in place, it provides an important natural feedback loop, slowing the team down when there are problems caused by integration risk. This provides the space necessary to get to the root cause of the problem. It becomes literally impossible for someone to ignore the failures and just keep on working as normal.

As you get more comfortable with continuous integration, you can take on more advanced tactics. For example, when tests fail, I encourage you to get into the habit of running a five whys root-cause analysis to take corrective action. And as the team grows, the clear-cut "no check-ins allowed" rule becomes too heavy-handed. At IMVU, we eventually built out a system that preserved the speed feedback, but had finer-grained effects on each person's productivity. Still, my experience working with startups has been that too much time spent talking about advanced topics can lead to inaction. So don't sweat the details - jump in and start experimenting.

Reblog this post [with Zemanta]

Sunday, December 7, 2008

The hacker's lament

One of the thrilling parts of working and writing in Silicon Valley is the incredible variety of people I've had the chance to meet. Sometimes, I meet someone that I feel a visceral connection with, because they are struggling with challenges that I've experienced myself. In a few cases, they are clearly smart people in a bad situation, and I've written about their pain in The product manager's lament and The engineering manager's lament.

Today I want to talk about another archetype: the incredibly high-IQ hacker who's trying to be a leader. (As always, this is a fictionalized account; I'm blending several people I've known into a single composite. And please forgive the fact that I use male pronouns to describe the archetype. There is terrible gender bias in our profession, but that's a subject for another day. Suffice to say, most of the hackers I've known have been men. As a last disclaimer, please consult the definition of the word hacker if you're not familiar with the controversies surrounding that term.)

It's common to find a hacker at the heart of almost any successful technology company. I know them right away - we can talk high-level architecture all the way down to the bits-and-bytes of his system. When I want to know about some concurrency issues between services in his cluster, he doesn't blink an eye when I suggest we get the source code and take a look. And as soon as I point out an issue, he can instantly work out the consequences in his head, and invent solutions on the fly.

This kind of person is used to being the smartest person in the room. In fact, it's a rare person who can be subjected to recurring evidence of just how stupid the people around them are, and not become incredibly arrogant. Those who have the endurance are the ones that tend to lead teams and join startups, because you just can't be successful in a startup situation without empathy. I would characterize them as intolerant but not arrogant.

When a startup encounters difficult technical problems, this is the guy you want solving them. He's just as comfortable writing code as racking servers, debugging windows drivers, or devising new interview questions. As the company grows, he's the go-to person for almost everything technical, and so he's very much in demand. He throws off volumes of code, and it works. When scalability issues arise, for example, he's in the colo until 2am doing whatever it takes to fix them.

But life is not easy, either. As the company grows, the number of things he's called on to do is enormous, and the level of interruptions are getting intense. It's almost as if he's a country that was immune to the economic theory of comparative advantage. Since he's better at everything, he winds up doing everything - even the unimportant stuff. There's constant pressure for him to delegate, of course, but that doesn't necessarily work. If he delegates a task, and it gets messed up, he's the one that will get called in to deal with it. Better just to take care of it himself, and see that it's done right.

When you're the physical backstop putting dozens of fingers in the damn to prevent it from bursting, you might get a little irritated when people try to "help" you. The last thing you need is a manager telling you how to do your job. You're not very receptive to complaints that when you take on a task, it's unpredictable when you'll finish: "you try getting anything done on schedule when you're under constant interruptions!" Worst of all, your teammates are constantly wanting to have meetings. When they see a problem with the team's process, why don't they just fix it? When the architecture needs modifying - why do we need a meeting? Just change it. And we can't hire new engineers any faster, because you can't be interviewing and debugging and fixing all at the same time!

The picture I'm trying to paint is one of a bright individual contributor stretched to the breaking point. I've been there. Trust me, it's not a lot of fun. And I've also been on the receiving end; and that's not much fun either. Yet, quite often these dynamics play out with ever-increasing amplitude, until finally something drastic happens. Unfortunately, more often than not, it's the hacker who gets fired. What a waste.

What's wrong with this picture?

One of the most exhilarating things about a startup is that feeling of intense no-holds-barred execution. Especially in the early days, you're fighting for survival every day. Every day counts, every minute counts. Even if, in a previous life, you were a world expert in some functional specialty, like in-depth market research or scalable systems design, the compressed timeline of a startup makes it irrelevant. You get to figure things out from first principles all the time, experiment wildly, and invest heavily in what works. From the outside, it looks a lot like chaos. To a hacker, it looks a lot like heaven.

But even a tiny amount of success requires growth. Even with the highest standards imaginable, there's no way to hire just genius hackers. You need a diversity of skills and backgrounds. Suddenly, things slow down a little bit. To me, this is the critical moment, when startups either accept that "process = bureaucracy" or reject that thinking to realize that "process = discipline." And it's here that hackers fall down the most. We're just not naturally that good at thinking about systems of people; we're more comfortable with systems of computers.

If you've ever been abused by a bad manager in your career, it's easy to become traumatized. I think this is the origin of the idea among hackers that managers are idiots who just get in the way. The variations on this theme are legion: the pointy-haired boss, the ivory-tower architect, and of course the infinite variety of marketroids. But whenever groups of people assemble for a common purpose, they adopt process and create culture. If nobody is thinking about it, you're rolling the dice on how they turn out. And, at first, it's OK if the person who's doing that thinking is part-time, but eventually you're going to need to specialize. The alpha-hacker simply can't do everything.

Even in the areas that hackers specialize in, this go-it-alone attitude doesn't work. Building a good application architecture is not just coding. It's more like creating a space for other people to work in. A good architect should be judged, not by the beauty of the diagram, but by the quality of the work that the team does using it. The "just fix it" mentality is counter-productive here. Every bug or defect needs to go through the meta-analysis of what it means for the architecture. But that's impossible if you're constantly fire-fighting. You need to make time to do root cause analysis, to correct the systemic mistakes all of us tend to make.

And taking on too many projects at once is a classic sub-optimization. Sure, it seems efficient. But when there is a task half-done, it's actually slowing the team down. That's because nobody else can work on the task, but it's costly to hand it off. Imagine a team working from a forced-rank priority queue. Naturally, the best person should work on the #1 priority task, right? Not necessarily. If that person is subject to a lot of interruptions, as the people working on the less-important tasks finish, they're forced to keep working down the list. Meanwhile, the #1 task is still not done. It would have been faster for the team as a whole to have someone else work on the task, even if they were much slower. And of course there's the secondary benefit of the fact that as people work on tasks they don't know anything about, they learn and become more capable.

The reason this situation reaches a breaking-point is that it's constantly getting worse. As the team grows, the number of things that can go wrong grows with it. If a single person stays the bottleneck, they can't scale fast enough to handle all those interruptions - no matter how smart they are. And the interruptions themselves make looking for solutions increasingly difficult. Each time you look for solutions, you see a conundrum of this form: you can't hire because you're too busy, but you can't delegate because you can't hire.

All is not lost, though. When I get involved in companies that struggle with this problem, here is the kind of advice I think can help:
  • Introduce TDD and continuous integration. This is one of the bedrock practices of any lean startup, and so it's a common piece of advice I give out. However, it's particularly helpful in this situation. Without requiring a lot of meetings, it changes the perspective of the team (and its leadership) from fire-fighting to prevention. Every test is a small investment in preventing a specific class of bugs from recurring; once you've been successful at building this system, it's pretty easy to see the analogy to other kinds of preventative work you could do. It also helps ratchet down the pressure, since so many of the interruptions that plague the typical hacker are actually the same bugs recurring over and over. TDD plus continuous integration works as a natural feedback loop: if the team is working "too fast" to produce quality code reliably, tests fail, which requires the team to slow down and fix them.

  • Use pair programming and collective code ownership. These are two other Extreme Programming practices that are explicitly designed to counteract the problems inherent in this situation. Pair programming is the most radical, but also the most helpful. If your team isn't ready or able to adopt pair-programming across the board, try this technique instead: whenever anyone is becoming a bottleneck (like the proverbial hacker in this post), pass a rule that they are only allowed to pair program until they are not the bottleneck anymore. So each time someone comes to interrupt them, that person will be forced to pair in order to get their problem solved. In the short term, that may seem slower, but the benefits will quickly become obvious. It's another natural feedback loop: as the interruptions increase, so does the knowledge-transfer needed to prevent them.

  • Do five whys. This is a generalization of the previous two suggestions. It requires that we change our perspective, and instead treat every interruption as an opportunity to learn and invest in prevention.

  • Hire a CTO or VP Engineering. A really good technology executive can notice problems like the ones I'm talking about today and address them proactively. The trick is to hire a good one - I wrote a little about this in What does a startup CTO actually do? Sometimes, a great hacker has the potential to grow into the CTO of a company, and in those cases all you need is an outside mentor who can work with them to develop those skills. I've been privileged to have been the recipient of that kind of coaching, and to have done it a few times myself.
At the end of the day, the product development team of a startup (large or small) is a service organization. It exists to serve the needs of customers, and it does this by offering its capabilities to other functions in the company, and partnering with them. That's only possible if those interactions are constructive, which means having the time and space for people of different backgrounds and skills to come together for common purpose. That's the ultimate task for the company's technology leadership.

I strongly believe that all hackers have the innate ability to become great leaders. All that's required is a shift in perspective: at their root, all technology problems are human problems. So, fellow hackers, I'd love to hear from you. Does this sound familiar? Are you ready to try something different?

Getting started with split-testing

One of the startup founders I work with asked me a smart question recently, and I thought I'd share it. Unlike most of the people who've endured my one-line split-testing talk, this team has taken it to heart. They're getting started creating their first A/B tests, and asked "Should we split-test EVERYTHING?" In other words, how do you know what to split-test, and what just to ship as-is? After all, isn't it a form of waste to split-test something like SSL certs, that you know you have to do?

I love questions like this, because there is absolutely no right answer. Split-testing, like almost everything in a lean startup, requires judgment. It's an art, not a science. When I was just starting out with practices like split-testing, I too sought out hard and fast rules. (You can read about some of the pitfalls I ran into with split-testing in "When NOT to listen to your users; when NOT to rely on split-tests"). That said, I think it's important to get your split-testing initiative off to a good start, and that means being selective about what features you tackle with it.

The goal in the early days of split-testing is to produce unequivocal results. It's a form of waste to generate reports that people don't understand, and it impedes learning if there is a lot of disagreement about the facts. As you get better at it, you can start to apply it to pretty much everything.

In the meantime, here are three guidelines to get you started:
  1. Start simple. Teams that have been split-testing for a long time get pretty good at constructing tests for complex features, but this is a learned skill. In the short term, tackling something too complex is more likely to lead to a lot of wasted time arguing about the validity of the test. A good place to start is to try moving UI elements around. My favorite is to rearrange the steps of your registration process for new customers. That almost always has an effect, and is usually pretty easy to change.

  2. Make a firm prediction. Later, you'll want to use split-tests as exploratory probes, trying things you truly don't understand well. Don't start there. It's too easy to engage in after-the-fact rationalization when you don't go in with a strong opinion about what's going to happen. Split-testing is most powerful when it causes you to make your assumptions explicit, and then challenge them. So, before you launch the test, write down your belief about what's going to happen. Try to be specific; it's OK to be wrong. This can work for a change you are sure is going to have no effect, too, like changing the color of a button or some minor wording. Either way, make sure you can be wrong. If you're a founder or top-level executive, have the courage to be wrong in a very public way. You send the signal that learning always trumps opinion, even for you.

  3. Don't give up. If the first test shows that your feature has no effect, avoid these two common extreme reactions: abandoning the feature altogether, or abandoning split-testing forever. Set the expectation ahead of time that you will probably have to iterate the feature a few times before you know if it's any good. Use the data you collect from each test to affect the next iteration. If you don't get any effect after a few tries, then you can safely conclude that you're on the right track.

Most importantly, have fun with split-testing. Each experiment is like a little mystery, and if you can get into a mindset of open-mindedness about the answer, the answers will continually surprise and amaze you. Once you get the hang of it, I promise you'll learn a great deal.

Saturday, December 6, 2008

The four kinds of work, and how to get them done: part three

Those startups that manage to build a product people want have to deal with the consequences of that success. Having early customers means balancing the needs of your existing customers with the desire to find new ones. And still being a startup means continuing to innovate as well as keep the lights on. In short, it's hard, and easy to mess up - as I've certainly done more than my fair share.

In part one of this series, I talked about the four fundamental kinds of work companies do, and in part two I talked in some detail about the conflicts these differences inevitably create. Today, I want to talk about solutions. In short, how do you grow your team when it's time to grow your company?

For starters, there's whole volumes that need to be written about how to actually find and hire the people your startup needs. For example, you can read my previous post about how to do a a technical interview. But most startups succeed in hiring, one way or another, and are still left with the problems of organizing the people that rapid growth brings in.

To mitigate these problems, we need a process that recognizes the different kinds of work a company does, and creates teams to get them done. Here are my criteria for a good growth process: it should let the company set global priorities and express them as budgets, it should delegate trade-off decisions to a clearly-identified leader who is tasked with making trade-offs within a single kind of work, and it should allow personnel to circulate "backwards" through the kinds of work by facilitating clear hand-offs of products and features between teams.
  1. Build strong cross-functional teams for each kind of work. Let's start with the most important thing you can do to help product teams succeed: make them cross-functional. That means every team should have a representative from absolutely every department that the team will have to interact with. To begin with, see if you can get designers, programmers, and QA on the same team together. When you've mastered that, consider adding operations, customer service, marketing, product management, business development - the idea is that when the team needs to get approval or support from another department, they already have an "insider" who can make it happen.

    The advantages of cross-functional teams are well documented, and for a thorough treatment I recommend the theory in the second half of Agile Software Development with Scrum. I want to focus here on one particular strength: the ability to operate semi-autonomously. Whatever process you use for deciding what work is going to get done at a macro-level, once the team accepts its mission, it's free to get the job done by whatever means necessary. If the team has true representation from all parts of the company, it won't have the incredibly demoralizing experience of coming up with a solution and then having some outsider veto it. The flip side of that is that you can have strong accountability: all the usual political excuses are removed.

    I have been amazed at the levels of creativity these kinds of teams unlock.

  2. Create iteration cycles that provide clear deadlines and opportunities to revise budgets. Opinions vary about the optimal cycle length, but I do think it's important to pick a standard length. Scrum recommends 30 days; I have worked in one or two-week cycles up to about three months. At IMVU, we found 60 days was just about right. It allows each team to do multiple iterations themselves, before having to be accountable to the whole company for their work. You could easily manage that as two 30-day scrums, or four two-week sprints. Either way, create a clear cycle begin and end date, and use the cycle breaks for two things: to hold teams accountable and to reallocate personnel between teams.

    In order to prevent people from bunching up in the later stages of the work pipeline, those leaders need to be focused on automation and continuous improvement. Make sure you keep track of whether that's happening. Between cycles, put some pressure on those later teams to do more with fewer people. If they can keep their team sizes constant as the company grows, you'll be making it possible to have more people working on R&D and Strategy.

  3. Find a balance between long-term ownership and learning new things. As the team grows, it's tempting to want to keep people on the same teams for long duration. That's more efficient, in a way, because they get to know each other and the better they get to know the part of the product their team is working on. However, I think it's more efficient for the company overall if people are moving between teams and learning new skills. That keeps the work interesting, and prevents specialists from becoming company-wide bottlenecks.

    Some people will naturally move with the work they do, as projects pass between teams. Manager need to keep an eye out for opportunities to move people simply for the sake of helping them take on new challenges. Team leaders should rotate less frequently, because too much churn works against accountability. A good target is to try and circulate about 30% of a given team between cycles.

  4. Formalize and celebrate the hand-off between teams. When a project is done, there are three possible outcomes. Either that team is going to keep iterating it in the next cycle, or they are going to hand it off to another team, or they are going to euthanize it and remove it from the project. If you get serious about these hand-offs, you can prevent a common problem that startups encounter: lots of features that nobody has ownership of. If you celebrate these hand-offs, you reinforce the idea that different kinds of work are equally valuable, and you help people let go when it's time for transitions to happen. For example, the level of attention a project gets when it moves into maintenance mode is going to be less than it had before. Acknowledge that up-front, and celebrate the people who made it possible for that feature to run its course and help the company succeed.

  5. Prefer serializing work to doing it in parallel. When possible, try to have the whole team work on a one project at a time, rather than have many going simultaneously. It may seem more efficient to have lots of projects going (each with one or two people on them), but that's not been true in my experience. Lots of little projects erodes teamwork and builds up work-in-progress. I believe that having the whole team work from a common task list, in priority order, lets everyone feel that "all hands on deck" sense that the founders experienced in the early days of the company. You can still do multiple projects in a single cycle, just try and have as few "in flight" at any given time as possible. You'll also be able to put more of an emphasis on finishing and shipping, than if everyone has to rush to finish their projects all at the same time (stepping on each others' toes, naturally).
This process can seem daunting, especially if you're currently running in ad-hoc mode. That's OK; like all lean transformations, you need to undertake this one in small steps. Remember the mantra: incremental investment for incremental gain. Each process change you make should pay for itself in short order, so as to build credibility and momentum for further refinements.

Where should you start? I rcommend you try building a cross-functional team and see what happens. At first, choose only a few functions, keep the team small, and give them a modest goal. Try your hardest to give them space to operate without supervision, which means choosing a problem that doesn't have huge risks associated with it (or you'll be too scared to let them out of your sight). Keep the team size small, maybe just two or three people. If the goal is clear, and the team is willing to embrace it, you may be surprised just how fast they can execute.

One last thought. These ideas are easier to give lip service to than to actually implement. So if it doesn't work right away, don't give up. Maybe you think you've created a semi-autonomous team, but the team feels like it's just business as usual. Or you may have too high a level of background interruptions for them to really stay the course. Or they may decide just to slack off instead of accepting the mission. Try to see these as opportunities to learn: at the end of each cycle do a five whys post-mortem. I gurantee you'll learn something interesting.