Sunday, October 5, 2008

The product manager's lament

Life is not easy when you're working in an old-fashioned waterfall development process, no matter what role you play. But I have a special sympathy for the "product manager" in a startup that is bringing a new product to a new market, and doing their work in large batches. I met one recently that is working on a really innovative product, and the stories I heard from their development team made me want to cringe. The product manager was clearly struggling to get results from the rest of the team. These are smart people trying hard to all row in the same direction. So why are they having so much difficulty?

Let's start with what the product manager does. He's supposed to be the person who specifies what the product will do. He writes detailed specs which lay out exactly what features the team should build in its next iteration. These specs are handed to a designer, who builds layouts and mockups of all the salient points. Then the designs are handed to a team of programmers with various specialties. Each specialist takes up his part of the spec (UI, middleware, backend) and cranks out code. Last, the QA team builds a test plan based on the spec, and then tests the features to make sure they conform to the plan.

This system naturally lends itself to a pipeline approach, which the product manager organizes. While the programmers are off building the next major feature, he is busy writing specs so that, when they finish, there won't be any idle time before they can start the next iteration. If the programmers go idle, it's bad news for the product manager, because he's supposed to keep them busy building product. Otherwise, the company is wasting serious money.

When I met this team, some acrimony had built up. The last few features came out pretty different from what was origianlly spec'd, and took far too long, to boot. The programmers keep asking for more say in the designs and direction that they work on. So the team has been spending more and more time in the spec and design phases, trying to get team buy-in on what they are going to build. For some reason, though, despite the increased buy-in, the final product often doesn't look anything like the original spec. The VP Engineering spends all of his time trying to make sure the programmers understand and implement the spec. Each iteration takes longer than the previous one. Frustration is mounting.

It doesn't take long to discover that the product manager is being forced to write every spec five times. First, he writes it nice and clear. Next he works with the designer to build out the design spec, and with QA to build out the test plan. When the programmers get it, they often start negotiating with him about what's going to be built. They exchange countless emails, and he's being constantly interrupted and being asked to clarify exactly what the spec means. The fourth spec exists only in these emails, which are changing the design in an ad-hoc fashion. By the time QA gets the feature, their test plan is badly out of date. So the product manager winds up actually having to use the software, by hand, updating the spec and helping create a new test plan. Naturally, the deviations from the spec are so severe, that he has to rewrite the spec he's currently working on (for the next major feature) to take them into account.

Ironically, this system was designed to keep each functional group 100% utilized, so nobody goes idle, including the product manager. But as the iterations get longer, he's spending more and more of his time answering questions. The interruptions are so bad, he has to write his new specs at 3AM, just to keep the pipeline full.

What's wrong with this picture? Everyone is working at 110%, with full dedication. But the team is falling further and further behind.

Here are the changes I'm working with this team to implement
  • Work in cross-functional teams. Each team has a representative of each function. To start, we'll try a product manager, designer, programmer or two, and QA. The team owns a complete feature end-to-end.
  • Focus on speed of iteration rather than utilization of every function. Let people go idle, if they can't help the current iteration succeed. I'm contiuously amazed how many people have untapped creativity and resourcefulness. They don't want to be idle. By letting them focus on the success of their team exclusively, you empower them to do whatever it takes to make the team successful. Will that mean someone in design will jump in to help QA get the release certified? We'll see.
  • Switch the spec process from push to pull. Start with a one-page spec, no more. Then, let the team ask questions of the product manager whenever they need clarification. In exchange, the team agrees to show each piece of working code to the product manager for his approval. They'll find points of disagreement much faster, and resolve them in realtime. Plus, the team will get better and better at interpreting the concise specs that only have to be written once. (Eventually, they may abolish specs altogether)
There's much more this team can do to eliminate waste in the way that they work and thereby iterate faster. Eventually, I hope to get them on a full agile diet, with TDD, scrums, sprints, pair programming, and more. But first I think we need to save the product manager from that special form of torture only a waterfall product development team can create. Once the different parts of the team can trust each other again, we'll have the basis we need to start a true continuous improvement feedback loop.

8 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Nice write-up. Indeed the first thing I thought when I read the first paragraphs that the product manager was operating in a white tower and that the developers felt disenfranchised. It may be scary to involve that many "unqualified" people that early on, but the pay-off seems greater.

    ReplyDelete
  3. I went through some of this in bringing agile methodologies to my current firm - except that I am the product manager in question at this case.

    Getting people into quicker iterations and more productive communication patterns has greatly accelerated our product output cycle, and we should see the fruits of those labors pay off in a major way at the end of this month.

    ReplyDelete
  4. @r& - let us know how it turns out and what you learned. Sounds like it'll be a very educational story.

    ReplyDelete
  5. Having someone from each functional team working with the product manager as stories are churned out is, I think, the only real way to get around this. We've spent way too much time doing the real-time back-and-forth approach and it only serves to produce frustration and confusion. Doing stories right, though, is tough.

    For what it's worth, we came up with our own solution in an attempt to deal with the issue of "no one knows what the spec really is." The jury is still out on whether it'll work long-term or not. I've written about it on our development blog at http://guidewiredevelopment.wordpress.com/2008/08/22/the-birth-of-the-requirements-wiki/

    ReplyDelete
  6. Great to read posts about introducing lean approaches into more teams. I've played the role of that product manager before and it isn't fun.

    I'm currently introducing lean to my new team. Our current deliverables are reliability, availability, and scalability so the product management lament is not so much of an issue.

    We are starting by reducing wasted work and decreasing time from deciding to make a change to shipping that change. My hope is if the team can make improvements in these areas, we'll be ready to make improvements in the product management process later.

    ReplyDelete
  7. In "Developing Products in Half the Time" Smith and Reinertsen make the point that you can't have fast cycle time without having some experts idle to handle issues. To keep an expert busy a queue has to build up at each expert and in those queues delays accumulate for the sake of high utilization at the expense of fast throughput

    ReplyDelete
  8. Interesting read. There are a few things I would make specific mention of: first, QA really needs to work closely with the design folks and development needs to be in the same room as well when the QA plan is being developed. This really needs to be an iterative approach to be certain that nothing is being missed. Secondly, and perhaps most importantly, in the initial kickoff meeting, expectations need to be spelled out in a communication plan - where was this plan? If the team is continually falling behind, it would seem that these teams were no fully aware of their expectations to begin with. Oh.. one more thought, where were the code reviews?

    ReplyDelete