tag:blogger.com,1999:blog-7533727264507128560.post7365628478618560081..comments2024-03-22T04:49:13.363-07:00Comments on Lessons Learned: Embrace technical debtErichttp://www.blogger.com/profile/12249063135381216090noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-7533727264507128560.post-60320453709440148122010-02-06T20:34:20.814-08:002010-02-06T20:34:20.814-08:00Eric, great post. I'd like to understand how e...Eric, great post. I'd like to understand how embracing technical debt works for you with a cross-functional problem team and prototyping. <br /><br />Are the engineers in the customer development team allowed to push quick and dirty "prototypes" to production?.<br /><br />It would certainly make sense to validate an internal feature proposal with split-testing, but I'm worried about the safety measures to avoid major customer complaints and loosing customers: <br />* the continous deplayment inmune system would be able (by itself) to prevent it? <br />* should the test subjects be part of the customer advisory board?<br />* what would be the profile of these engineers? I t makes sense to me to put seasoned programmers here, specially if they had made the shift to "marketing" after frustration for the lack of customer development.<br /><br />Wow, a lot of questions, I just hope is not to late to add a comment to this post.Erich von Hauskehttps://www.blogger.com/profile/14694806782753303959noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-22120358884997361212009-07-31T06:08:06.483-07:002009-07-31T06:08:06.483-07:00Darn good - I have struggled in relevant conversat...Darn good - I have struggled in relevant conversations with our in house move to scrum/agile. Things like "where is the design time?" and "isn't planning on refactoring just waste?". It all seems so clear to me. But this blog put into better perspective. Good job.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-40537055669018042782009-07-30T13:29:41.553-07:002009-07-30T13:29:41.553-07:00> Low technical debt now doesn't mean that ...> Low technical debt now doesn't mean that you <br />> don't improve the design later as the feature <br />> gets bigger. It just means that this tiny or <br />> provisional thing is shipshape given that it is a <br />> tiny or provisional thing.<br /><br />Let me push back a little more. It's not that I disagree that product and technical design are separate, it's just that I think we under-appreciate the level of interaction, especially under conditions of uncertainty such as those that startups face.<br /><br />Let me take a concrete example. Remember IMVU's initial IM add-on product? It had a pretty good technical design. Here why:<br /><br />- it kept each IM network in its own separate module, and made it really easy to add new IM networks by composing a set of common objects<br />- it separated the underlying transport from the IM "session" itself, so it was robust in the face of the underlying client acting strangely, going away, or even having conversations switch clients altogether<br />- it compacted all of its information into brief, human-readable text messages that could be sent over any IM network in the clear<br /><br />Those were strictly technical design decisions, and I think they were really good. Unfortunately, when we realized the product design was not what customers wanted, we had to pivot to a new product. But we had to bring that old codebase with us. Now the assumptions and abstractions that had served us well started to serve us badly. When we became a standalone network, it didn't matter how easy it was to add new networks, since we never did. And having the session abstracted from the transport made debugging much harder. Worse of all, the plaintext codes we were used to sending were considered non-authoritative, since they could be pulled off a third-party network. This made the actual transport much more difficult on our first-party network than was really necessary.<br /><br />As a result, we have had to be constantly refactoring this design, a little bit at a time, to smooth out these rough edges. These design changes feel a lot like the interest payments incurred by technical debt. My argument is that there is no distinction to be had. That "good design" turned out to be technical debt, after all.<br /><br />What I object to most is the idea that technical design is a linear quantity. There's no such thing as "improving the technical design" in any absolute sense. You can only improve it with regard to whatever the purpose of the current product is. When that purpose is changing, we're necessarily chasing a moving target. <br /><br />I think the reasons our intuitions as engineers are so messed up here is that they were formed in an era where the rate of change was simply much lower. When we work on a product that takes many years to design and build, it natural to start seeing design decisions on a linear scale - because the product's purpose is clear and relatively stable throughout. Imagine the architect of a cathedral trying to make sense of this distinction. They'd think we were crazy.<br /><br />So far, this is all an academic exercise in naming - but I think it has an important real-world effect. Once we recognize that time invested in good design can wind up as technical debt, we're can start to ask ourselves "what else should we invest that time in?" And, as I tried to outline in the article, process improvements that reduce overall batch size might be a better choice.Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-51699822869062584702009-07-30T01:09:31.925-07:002009-07-30T01:09:31.925-07:00Eric, thanks for the post.
couple of points I th...Eric, thanks for the post. <br /><br />couple of points I thought worth mentioning. <br /><br />First, I really find it hard to imagine a circa 2009 web start-up that doesn't start with a huge technical date. Absurdly, it works both for start-ups that are thinking structuraly about customer discovery (and hence want to experiment first, and design nicely second) and for start-ups that still follow the good old product dev. path hence urgently pushing to release asap. <br /><br />I think that if a web founder would come to me asking for an investment talking about how he's coding an immensly scalable, perfectly designed system, I will throw him out in an instant. Perhaps the best example of that was Cuil, who was actually built on the ability to scale faster and better and found out that that's not as important as figuring out what urgent problem do you solve to which customer.<br /><br />This btw, is not true to all aspects of software and all realms of life. For example, the rating system of the iPhone app store makes it extremely important for a new app front end to be very close to perfectf rom day one, even if the backend is held by a shoestring and a piece of gum.<br /><br />Finally, sometime embracing technical debt is happening in much larger orders of magnitude than we can imagine. For example, back in the early 90's two multi-billion dollar r&d projects were racing to build a missile defense system. The Israeli Arrow system, and the US THAAD. The Israeli strategists made a conscious decision of incurring a huge technical debt by building a system they knew would only answer a certain portion of the problem, and would need to be gradually evolved and replaced (i.e. pay 'interest'). The Americans, on the other hand, decided to build a full system, perfectly designed, with no short-cuts. The results were that the Israeli system became operational 10 years ago, while the THAAD project is starting to finally be deployed this year after 10 years of delay. On the other hand, now that it's deployed, it's a much more capabale system. <br /><br />This example shows that really the type of debt and its extent depend on what are your more strategic objectives. If you're an Israeli who need something ASAP because the threat of getting fired on with scuds is rather real, then taking on huge technical debts makes a lot of sense. If you're an American who can take all the time and most of the money in the world to get it just right, then you might as well do that :)<br /><br />Keep on the good work,<br /><br />EranEran Shirhttp://shir.posterous.comnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-76460067982737127452009-07-29T22:58:01.111-07:002009-07-29T22:58:01.111-07:00> Eric, could you please explain what you mean ...> Eric, could you please explain what you mean by "process" ?<br /><br />Sure. I don't have a formal definition, but what I mean is the set of practices that we adopt as a team to build our product. How do we make decisions about the platform? How do we prioritize new features? How do we debug them? How do we deploy them?<br /><br />Our process may include the use of tools and infrastructure, too. So when I say something like "invest in process" I intend that to mean that we can evolve our tools and our use of them, too. <br /><br />Does that help?Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-46865635148630007452009-07-29T14:07:29.278-07:002009-07-29T14:07:29.278-07:00Hi, Eric. I think the confusion arises because the...Hi, Eric. I think the confusion arises because the you include more things under "technical" than I do. And I think my notion of the split is roughly where Beck and Fowler would draw the line, too.<br /><br />I agree the two kinds of design are somewhat related, and that technical people will benefit from understanding the deeper purpose. But I think they're still mostly orthogonal. Let's consider cases:<br /><br />Can you have a piece of software with good product design and bad technical design? Yes. In that case, the user experience and system performance might be good, but the cost of future development would be high.<br /><br />Can you have a piece of software with bad product design and good technical design? Surely. Market fit or user experience might be awful, but engineers looking at the code base would find little to quibble about.<br /><br />For your example of the right amount of fault tolerance, I believe that's mainly a business decision. I encourage putting that on cards and treating them as part of the regular feature flow. How to implement a given level of reliability is the technical decision. For more examples, see the "-ilities" section here:<br /><br />http://xp123.com/xplor/xp0512/index.shtml<br /><br />Design is definitely dynamic, and definitely benefits from feedback, so I'm also in favor of shipping early and often. Small batches rule.<br /><br />However, I don't think technical quality has to get in the way of that. If one is truly doing incremental design, then you rarely need to do much technical design to get something quick and easy out the door. Low technical debt now doesn't mean that you don't improve the design later as the feature gets bigger. It just means that this tiny or provisional thing is shipshape given that it is a tiny or provisional thing.Williamhttps://www.blogger.com/profile/08301434653929004584noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-47935884824322668772009-07-29T11:23:47.572-07:002009-07-29T11:23:47.572-07:00Eric, could you please explain what you mean by &q...Eric, could you please explain what you mean by "process" ?Joseph Turianhttps://www.blogger.com/profile/06249878639857416906noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-64637761997226606502009-07-29T10:19:38.945-07:002009-07-29T10:19:38.945-07:00Excellent article.
I find technical debt a really...Excellent article.<br /><br />I find technical debt a really helpful metaphor when introducing new teams to the responsibility they and their product managers have for maintaining the quality of the product over time and ensuring the lasting value of their companies investment.<br /><br />It's good to see that the metaphor continues to hold up and you have demonstrated a number of insights and opportunities that present themselves when you incur debt wisely.<br /><br />There is a dark side!<br /><br />Often technical debt is incurred with due consideration but without intent or budget for pay back. I have seen too many systems being re-written of suffering from crippled productivity due to a management team that do not expect or understand the nature of technical debt.David Draperhttp://www.agiledesign.co.uknoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-69067544129712661712009-07-29T10:18:59.658-07:002009-07-29T10:18:59.658-07:00I would suggest (from my experience) that getting ...I would suggest (from my experience) that getting it right first time is actually cheaper and faster than making what you think are short cuts at the time. Poor testing or poorly separated concerns within your code base often come back to bite you a lot sooner than you think.<br /><br />If you're gonna play the game of "when is it OK to create technical debt" you firstly have to know that it's debt in the first place, but also have to have a very clear idea of the consequences. How many of us could say we're in a position to be able to do that? I don't think I am.<br /><br />As always, it's our man Ward Cunningham who came up with the phrase technical debt. Interestingly he never intended it be inferred that technical debt is something you could make the conscious decision to accrue:<br /><br />"I'm never in the favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of the problem even if that understanding is partial."<br />http://c2.com/cgi/wiki?WardExplainsDebtMetaphorRob Bowleyhttp://blog.robbowley.netnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-66025283168570691672009-07-29T09:50:07.528-07:002009-07-29T09:50:07.528-07:00Nice post~
I really like the idea of writing test...Nice post~<br /><br />I really like the idea of writing test coverage for legacy code before actually making the changes. Unfortunately some legacy projects make test coverage almost impossible. At least in strongly typed languages such as Java. Test Harnesses usually work with MVC architectur,e which is still a newer concept as far as big business goes.<br /><br />Either way I feel that it is a great idea if the Test to Implementation timeline is about 1 to 1.Ryan Higdonhttp://rhigdon.comnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-19929842182248768782009-07-29T09:40:28.941-07:002009-07-29T09:40:28.941-07:00> However, I get the impression you're mixi...> However, I get the impression you're mixing the <br />> notions of something that's well designed from <br />> the product perspective and something that's well <br />> designed from the engineering perspective. These <br />> kinds of design are mostly orthogonal. Technical <br />> debt only refers to the latter.<br /><br />Thanks for another thoughtful comment. I'll continue to push back on this distinction. The technical design of a product is intimately linked to its product design. For example, how fault tolerant should it be? A good technical design should have "just the right amount" of fault tolerance. How costly is a failure? How does the fault tolerance affect our ability to make changes in the future? How desirable is the ability to make changes in the future? The only way to to answer these questions is to ask product questions, yet they are essential for a good technical design.<br /><br />Even more tricky is the fact that there are interaction effects between the two designs. The faster we get the technical design done, the faster we start to get feedback about _both_ the technical and product designs. We have to look at design as a dynamic process. Sometimes choosing a worse design results in a better design. <br /><br />That's not a practice, that's a fact. Deciding how to deal with that fact is what we have to struggle with when we build teams.<br /><br />I don't think that "spikes" are sufficient, although they are very useful in situations where we know in advance we are going to throw away our experimental code. But most design problems occur midway through implementation right inside a larger codebase. What then? There's nothing to throw away, unless we revert - but having the ability to revert large amounts of work implies that we're working with a very large batch size.<br /><br />I think we're much better off focusing on shrinking the batch size of work. That gives us better options and faster feedback with regard to debt - and design.Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-16299483810529739082009-07-29T09:12:58.671-07:002009-07-29T09:12:58.671-07:00Hi, Eric. As usual, an interesting post.
Advocati...Hi, Eric. As usual, an interesting post.<br /><br />Advocating for use of technical debt is sort of like saying American consumers could make good use of more financial debt. This is true for some small number of very responsible people, but extremely dangerous advice for the majority, who already are in a lot of debt, have poor debt management skills, and haven't solved the personal and cultural issues that got them in so deep in the first place.<br /><br />That said, I agree with a lot of your more detailed points. It definitely can be useful to judiciously take on small amounts of debt. There is also some level of architectural exploration where there are so many unknowns that it makes sense to throw away all good technical practice and just hack. (The Extreme Programming folks call those "spikes", and generally don't check them in.) From a product perspective, it can also be very useful to try something quick out to see what the response is before investing more deeply.<br /><br />However, I get the impression you're mixing the notions of something that's well designed from the product perspective and something that's well designed from the engineering perspective. These kinds of design are mostly orthogonal. Technical debt only refers to the latter.<br /><br />The questions of whether something should ship on CDs, or what the user behavior will be, or how many users to build for? Those are product questions. Technical debt refers only to the technical decisions made to meet those product goals.<br /><br />I agree totally with you that a disciplined, experienced team with the right culture will eventually learn to make the right decisions in regard to this.<br /><br />However, until engineers have spent at least a year developing in a very low-debt code base, they don't have the right experience, and will tend to go down the familiar high-debt path. They quite literally cannot imagine what people like Fowler and Beck are talking about. It's like asking a 1930s coal miner to be good to the environment.<br /><br />And I want to close by mentioning, mainly as a caution to readers who aren't as familiar with your work, that the approach you describe can only work in the cultural context you promote. I have never seen (or even heard of) a normal team in your average large organization ever learn to manage technical debt dynamically. Poor feedback loops and command-and-control nonsense always dump them in the weeds. For them, "no technical debt" is a great rule to live by.Williamhttps://www.blogger.com/profile/08301434653929004584noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-52600294647861855812009-07-29T08:48:43.030-07:002009-07-29T08:48:43.030-07:00It is so true, but non-tech managers usually try t...It is so true, but non-tech managers usually try to ignore the facts away.Heinerhttps://www.blogger.com/profile/09520563824611073104noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-69555020594327631102009-07-29T06:57:56.122-07:002009-07-29T06:57:56.122-07:00Kevin, really appreciate the thoughtful comments.
...Kevin, really appreciate the thoughtful comments.<br /><br />I want to challenge the idea that you can have "well-designed, well-implemented code that doesn't help you meet your goals." My feeling is that good design has to encompass the goals of the artifact being designed. Unfortunately, in a lot of situations, the true goals aren't really clear until the design is complete or nearly-so. In that case, what you thought was good design can suddenly turn out to be really bad design. I think that's a kind of technical debt. <br /><br />That's still not to say that there aren't just regular old bad designs out there - but I want us to pay more attention to this particular case because it is so often overlooked. It's not that good design "inevitably" leads to debt, it's that spending time on good design doesn't necessarily prevent debt. In such cases, you're better off experimenting or investing in lean process.<br /><br />With regards to third-party API's, I stand by the idea that you should consider all the debt that affects your team, whether you control it or not. When you incorporate a third-party library that is debt-ridden, that counts. When a third-party hosts that same library for you, and you acess it via an API, it should count, too. The same intermittent bugs that you waste time debugging in your own code can happen behind an API. You'll still waste a lot of time attempting to debug them.Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-24611023502842482062009-07-29T06:46:08.779-07:002009-07-29T06:46:08.779-07:00This is a good analysis of technical debt, but I h...This is a good analysis of technical debt, but I have a hard time extracting a clear point from it. <br /><br />The title 'embrace technical debt' and much of your early wording suggests that technical debt is 'valuable' and even advocates it as something that's desirable, but most of the points you make merely provide reasons why technical debt is an acceptable cost for making progress, or an unavoidable consequence of working with third parties.<br /><br />I really like the depth and breadth of your analysis, but the concept of embracing technical debt makes me a bit nervous. <br /><br />I think you hit this point on the head near the end by suggesting the application of a technique like Five Whys for balancing the two extremes (disallowing debt entirely, and considering any form of debt acceptable).<br /><br />I'm not sure that it makes sense to consider, say, PayPal's subpar API, as technical debt. You pay an ongoing cost in order to use that API, in the form of maintenance and debugging, but it's not as if it's debt you decided to take on and can decide to pay off later on down the road. The PayPal team might have technical debt in *their* code base, but as a consumer of their API, you don't.<br /><br />Now, on the other hand, if your codebase for dealing with PayPal also incurs technical debt, perhaps as a result of trying to quickly work around their issues, or fixing problems without fully understanding them - then that makes sense. But that's not necessarily a given.<br /><br />I am also uncertain about your suggestion that good design inevitably leads to technical debt. <br />If you have a piece of well-designed, well-implemented code that doesn't help you meet your goals, I wouldn't call that technical debt. The code does exactly what was intended, and it's easy to understand, improve upon, or throw out, because you factored it correctly in the first place.<br /><br />That feels more like wasted effort as a result of bad decisions than technical debt to me. And in that particular case, I would be especially cautious about embracing *that* kind of 'technical debt', since one thing programmers enjoy doing is building things the company doesn't need :)<br /><br />The line is admittedly pretty blurry in some cases; if the entire problem you're trying to solve is a technical one, I suppose you'd be well-justified in labelling almost any historical screwup or bad decision as 'technical debt'.Kevin Gaddhttps://www.blogger.com/profile/04689186557672996705noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-29630647526241678762009-07-29T06:24:14.126-07:002009-07-29T06:24:14.126-07:00Eric, Great article - provides context and vocabu...Eric, Great article - provides context and vocabulary for dialog on a very interesting tension in startups.<br /><br />Technical debt is a necessary instrument, but does cause real pain in the lives of developers - in terms of lost time and in terms of the psychological pain of going against engineering principles and experience. ;) But I think the article strikes the right tone in terms of balance.<br /><br /> Very much appreciate the suggested solutions (Five Why's, split testing etc)hendlerhttps://www.blogger.com/profile/05361002623416929401noreply@blogger.com