tag:blogger.com,1999:blog-7533727264507128560.post5706065594261700106..comments2024-03-28T10:49:53.597-07:00Comments on Lessons Learned: Five WhysErichttp://www.blogger.com/profile/12249063135381216090noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-7533727264507128560.post-29958770117177185722010-06-15T04:25:57.753-07:002010-06-15T04:25:57.753-07:00The link to ‘cool paper’ is broken, should be http...The link to ‘cool paper’ is broken, should be http://www.evanmiller.org/poisson.pdfAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-29779407308110580792009-06-20T14:52:19.216-07:002009-06-20T14:52:19.216-07:00Great related post by John Shook at the Lean Enter...Great related post by John Shook at the Lean Enterprise Institute about technical vs. social sides of problems.<br /><br />You're right to look at "lack of training" as a root cause (and to continue to ask why there was a lack of training). Most organizations would blame the person for not being trained, as if that's their fault!<br /><br />http://www.lean.org/shook/2009/05/is-your-technical-person-technical.htmlMark Grabanhttps://www.blogger.com/profile/07953086531083611251noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-60555482165459622642009-06-15T21:27:47.017-07:002009-06-15T21:27:47.017-07:00my data warehouse project has a "single share...my data warehouse project has a "single shared database" in it too (the warehouse), and plenty of databases too (post-processing stages, marts). It also has multiple environments (in addition to production). All environments run basically the same (nightly) ETL data integration pipeline - it's a batch process environment.<br /><br />I don't see how aplying Continuous Deployment (CD) to data marts (and the analytic products that depend on them) would be any different to any other online application.<br /><br />The ETL pipeline is a batch process, so continuous deployment would always be batched-up to "catch the next train leaving the station". Is that not continuous? To be honest, I'm nervous about CD for my ETL, I guess that's because when it goes wrong it tends to fail completely. Our dev/test/prod routine tends catch one of these a few times a year. But also, the ETL doesn't change much anymore (except when integrating a new system), now that the warehouse is in production most of the change occurs in the marts and analytic products.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-62171218179630853582009-02-11T08:05:00.000-08:002009-02-11T08:05:00.000-08:00Love the discussion. Wish I could figure out how t...Love the discussion. Wish I could figure out how to apply it to the average data warehousing project which has a single shareed database right at the heart of the entire solution...<BR/>It seems your cluster architecture is one of the key architectural constraints making continuous deployment possible.<BR/>If you can't deploy to 5% of the nodes and check the results, then how would you accomplish continuous deployment?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-28645198781451390252008-11-17T17:48:00.000-08:002008-11-17T17:48:00.000-08:00@"What started as a technical problem actually tur...@"What started as a technical problem actually turned out to be a human and process problem."<BR/><BR/>yes yes yes a thousand times yes. it kills me how apparently most people don't seem to grok that or care about it, from what empirical evidence i've seen over the years.Raoul Dukehttps://www.blogger.com/profile/07354740962526930549noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-77031631313712076112008-11-15T11:18:00.000-08:002008-11-15T11:18:00.000-08:00The five whys methodology has been around for some...The five whys methodology has been around for some time, not just as a continuous improvement technique, but as a general problem solving method in all areas of management. <BR/><BR/>As a management consultant who concentrates on thinking and problem solving, I use it often. But like any other tool, it is more appropriate for specific problems, and so in a general sense shouldn't be used alone. It is a great supplement to other problem solving techniques, such as brainstorming, especially if you let it be a little more free form. <BR/>Tony Wanless, Knowpreneur ConsultantsReinventionisthttps://www.blogger.com/profile/04316957347564705360noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-63423352485657783372008-11-14T17:00:00.000-08:002008-11-14T17:00:00.000-08:00@hitchens - I'm not dismayed at all. Five whys is ...@hitchens - I'm not dismayed at all. Five whys is a technique for continuous improvement, and it can be applied to itself as well. As long as you're getting to the true root cause of each problem, including the human dimension, it really doesn't matter how many questions you ask. "Five whys" is not meant literally. <BR/><BR/>Glad to hear the discipline continues, albeit in modified form.Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-33576449946246817842008-11-14T16:41:00.000-08:002008-11-14T16:41:00.000-08:00Eric and Chris, you may be dismayed to hear that w...Eric and Chris, you may be dismayed to hear that we don't actually do 5Ys at IMVU any more. The rigid structure was too cumbersome.<BR/><BR/>Instead, whenever there is a failure, we have a postmortem meeting with everyone involved and come up with follow-up tasks. These tasks are then given high priority.<BR/><BR/>This gives us more flexibility about how much follow up work to do, and how many levels of abstractions to take an investigation. Fixing it at 5 was usually too many, although it was unarguably helpful at its inception, for getting us rid of some bad habits.jmatheshttps://www.blogger.com/profile/17264560984365077588noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-9015156422653584692008-11-14T03:19:00.000-08:002008-11-14T03:19:00.000-08:00Good stuff. I picked up on the 5 whys from Joel o...Good stuff. I picked up on the 5 whys from Joel on Software earlier this year.<BR/><BR/>The piece you add about committing to each stage of the process is news to me. It should be obvious, but it's often not. Seems to be a key part of the discipline.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-66968800095921333802008-11-13T20:59:00.000-08:002008-11-13T20:59:00.000-08:00Wanted to add one additional note from the perspec...Wanted to add one additional note from the perspective of someone who was intimately involved in developing this system. <BR/><BR/>When Eric writes, "By the time we started doing continuous integration, we had tens of thousands of lines of code, all not under test coverage." he is substantially understating how far we were from the eventual system. <BR/><BR/>We had closer to several hundreds of thousands of lines of code. Most of this code was from a variety of open source PHP projects that were glued together with the shortest path to goal possible. Most of this was code that not scalable, not secure, and not particularly extensible. Most of the code had an internal structure that would cause reasonable architect to conclude that the best thing to do was start a re-write project.<BR/><BR/>Nonetheless through a consistent application of root cause analysis over we migrated to a highly scalable and highly available system.Chris Hondlhttps://www.blogger.com/profile/03282556433049642012noreply@blogger.com