tag:blogger.com,1999:blog-7533727264507128560.post2430828577525845474..comments2024-03-28T10:49:53.597-07:00Comments on Lessons Learned: When NOT to listen to your users; when NOT to rely on split-testsErichttp://www.blogger.com/profile/12249063135381216090noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-7533727264507128560.post-64116750925219871652009-05-26T20:53:38.942-07:002009-05-26T20:53:38.942-07:00For a quantitative "methodology" on identifying op...For a quantitative "methodology" on identifying opportunities to deliver customer value, have a look at "What Customers Want" by Anthony Ulwick. I have found it an excellent place to begin when focusing in on the minimum priority features that customers are after. The process is empirical which suited my system-oriented background.<br /><br />As with any "methodology" don't simply follow the process blindly but take the principles and develop your own practices.Marknoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-28632859370168734942008-10-07T23:02:00.000-07:002008-10-07T23:02:00.000-07:00@Chris - thanks for the kind words (and for appare...@Chris - thanks for the kind words (and for apparently boosting my subscriber count!). I think you nailed the summary. In fact, when you lay it out so clearly, it almost sounds easy. <BR/><BR/>Why do we find this so easy to say and yet so hard to do? And what can we do to mitigate these difficulties, even as we scale?Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-41082747955256073622008-10-07T22:38:00.000-07:002008-10-07T22:38:00.000-07:00Eric,Fellow PBwiki shareholder Chris Yeh here. I ...Eric,<BR/><BR/>Fellow PBwiki shareholder Chris Yeh here. I thought that this was a great post, and I've quoted it extensively to folks via email. Here is the quick summary I've sent...let me know if you think I'm oversimplifying:<BR/><BR/>1) When trying to be customer-centric, emphasize the strategy (learning) rather than the tactic (listening to customer feedback).<BR/><BR/>2) Split-testing has a strong appeal to our desire for data but can't substitute for judgment.*<BR/><BR/>3) Try to specify the fewest number of features required to meet the company's goals.<BR/><BR/>4) When crossing the chasm, you may need to focus away from the archetypes that currently dominate your customer base.Chrishttps://www.blogger.com/profile/00927628412285314176noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-46026927928719909072008-10-06T23:25:00.000-07:002008-10-06T23:25:00.000-07:00Eric,Yeah, it has to be part of the culture or lea...Eric,<BR/><BR/>Yeah, it has to be part of the culture or leads to tedious arguments about process. Engineers just want to build, designers just want to create, product managers just want to spec, etc.<BR/><BR/>Why?<BR/><BR/>It's hard to get buy in. It means some amount of additional work for the people involved (engineers, designers, PMs) with the additional risk that their opinion will be invalidated.<BR/><BR/>The end results is a better business and product, but each individual feels like they're being more actively scrutinized.<BR/><BR/>You have to be in a culture that takes away the stigma from this scrutiny and embraces scientific experimentation and part of the day-to-day job or it causes too many bruised egos.<BR/><BR/>Anyhow, it's clear you've been thinking about the general issue for some time. There are lots of people out there talking about metrics and measurement, but not a lot of people talking about this from a scientific perspective.<BR/><BR/>I'm trying to distill this perspective, which lots of people I know share, into something you can hang your hat on. Any thoughts?Jesse Farmerhttps://www.blogger.com/profile/12325658166389857388noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-22742458275736182412008-10-06T22:10:00.000-07:002008-10-06T22:10:00.000-07:00@jesse - I completely agree. Those who claim that ...@jesse - I completely agree. Those who claim that science is non-creative, static, or just numbers don't really understand science at all. <BR/><BR/>Have you ever had the problem where people with those misconceptions resist subjecting their product insights to quantitative analysis? I seem to come up against it quite often.Erichttps://www.blogger.com/profile/12249063135381216090noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-84568166942417072062008-10-06T21:41:00.000-07:002008-10-06T21:41:00.000-07:00I don't see any contradiction between a scientific...I don't see any contradiction between a scientific mindset and your conclusions.<BR/><BR/>In fact, it more-or-less mirrors how scientific breakthroughs happen. People spend years looking at the trees but every so often someone comes along and points out the forest.<BR/><BR/>And qualitative feedback certainly has its place in this system, too, with the caveat that users are often wrong, confused, and will mislead you. Nothing the users tell you should contradict your empirical findings, after all, unless you're doing something wrong.<BR/><BR/>Any time I'm focusing on small optimizations I think to myself, "Is there any way I can move the dial 500% rather than 50%?"<BR/><BR/>That's not pushing aside science -- that's doing it better.Jesse Farmerhttps://www.blogger.com/profile/12325658166389857388noreply@blogger.com