tag:blogger.com,1999:blog-7533727264507128560.post6674934225618312396..comments2024-03-12T23:38:00.147-07:00Comments on Lessons Learned: The one line split-test, or how to A/B all the timeErichttp://www.blogger.com/profile/12249063135381216090noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-7533727264507128560.post-19053552420003785022011-02-22T06:41:37.465-08:002011-02-22T06:41:37.465-08:00Hi Todd,
I think you're right that A/B testing...Hi Todd,<br />I think you're right that A/B testing is about optimization (choosing which option is optimal) - whereas you are asking a different question "Should I build an admin system at all?".<br /><br />This article on product discovery might be helpful - http://www.svpg.com/product-discovery-with-live-data-prototypes/.<br /><br />You still need quick feedback before committing a lot of resource to the opinion/guess/hunch that building an admin system is the right thing to do.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-49027382635390900962011-02-19T14:24:56.752-08:002011-02-19T14:24:56.752-08:00I like the theory of A/B testing, but don't un...I like the theory of A/B testing, but don't understand how it could be used for much more than cosmetic changes such as text and layout.<br /><br />For example, I have a service that pays people money for performing an online activity. For each user's reported activity, my service sends them a check every thirty days. Feedback at first was truly adoring and uptake was huge. Two years ago.<br /><br />Now new sign-ups are flat, even though the efficiency and reliability of my service has improved ten-fold. I'm thinking of building an online administrative section of my service where users can log in and actually see how much money they have earned historically and are earning in the current period. It will take more work to build this section than it took to build my original service. How can I use split-testing to determine whether I should build this new section before I actually spend time and money on it?Todd Rimeshttps://www.blogger.com/profile/10910682413880278037noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-41951201070125640632009-04-05T20:44:00.000-07:002009-04-05T20:44:00.000-07:00If you are trying to find a product that sells, th...If you are trying to find a product that sells, than each test would require a different product to be created (or at least a different sales page)...in many cases an example is required - demo, or in my case video guitar lesson along with google ads for quick test data. Maybe some things be sold with no sample for customers to see whatsover.<BR/><BR/>This adds to the complexity of decision making since you don't know it's the ad, sales page, product, keywords and so on.Willhttp://onlineguitarcoaching.comnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-12370485533680092952009-03-26T18:15:00.000-07:002009-03-26T18:15:00.000-07:00Great post Eric! I saw you speak yesterday at the...Great post Eric! I saw you speak yesterday at the FB garage and it reminded me of this article. <BR/><BR/>Our AB stuff always seemed to get pushed back a bit due to higher priority needs. So when we started chatting about AB testing here the discussion led to how fast an AB testing system could be implemented. To rectify this problem, a wager was made for a slurpee from 7-11 if a working AB test system was completed under 30 min. <BR/><BR/>The results... Had to drop down to a medium slurpee for an extra 10 minutes, but it was only 40 minutes from "Experiment" table creation to having a working AB testing system as described. It's not ready for production yet (it's missing nice reporting), but it goes to show there's no reason NOT to implement and start using something like this ASAP.<BR/><BR/>Thanks for the inspiration (and the slurpee)!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-32826025522370118422008-10-04T10:33:00.000-07:002008-10-04T10:33:00.000-07:00Since you're looking for big improvements, the res...Since you're looking for big improvements, the results should jump off the page, and slight changes in variance and other statistical measures should not be be a serious barrier to learning.<BR/><BR/>Of course, when you set up your A/B comparison, you need critical thinking to ensure you are not making silly mistakes.Sean Shttps://www.blogger.com/profile/03902084229952883298noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-78679080609460674832008-10-03T14:23:00.000-07:002008-10-03T14:23:00.000-07:00jesse,What you said is true, but the bias that hit...jesse,<BR/><BR/>What you said is true, but the bias that hitchens is talking about would still exist if you include in your final measurement ALL the visitors who were assigned into either group.<BR/><BR/>I think the change hitchens is suggesting, and which you may already be assuming, is pretty straightforward. In hitchens' example, where in month 1, 100% of visitors were assigned into the control group, and in month 2, the split is 50/50 between control and your new hypothesis, you'd get a biased result if at the end of month 2 you examined all the visitors' actions from both months 1 and 2. You remove the bias if you only look at the performance of visitors who were assigned into a group during month 2.blakeborgesonhttps://www.blogger.com/profile/18297435320300077910noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-74676619042111657312008-09-29T08:30:00.000-07:002008-09-29T08:30:00.000-07:00Hitchens,I don't see how that matters. When you t...Hitchens,<BR/><BR/>I don't see how that matters. When you test for statistical significance that all falls out.<BR/><BR/>In your example the control samples will have a lower variance than the test sample, that's all.Jesse Farmerhttps://www.blogger.com/profile/12325658166389857388noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-14876005360640062282008-09-24T00:29:00.000-07:002008-09-24T00:29:00.000-07:00Hi Eric! It's cool to hear you talk about this, b...Hi Eric! It's cool to hear you talk about this, because I just spent a week working on the experiment system at IMVU. What motivated this work was what I imagine is a common pitfall in experiment systems like the one you outlined:<BR/><BR/>The experiment branch assignment that chooses hypothesis vs control for each customer needs to know what probabilities it gives each group.<BR/>Every time you change those probabilities after the experiment is live, you bias the experiment.<BR/> <BR/>To explain why, I'll use the pathological example. Suppose your new experiment starts with 100% of users in the control branch. This is fairly common, because you're not ready to start the experiment yet. Then, a week later, you switch it to 50/50. But wait! During that first week, everyone was getting assigned to the control branch. Since you don't want to give anyone mixed behavior, all those users are <I>stuck there</I>. Only new users get the 50/50 split, so your hypothesis branch over represents new users and your control branch over represents existing ones.jmatheshttps://www.blogger.com/profile/17264560984365077588noreply@blogger.comtag:blogger.com,1999:blog-7533727264507128560.post-34417578019587517252008-09-15T21:12:00.000-07:002008-09-15T21:12:00.000-07:00Thanks for writing this article. I found your spec...Thanks for writing this article. I found your specific insights on a/b testing to be quite valuable. <BR/><BR/>Max Levchin of Slide and Paypal has noted that 10% of Slide's headcount is devoted to metrics only. This means optimizing everything possible in order to attract the maximum number of users for the longest amount of time.<BR/><BR/>In traditional marketing there are three main objectives. These involve persuading people to:<BR/><BR/>1. Buy a higher quantity<BR/>2. of higher priced items<BR/>3. More often <BR/><BR/>This logic can be extrapolated to the web world to mean: more users, visiting your site more often, for longer amounts of time to maximize advertising exposure. <BR/><BR/>Multivariate and A/B split testing combined with vision and lots of customer interactions are the best ways I know of to make something people want.Editorhttps://www.blogger.com/profile/18081096748243192297noreply@blogger.com