- having engineers post on the forums in their own name when they make a change
- routinely split-testing new changes
- routinely conducting in-person usability tests and interviews
- Net Promoter Score
Monday, October 26, 2009
Friday, October 23, 2009
This case study illustrates one company’s attempt to do customer development by testing their vision with customers before writing a single line of code. In the process, they learned a lot by asking initial prospects to sign a non-binding letter of intent to buy the software. As you’ll see, this quickly separated the serious early adopters from everyone else. Mainstream customers don’t have enough motivation to buy an early product, and so building in response to their feedback is futile.
Along the way, this case study raises interesting ethical issues. The lean startup methodology is based on enlisting customers as allies, which requires honesty and integrity. If you deceive customers by showing them screenshots of a product that is “in-development” but for which you have written no code, are you lying to them? And, if so, will that deception come back to haunt you later? Read on and judge for yourself.
The following was written an actual lean startup practitioner. It was originally posted anonymously to the Lean Startup Circle mailing list, and then further developed on the Lean Startup Wiki’s Case Studies section. If you’re interested in writing a future case study, or commenting/contributing to one, please join the mailing list or head on over to the wiki. What follows is a brief introduction by me, the case study itself, and then some Q&A led by LSC creator Rich Collins. Disclaimer: claims and opinions expressed by the authors of case studies are theirs alone; I can’t take credit or responsibility. – Eric Ries
In April of 2009 my partner and I had an idea for a web app, a B2C platform that we are selling as SaaS [software-as-a-service]. We decided from the get-go that, while we clearly saw the benefits and necessity of our concept, we would remain fiercely skeptical of our own ideas and implement the customer development process to vet the idea, market, customers etc, before writing a single line of code.
My partner was especially adamant about this as he had spent the last 6 months in a cave writing a monster, feature-rich web app for the financial sector that a potential client had promised to buy, but backed out at the last second. They then tried to shop the app around, and found no takers. Thousands of lines of code, all for naught -- as is usually the case without a customer development process. (See Throwing away working code
for more on this unfortunate phenomenon. -Eric)
We made a few pencil drawings of what the app would look like which we then gave to a graphic designer. With that, the graphic designer created a Photoshop image. We had him create what we called our "screenshots" (which suggests that an app actually existed at the time) and had him wrap them in one of these freely available PS Browser Templates. Now armed, with 4 "screenshots" and a story, we approached our target market, some of which was through warm introductions, and some, very literally, was through simple cold-calling.
Once we secured a meeting, we told our potential customers that we were actively developing our web app (implying that code was being written) and wanted to get potential user input into the development process early on. Looking at paper print-outs of our "screenshots", no one could tell that this was simply a printout of a PSD, and not a live app sitting on a server somewhere. We walked them through what we thought would be the major application of our product. Most people were quite receptive and encouraging. What proved to be very interesting was that we quickly observed a bimodal distribution with regards to understanding the problem and our proposed solution:
- people either became very excited and started telling us what we should do, what features it needed and how to run with this, or
- they didn't think there was a real problem here, much less a needed solution.
On the third visit, we pressed those who saw merit in the idea to sign a legally non-binding Letter of Intent. Namely, that they agree to use it free of charge if we deliver it to them and it is capable of X, Y and Z. And not only do they agree to use it, but that they intend to purchase if by Y date at X price if it meets their needs.
By the way, this LOI was not written in legalese. Three quarters of it was simple everyday English. In fact, we customer dev-ed the LOI itself. The first time, we asked a client to sign it before we had even written it. When they agreed to sign it, we quickly whipped it up while sitting in a coffee shop and emailed it off to them. This would help us separate the wheat from the chaff when it came to determining interest and commercial viability. Once we had two LOIs signed and in-hand, we actually began to write code.
We also implicitly used the LOIs for price structure and price discovery - which we are still working on. We backed into prices from all sorts of angles, estimating the time-cost of equivalent functionality, competitive offerings, other tools we were potentially displacing -- but in the end, we lobbed a few numbers at them and waited to see if they flinched.
Customer A got X price, Customer B got X + Y price, and so on. So far, our customers have never mentioned price as an objection, which suggests to me that at this point we are very much underpriced. The LOI was also useful as we leveraged it by approaching the competitor of one of those who signed by simply letting them know that their competitor will be using our app. They returned our cold intro email within 8 mins.
We have two customers that have balked at signing LOIs, but want to use our product. This has been somewhat of a quandary for us. When we decided to go the LOI route, we thought that we would not bend and that we would only service those customers who would sign the LOI. In the end, we decided that these two customers were large enough to help us with exposure, provide good usage data and worth the risk of them wasting our time. Time will tell if this theory proves correct.
Right now, the app itself is pretty ugly, a bit buggy and slow -- and doesn't even do a lot. It is borderline embarrassing. Don't get me wrong, it does the few necessary things. BUT it definitely does NOT have the super-duper-hyper-ultra-cool Web 2.0 spit and polish about it. Interestingly enough, our ratio of positive comments to negative comments from actual users is about 10 to 1. One of our first customers had a disastrous launch with it, yet, has signed on to try it again (granted, they did get it for free and we did offer it for free for this next time). But they didn't hesitate to try it again. I thought we would have to plead, beg and beseech. But for them, it was a no-brainer. So, we have to be doing something right.
Our feature set is very limited and being developed almost strictly from user input. While I personally have all sorts of super-duper-hyper-ultra-cool Web 2.0 ideas --- we are holding ourselves back, and forcing ourselves to wait for multiple, explicit and overlapping user requests. We have seen our competitors whose feature sets are very rich, to say the least, but we think in some cases, are as over-engineered as they are feature-rich.
Only time and the market will tell if they are innovative and we are slow, lazy pigs or they have gotten ahead of themselves/the market and our minimalist solution will be better received.
Rich Collins, founder of the Lean Startup Circle, responded to the poster with some Q&A.
LSC: What is your response to some of the people on Hacker News that questioned the ethics of taking this approach?
Some of the commenters have some good points. It definitely explores ethical boundaries. However, I don't think we indulged in any zero-sum game type deception. By that, I mean our intentional fuzziness about the state of development did not cause harm in any manner to our prospective clients. In fact, just us showing up at their offices and talking about our screenshots benefited our prospective clients tremendously as:
- Those clients who had never even entertained the functionality we were proposing gained significant knowledge.
- With that knowledge, they could (and did) Google our competition and start exploring the space and current offerings.
"Oh, this is just a Photoshop file? Well, come back to us when you are further along." which defeats the whole purpose of getting face time for Customer Development!
When you tell them, the app is in development (and it was, even before coding, we were spending a lot of time on what we wanted and didn't want, how it would look, use cases ‚ etc) the prospects are interested in providing input and shaping the product. They need to feel and see some momentum.
LSC: Your use of a non-binding letter of intent was another interesting tactic. Did the customers that signed it end up paying for your product?
Yes and no. We had a dispute with one signee and couldn't convert them. However, we successfully converted others. I should also mention that there was one client who refused to sign an LOI, but we are in the process of converting them.
The LOI was designed to give us hard, non-bullshit-able feedback instantly. Too often people will affirm your idea so that you (or they) can save face, which BTW is a form of well-intentioned and socially acceptable deception. This is why, IMHO, friends, wives, and significant others are probably not good people to talk to about your idea. At the end of the day, no one knows if the idea is any good. The market will tell you.
LSC: Would you respond to a few selected Hacker News comments?
"If I were one of your prospects, I would never sign a letter of intent based on drawings only. I'd make you come back later with something, anything I could play with ... Come back when you have something real to show. Until then you're no different from any other poser."
I myself probably would never sign an LOI on screenshots only. However, our customers did a lot of stuff that I would never do. Lesson learned: I am not my customer. We think differently. We solve our problems differently. We have different needs and wants. Repeat after me: You are not your customer.
LSC: And one more:
Wrong. We got instantaneous feedback on the validity of the idea and started our sales process concurrently. While legally non-binding, customers who have signed an LOI are a lot less likely to disappear or make themselves hard to get a hold of. LOIs, while clearly not as good as signed sales contract, do have meaning and are valuable. I encourage B2B startups to keep them in their customer development arsenal.
Special thanks to Rich Collins, the Lean Startup Circle practitioners, and to everyone who has contributed to the Case Studies on the wiki. And thanks to these entrepreneurs for sharing their story. Have a case study you’d like to share? Head on over to the Lean Startup Wiki.
Monday, October 19, 2009
One of the unfortunate side effects of all the publicity and hype surrounding startups is the idea that entrepreneurship is a guaranteed path to fame and riches. It isn’t. Building a startup is incredibly hard, stressful, chaotic and –- more often than not –- results in failure. That doesn’t mean it’s not a worthwhile thing to do, just that it’s not a good way to make money.
A more rational career path for money-making is one that rewards effort, in the form of promotions, increased security, salary and status. Startups, unfortunately, punish effort that doesn’t yield results. In fact, the biggest source of waste in a startup is building something nobody wants. While in an academic R&D lab, creation for creation’s sake will often get you praise, in a startup, it will often put you out of business.
So why become an entrepreneur instead of developing technology in an R&D lab? Three reasons: change the world, make customers’ lives better and create an organization of lasting value. If you only want to do one of these things, there are better options. But only startups combine all three.
Take this fictional example of a Seedcamp attendee (actually a composite), which I will refer to as Hairbrush 2.0...
Read the rest of Myth: Entrepreneurship Will Make You Rich
Also take a look at the great Hacker News discussion of this essay. It includes several gems, including this comment from davidu:
1) Being an entrepreneur, for me, isn't about being wealthy, it's about being successful.and this one from gits_tokyo:
2) Rich is a variable term, and intended to be so.
Entrepreneurship may not make you wealthy, but it can certainly make you rich.
I enjoy the freedom and independence afforded by starting EveryDNS and OpenDNS. Both contain a passion for a system I love, the DNS, and both have let me help millions of consumers around the world. I even like knowing I control the DNS for millions and millions of Internet users. That's an awesome responsibility and it certainly makes me feel rich about everything I do.
And when it comes to money, Eric is only somewhat right. He says you should get a job that rewards and promotes effort. But lots of lawyers and finance kids in New York thought they had stable jobs that would make them rich. Ask them today and most will tell you a different story altogether. Now they hate their jobs and have no job security or path to becoming really wealthy.
So like I said, being entrepreneur, for me, isn't about being wealthy, it's about being successful. That's a measuring stick that's far more important.
People that I've spoken with in the past more often than not associated the idea of me doing a startup in the tech industry with gaining massive wealth. While I may entertain this, deep down I find it lacking as there's so much more than wealth to be had.
How about, living in a world... some distant future from the everyday-everyday where day-by-day you toil piecing together a vision, one day injecting it into the present, in order to influence a whole new set of social behaviors while also unfolding valuable opportunities. How about, the day of flipping that proverbial switch, releasing this vision out in the wild. How about, the potential of millions interacting with your vision, it becoming a staple part of a users online experiences. There's something undeniably provoking about all this, rush of my life.
Wealth, although a welcomed aside pales in comparison. Hell I would even go so far as to say, in a world where sex is constantly peddled as a cure all, let me say it, sex pales in comparison to the feeling I get from being an entrepreneur.
One of the most gut-wrenching moments for a company is the rollout of a new product. A significant swing and miss can break a company's momentum -- and maybe its bank account. Unfortunately, after months or even years of development, many companies discover that customers aren't willing to buy their new wares. That's why some entrepreneurs are trying another approach to product launches: marketing a product online before spending much on research and development or inventory.
Consider the method used by TPGTEX Label Solutions, a Houston-based software company that specializes in bar codes and labels for manufacturers and chemical companies. Like many companies, TPGTEX rolls out new products several times a year. But instead of spending the time and money to develop products on spec, TPGTEX creates mocked-up webpages that list the features of a potential new product -- such as a system for making radio-frequency identification, or RFID, labels -- along with its price. Then, the company spends no more than a few hundred dollars marketing the product through search engines and to the contacts in its sales database and LinkedIn. It isn't until a customer actually clicks or calls to place an order that TPGTEX's developers will build the software. "We do not develop a product until we get a paying customer," says Orit Pennington, who co-founded the six-employee company with her husband in 2002. Development time is typically no more than two to three weeks, and it generally takes just a few orders to cover development costs.
TPGTEX's approach is an example of a trend in business that has been dubbed minimum viable product or microtesting. The idea is to develop something with the minimum amount of features or information needed to gauge the marketability of a product online. That might mean mocking up a website with potential features and seeing how many visitors click on the item. It might also involve buying pay-per-click ads to see how easy it is to gain potential customers. Or it might mean selling a few products on a site like eBay to see how well they perform before ordering in bulk from a wholesaler.
What sets this approach apart from practices like using focus groups is that companies base product development decisions not just on what customers say they want but on how they vote with their wallets.
Read the rest...
Sunday, October 11, 2009
- Any team can create a true split-test experiment that affects only the sandboxed parts of the product, however:
- One team must see the whole experiment through end-to-end.
- No experiment can run longer than a specified amount of time (usually a few weeks).
- No experiment can affect more than a specified number of customers (usually expressed as a % of total).
- Every experiment has to be evaluated based on a single standard report of 5-10 (no more) key metrics.
- Any team that creates an experiment must monitor the metrics and customer reactions (support calls, forum threads, etc) while the experiment is in-progress, and abort if something catastrophic happens.
- It forces teams to work cross-functionally. The first few changes, like a price change, may not require a lot of engineering effort. But they require coordination across departments – engineering, marketing, customers service. Teams that work this way are more productive, as long as productivity is measured by their ability to create customer value (and not just stay busy).
- Everyone understands the results. True split-test experiments are easy to classify as successes or failures, because top-level metrics either move or they don’t. Either way, the team learns immediately whether their assumptions about how customers would behave were correct. By using the same metrics each time, the team builds literacy across the whole company about those key metrics.
- It promotes rapid iteration. When people have a chance to see a project through end-to-end, and the work is done in small batches, and has a clear verdict delivered quickly, they benefit from the power of feedback. Each time they fail to move the numbers, they have a real opportunity for introspection. And, even more importantly, to act on their findings immediately. Thus, these teams tend to converge on optimal solutions rapidly, even if they start out with really bad ideas.
Tuesday, October 6, 2009
In the meantime, I hope you enjoy all this multimedia content. In addition to some of my recent talks, you can learn more about the Startup Visa movement and enjoy two really interesting lean startup case studies.
if you'd like to follow along with slides, they are here:
From high atop the BT Tower in London, this brief BT Tradespace interview:
Why do we need a Startup Visa? A Tale of 2 Erics:
Also in London, I took up a lot of airtime during day two of Seedcamp. You can read highlights on their blog, or watch this short video:
Seedcamp - Day 2 Highlights from Seedcamp on Vimeo.
Or watch my full #leanstartup presentation at Seedcamp in London:
And two bonus videos that are well worth watching (weally):
Timothy Fitz, who worked for me at IMVU, giving an in-depth presentation on the details of the continuous deployment system that we built there.
With accompanying slides:
pbWorks (formerly pbWiki) was one of the first companies that ever invited me to join their advisory board. I like to think that had some small part in causing their subsequent success. Judge for yourself by watching David Weekly's #leanstartup case study (pbWorks):
Thanks to everyone who has helped plan, organize, record and attend these many events!