A couple of weeks ago I shared the idea of validating your product and creating an experimentation plan for teaching strategic skills to product managers and product owners.
How can I execute on the experiments?
Brief Background
My first business was building Personal Computers and selling them to local businesses at the age of 22 when I graduated from Engineering College in India. I would go to lawyer offices, Doctors and other professionals. Selling them a home built PC with service included for a year. Once I got the order, I would ask for a week to deliver the PC. I would then go to a friend who sold computer parts, he would loan me the parts in advance, I would build the computer, sell it and then pay my friend for the parts. It was a very successful business which I had to close down as I decided to pursue my Master’s in Computer Science in the USA.
Why is this significant because as an entrepreneur several years later I built an email marketing platform. I failed miserably in selling it. There were a couple of companies that had just started Constant Contact and Mailchimp. I was deeply impressed with that they did and my hypothesis was that I can sell to small and medium sized businesses who had not yet heard of email marketing.
The product was assembled using an open source framework and we built a beautiful user interface. But by then I had forgotten the lessons from my previous venture and thought that the only way to validate a business idea was to build a working product to prove to sell it. Needless to say my business was a failure as I was not able to get enough subscribers.
Over the years I have talked to several entrepreneurs that have spent several years and significant money to build solutions that eventually did not sell.
When I started working as a Product Manager one of the first books I came across was Testing Business Ideas written by David Bland – https://amzn.to/40jsZ6P
A new and key learning from this book was you don’t rely on only one experiment. You conduct a series of experiments that makes the evidence stronger and you pivot based on the data until you find the right product or feature that your lead will line up to buy.
So I decided to explore a few different companies that were successful and what experiments they ran. The most striking pattern across all companies is the precision of their measurement approach – rather than pursuing broad growth initiatives, each focused on specific, measurable experiments that could be directly tied to revenue impact. This methodical validation approach enabled rapid scaling while maintaining unit economics, with several companies achieving profitability or major acquisitions within 2-4 years of their key validation breakthroughs.
The card format used below are from a template from the book and You can download the learning card template from https://www.strategyzer.com at https://cdn.prod.website-files.com/647655122de68d8272dce023/65c35cb35f3dfbdd49e29cdf_The%20Learning%20Card%20-%202023.pdf
Airbnb:
Experiment 1: Initial Concept Test
Hypothesis: We believed that people would pay to stay in strangers’ homes during sold-out events when hotels are unavailable.
Observation: We observed that all three air mattresses were rented during the design conference, guests were satisfied with the experience, and we made $1,000 in revenue.
Learning and Insights: From that we learned that people will overcome the trust barrier when they’re desperate for accommodation and the value proposition is clear.
Decisions and Actions: Therefore we will build a website to test if this works beyond crisis situations and one-off events.
Experiment 2: Website Launch
Hypothesis: We believed that we could scale the concept beyond events to regular travel by creating a platform for hosts and guests.
Observation: We observed extremely low booking rates, struggling to make $200/week, and both hosts and guests seemed hesitant to engage.
Learning and Insights: From that we learned that something fundamental was preventing transactions from happening at scale.
Decisions and Actions: Therefore we will go directly to hosts and guests to understand what’s blocking them from using the platform.
Experiment 3: User Research
Hypothesis: We believed that by interviewing hosts and observing their listings, we could identify the primary barrier to bookings.
Observation: We observed that host photos were consistently poor quality – blurry, dark, and didn’t showcase the spaces well.
Learning and Insights: From that we learned that guests need to visualize the space clearly before they’ll trust staying there, and most hosts don’t know how to take appealing photos.
Decisions and Actions: Therefore we will test if professionally photographed listings get significantly more bookings.
Experiment 4: Photo Quality Test
Hypothesis: We believed that professional photos would increase booking rates for listings.
Observation: We observed that listings with professional photos got 2-3x more bookings than those with amateur photos.
Learning and Insights: From that we learned that photo quality is a critical conversion factor, and improving it can dramatically impact our marketplace liquidity.
Decisions and Actions: Therefore we will offer free professional photography services to hosts to improve the overall quality of our platform.
Airbnb’s photo quality experiments delivered the most precisely measured validation results, with their living room background study showing a 35% increase in booking rates translating to $728 additional revenue during a 16-night holiday period. The company’s broader professional photography program analysis of 100,000+ listings revealed 28% more bookings, 26% higher nightly rates, and 40% increase in overall earnings for properties with professional photos.
The New York City pilot program demonstrated 2-3x more bookings with professional photography, leading to doubled monthly revenue after full implementation. These results were so compelling that enhanced listings became 2.5x more likely to be booked, with hosts earning an average of $1,025 per month compared to standard listings. The company’s algorithm improvements based on these insights generated an additional 4% lift in booking conversion rates across the platform.
SweetGreen
Experiment 1: College Food Desert
Hypothesis: We believed that college students wanted access to fresh, healthy food options but had limited choices on campus.
Observation: We observed long lines at our Georgetown University location and students expressing frustration about lack of healthy options.
Learning and Insights: From that we learned that young people will choose healthy food when it’s convenient, tasty, and reasonably priced.
Decisions and Actions: Therefore we will test if this concept works in other urban locations beyond college campuses.
Experiment 2: Urban Expansion
Hypothesis: We believed that young urban professionals would have similar demand for fast, healthy food as college students.
Observation: We observed strong demand in Washington DC locations, with customers willing to wait in long lines for customized salads.
Learning and Insights: From that we learned that fast-casual healthy food appeals to time-pressed urban professionals, not just students.
Decisions and Actions: Therefore we will test if we can scale operations while maintaining food quality and speed of service.
Experiment 3: Operational Scaling
Hypothesis: We believed that we could systematize our operations to maintain quality while opening multiple locations efficiently.
Observation: We observed that some locations maintained quality while others struggled with consistency, speed, and customer experience.
Learning and Insights: From that we learned that operational excellence requires standardized processes and intensive training, especially for food preparation.
Decisions and Actions: Therefore we will develop comprehensive training programs and test local sourcing to differentiate from competitors.
Experiment 4: Local Sourcing Focus
Hypothesis: We believed that emphasizing local farm partnerships would differentiate us from competitors and appeal to environmentally conscious consumers.
Observation: We observed that customers responded positively to knowing where their food came from and were willing to pay premium prices for local ingredients.
Learning and Insights: From that we learned that transparency and sustainability can be competitive advantages that justify higher prices.
Decisions and Actions: Therefore we will test digital ordering and loyalty programs to improve customer experience and capture more data.
Experiment 5: Digital Integration
Hypothesis: We believed that mobile ordering and loyalty programs would reduce wait times and increase customer retention.
Observation: We observed that digital customers visited more frequently, spent more per visit, and provided valuable data about preferences.
Learning and Insights: From that we learned that digital integration enhances rather than replaces the in-store experience for fast-casual dining.
Decisions and Actions: Therefore we will continue expanding while investing in technology that improves both customer and employee experience.
Sweetgreen’s digital validation experiments showed that customers moving from frontline to digital ordering came 1.5x more frequently and spent 20% more per transaction. Two-channel customers demonstrated 2.5x higher frequency than single-channel users, driving 67% of total revenue through digital channels by 2022, up from 30% in 2016.
Some other examples include:
Dollar Shave Club
Dollar Shave Club started by identifying personal frustration with expensive razor blade cartridges, which friends and colleagues validated as widespread consumer pain. Their breakthrough came with a humorous viral video explaining their value proposition. Dollar Shave Club’s viral video created immediate measurable impact with 12,000 new subscribers in the first 48 hours and 4.75 million views in three months. The $4,500 production budget yielded a customer acquisition cost of just $0.375 in the first 72 hours. The video ultimately generated 27+ million total views and helped the company capture 15% market share in just four years.
While initial fulfillment costs exceeded expectations, high customer satisfaction and retention rates showed subscription convenience created strong loyalty even with thin margins. Product line expansion into shave butter, body wash, and other personal care items significantly increased customer lifetime value, proving that building trust in one category enabled expansion into related categories with the same customers, ultimately leading to their $1 billion acquisition by Unilever.
Groupon:
Groupon began by manually negotiating deals with local businesses, starting with a pizza restaurant that sold over 500 vouchers in one day, proving businesses would sacrifice margin for guaranteed volume and predictable cash flow. They built an email list for daily deals in Chicago, observing rapid growth and 30-50% open rates, demonstrating strong consumer appetite for significant discounts delivered conveniently.
Geographic expansion to Boston showed similar success, proving the model wasn’t Chicago-specific, while testing beyond restaurants revealed that diverse categories like spas and activities actually performed even better. Their evolution toward merchant self-service revealed that businesses needed guidance and tools to create effective deals, not just platform access, leading them to build merchant tools with best practices to maintain deal quality while scaling globally.
Groupon’s geographic expansion delivered measurable subscriber conversion patterns, with their initial expansion from 5 to 175 North American markets showing consistent 19% subscriber-to-customer conversion rates. Key markets like Boston and Chicago achieved 33% of subscriber base converting to buyers, with average revenue per merchant of $11,000 and $23 average Groupon price. Slideshare
The company’s early deal performance provided precise validation metrics: their first deal (two pizzas for one) sold 25 units, while early financial results showed $2,595 gross sales and $174 revenue in the first 15 days. Kickstartsidehustle This scaled to 83.1 million subscribers by September 2011 from 152,203 in June 2009 (2,320% growth).
Dropbox
Dropbox started with founder Drew Houston recognizing his own file syncing pain point, which friends and colleagues immediately validated as a widespread problem. Rather than building the full product first, they created a simple demo video showing how seamless file syncing would work, resulting in 75,000 signups overnight from Digg – far exceeding expectations and proving massive market demand existed.
Dropbox achieved the most documented viral growth success, with their video MVP generating the famous 5,000 to 75,000 user jump overnight. But their referral program produced even more sustained impact: 35% of daily signups were attributed to referrals, creating a 60% permanent increase in signup ratesaccording to CEO Drew Houston.
Through beta user interviews and testing, they learned users cared most about reliability and simplicity over advanced features, leading them to focus on perfecting core syncing functionality. Their final validation came through testing a referral program offering extra storage space, which created viral growth and proved users would actively promote products that solved real problems for people they cared about, establishing referrals as their core growth strategy.
The referral mechanics were precisely calibrated – 500MB storage rewards for both referrer and referee, with referrers able to earn up to 16GB through successful referrals. This drove 2.8 million referral invites in April 2010 alone and maintained 15-20% month-over-month growth throughout the program’s peak performance period. The company’s user base doubled every three months during this period, growing from 100,000 to 4 million users in 15 months (3,900% growth).
Warby Parker
Warby Parker began by testing whether people would try on glasses at home, starting with friends and family who loved the convenience and felt more confident making decisions in their own environment. They then validated they could source high-quality frames directly from Italian manufacturers at fraction of retail prices, bypassing traditional markups. Their online showroom launch with home try-on service exceeded conversion expectations, proving convenience trumped traditional in-store fitting for many customers.
Warby Parker’s Home Try-On program achieved a 50% conversion rate for customers who completed the five-day trial, with these customers representing 30% of total sales by 2022. The program’s social media integration added another performance layer, with customers who shared content during their trial being 50% more likely to purchase.
The company discovered that physical retail locations actually complemented rather than competed with their online business, becoming community hubs that increased brand awareness and drove more online sales in surrounding areas. Each experiment built on the previous learning – from validating consumer desire for home try-on, to proving manufacturing economics, to demonstrating that omni channel retail could work synergistically rather than cannibalistically.
The company’s customer acquisition cost optimization was equally precise – maintaining $40-55 CAC while achieving 130-220% return on acquisition investment. Their retail versus online performance data showed interesting channel dynamics: 7.7% mobile app conversion rate versus 2.5-3.0% retail store conversion, though retail ultimately generated 64% of total revenue by creating a premium experience that supported higher average order values.
What did I learn from these case studies:
In my last article I created an experimentation plan with AI prompts. My product is a course to teach strategic skills to product managers. As I looked at my plan I realized this was a huge course. So I pivoted by niche it down to teaching one skill that I think is the most important especially when you are starting up – Validating your business idea with a series of experiments using AI to turn an Idea to Income. The audience for this course are Entrepreneurs and Product Leaders.
My hypothesis was that if I had this course with three 2 hour sessions along with self paced study and materials. After attending my course, My learners will walk away with a successful product or service, an offer their customer cannot refuse and a clear plan to generate interested and qualified leads.
Presently I am preparing to run a series of experiments for my course and I am going to share what I find in my next week’s newsletter.
What are your thoughts? Will a course like this be valuable to you? Would you like to help me shape the course? It will really help me a lot if you can complete this brief survey – https://maven.com/forms/41b701
Until next week…
Would you like to learn how to build a Vision, Strategy and Roadmap as a Product Leader? Join our Live Online Advanced Scrum Product Owner Workshop to elevate your strategic skills. Check out my A-CSPO schedule at https://www.eventbrite.com/cc/advanced-certified-scrum-product-owner-a-cspo-2770959
