SlideShare a Scribd company logo
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
The Statsig Team
statsig.com
Statsig is a modern experimentation
and feature flagging platform. We help
companies like Notion, OpenAI, Figma, and
Atlassian manage feature rollouts and
compute experimental results.
Statsig Cloud
? >200B events a day
? >20k total experiments across >1B unique user
identifiers.
Statsig Warehouse Native
? Full power of Statsig Cloud but raw data never
leaves your data warehouse.
Overview
Review of Experimentation 101 Experimentation 201
1. CUPED
2. Holdouts
3. The Peeking Problem and Sequential Testing
4. Stratified Sampling
5. Switchback Experiments
6. Multiarmed Bandits
7. Heterogeneous Treatment Effects
8. Experimental Meta Analysis
statsig.com
1. AB Testing Basics
Experimentation 101:
Why A/B Test?
Building products is hard
Scientific gold standard for measuring causality
Ideas are evaluated by causal user data not opinions
Product development becomes a scientific, evidence-driven process
How Does Testing Work?
POPULATION ASSIGNMENT TREATMENT ANALYSIS
Control
Test
17%
25%
? Start with a hypothesis
? Power Analysis (tradeoff between sample size, statistical power, and time)
? Standardized methodology
? Use 95% confidence intervals by default
? Don’t fret about interaction effects
Experimentation Best Practices
The HiPPO
Stats Engines Don’t Build Culture
Experimentation should be easy and automatic
Experimentation is a team sport,
the entire product team is on the field
Experiment Review
Optimize for velocity
Welcome to
Experimentation 201
Controlled Experiment Using
Pre-Experimental Data (CUPED)
Can reduce confidence
intervals by 30-60%, resulting
in more statistical power in
less time.
Craig Sexauer https://www.statsig.com/blog/cuped
Problem: The Winner’s Curse
!
Definition
The phenomenon where estimates from AB tests
do not hold up to their expectations.
Problem: The Winner’s Curse
!
Possible Causes
1. Long-term sustainability
2. Underpowered experiments
Actual Effect
Problem: The Winner’s Curse
!
Possible Causes
1. Long-term sustainability
2. Underpowered experiments
3. False positives
No Actual Effect
Problem: The Winner’s Curse
!
Possible Causes
1. Long-term sustainability
2. Underpowered experiments
3. False positives
4. Over-estimations
Underwhelming Effect
Problem: The Winner’s Curse
!
Possible Causes
1. Long-term sustainability
2. Underpowered experiments
3. False positives
4. Over-estimations
5. Biased Decision Making
Negative Effect
Solution: Holdouts
Definition
A small % of users who are intentionally withheld from a feature or
features after rollout, for a longer-than-normal period.
Several Types
? Team-wide
? Feature-specific
? Hypothesis-based
? Powerful
? Deceptively expensive
Problem: The Peeking Problem
!
Solution: Sequential Testing
Tradeoffs
? Statistical Power
? Sensitivity
? Speed
What about multiple metrics?
Maggie Stewart https://www.statsig.com/blog/sequential-testing-on-statsig
Problem: Randomization is Random
$5.78 $2.32
!
Solution: Stratified Sampling
$4.05 $4.05
Solution: Stratified Sampling
B2B Experimentation
? High heterogeneity
? High variance users, by orders of magnitude
? Subgroups are important to track and compare
? Impact on whales are very important to accurately track
? Limited sample size
Problem: Fixed Allocation
Examples
? Holiday Sale periods
? Non-durable goods (eg. news)
? Low statistical power
!
Learning can be expensive—Experiments take awhile to reach “certainty”
Inferior options are given equal traffic for a lengthy period
More variants markedly impact statistical power and experiment duration
Non-stationary effects
Solution: Multiarmed Bandit
Pros
? Automated decision making
? Good in situations with multiple options
? Great at eliminating “bad” options
Cons
? Learning opportunities are limited
? Cannot handle nuanced decision-making
Problem: Network Effects
Experimental groups can affect each other
? Eg. Social networks, two-sided marketplaces,
messaging apps
? Violation of independence assumption
? Cannot accurately measure individual impact
of change, nor project total impact.
!
Solution: Switchback Tests
? Testing the entire network, by
switching states over different
time periods.
? Interval Selection is critical
? Assumes long-term impact and
residual effects are minimal.
Heterogeneous Treatment Effects
Average Treatment Effect vs
Heterogeneous Treatment Effects
Detection
? Hypothesis-driven
? Automation across multiple attributes
Experimental Meta Analysis
Conclusion
Experiments take too long
Winner’s Curse
Peeking Problem
Randomization Sucks
Network Effects
No Average User
Non-generalizable Findings
CUPED
Holdouts
Sequential Testing
Stratified Sampling
Switchback Testing
Heterogeneous Effects Detection
Experimental Meta Analysis
Limitations Solution
?
?
?
?
?
?
?
Statsig.com
Thank you
linkedin.com/in/trchan
tim@statsig.com
@trchan1
statsig.com/pets
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

More Related Content

Similar to Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...
Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...
Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...
April Bright
?
Barga Data Science lecture 10
Barga Data Science lecture 10Barga Data Science lecture 10
Barga Data Science lecture 10
Roger Barga
?
Design testabilty
Design testabiltyDesign testabilty
Design testabilty
Richard Neeve
?
Seriously Advanced A/B Testing by Wyatt Jenkins
Seriously Advanced A/B Testing	by Wyatt JenkinsSeriously Advanced A/B Testing	by Wyatt Jenkins
Seriously Advanced A/B Testing by Wyatt Jenkins
Lean Startup Co.
?
2010 10 15 the lean startup at tech_hub london
2010 10 15 the lean startup at tech_hub london2010 10 15 the lean startup at tech_hub london
2010 10 15 the lean startup at tech_hub london
Eric Ries
?
Optimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and FeedbackOptimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and Feedback
Pronovix
?
Altonix-Presentation Analytics-Powerpoint LinkeDin2
Altonix-Presentation Analytics-Powerpoint LinkeDin2Altonix-Presentation Analytics-Powerpoint LinkeDin2
Altonix-Presentation Analytics-Powerpoint LinkeDin2
Mohsen Khademi
?
Predictive Analytics in Software Testing
Predictive Analytics in Software TestingPredictive Analytics in Software Testing
Predictive Analytics in Software Testing
Pavan Kumar Kodedela
?
AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...
AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...
AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...
Dr. Haxel Consult
?
Webinar: Experimentation & Product Management by Indeed Product Lead
Webinar: Experimentation & Product Management by Indeed Product LeadWebinar: Experimentation & Product Management by Indeed Product Lead
Webinar: Experimentation & Product Management by Indeed Product Lead
Product School
?
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine Learning
Randy Shoup
?
Improving Pharmacy Quality Using Six Sigma
Improving Pharmacy Quality Using Six SigmaImproving Pharmacy Quality Using Six Sigma
Improving Pharmacy Quality Using Six Sigma
John W. Watson
?
Emergency Department Throughput: Using DES as an effective tool for decision ...
Emergency Department Throughput: Using DES as an effective tool for decision ...Emergency Department Throughput: Using DES as an effective tool for decision ...
Emergency Department Throughput: Using DES as an effective tool for decision ...
SIMUL8 Corporation
?
UX research
UX researchUX research
UX research
Billy Choi
?
Statistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreStatistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignore
Turi, Inc.
?
Testing the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesisTesting the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesis
Ardita Karaj
?
QA process Presentation
QA process PresentationQA process Presentation
QA process Presentation
Nadeeshani Aththanagoda
?
DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...
Hakka Labs
?
Eureka Data Science Analytic Process
Eureka Data Science Analytic ProcessEureka Data Science Analytic Process
Eureka Data Science Analytic Process
Allen Nugent
?
A Software Testing Intro
A Software Testing IntroA Software Testing Intro
A Software Testing Intro
Evozon Test Lab
?

Similar to Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know (20)

Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...
Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...
Analyze and Optimize Your Supply Chain Operations for Higher Performance - OM...
?
Barga Data Science lecture 10
Barga Data Science lecture 10Barga Data Science lecture 10
Barga Data Science lecture 10
?
Design testabilty
Design testabiltyDesign testabilty
Design testabilty
?
Seriously Advanced A/B Testing by Wyatt Jenkins
Seriously Advanced A/B Testing	by Wyatt JenkinsSeriously Advanced A/B Testing	by Wyatt Jenkins
Seriously Advanced A/B Testing by Wyatt Jenkins
?
2010 10 15 the lean startup at tech_hub london
2010 10 15 the lean startup at tech_hub london2010 10 15 the lean startup at tech_hub london
2010 10 15 the lean startup at tech_hub london
?
Optimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and FeedbackOptimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and Feedback
?
Altonix-Presentation Analytics-Powerpoint LinkeDin2
Altonix-Presentation Analytics-Powerpoint LinkeDin2Altonix-Presentation Analytics-Powerpoint LinkeDin2
Altonix-Presentation Analytics-Powerpoint LinkeDin2
?
Predictive Analytics in Software Testing
Predictive Analytics in Software TestingPredictive Analytics in Software Testing
Predictive Analytics in Software Testing
?
AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...
AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...
AI-SDV 2022: Possibilities and limitations of AI-boosted multi-categorization...
?
Webinar: Experimentation & Product Management by Indeed Product Lead
Webinar: Experimentation & Product Management by Indeed Product LeadWebinar: Experimentation & Product Management by Indeed Product Lead
Webinar: Experimentation & Product Management by Indeed Product Lead
?
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine Learning
?
Improving Pharmacy Quality Using Six Sigma
Improving Pharmacy Quality Using Six SigmaImproving Pharmacy Quality Using Six Sigma
Improving Pharmacy Quality Using Six Sigma
?
Emergency Department Throughput: Using DES as an effective tool for decision ...
Emergency Department Throughput: Using DES as an effective tool for decision ...Emergency Department Throughput: Using DES as an effective tool for decision ...
Emergency Department Throughput: Using DES as an effective tool for decision ...
?
UX research
UX researchUX research
UX research
?
Statistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreStatistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignore
?
Testing the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesisTesting the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesis
?
QA process Presentation
QA process PresentationQA process Presentation
QA process Presentation
?
DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...
?
Eureka Data Science Analytic Process
Eureka Data Science Analytic ProcessEureka Data Science Analytic Process
Eureka Data Science Analytic Process
?
A Software Testing Intro
A Software Testing IntroA Software Testing Intro
A Software Testing Intro
?

More from Aggregage

Do's and Don'ts: How to Pitch Your Press Release to Journalists
Do's and Don'ts: How to Pitch Your Press Release to JournalistsDo's and Don'ts: How to Pitch Your Press Release to Journalists
Do's and Don'ts: How to Pitch Your Press Release to Journalists
Aggregage
?
Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...
Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...
Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...
Aggregage
?
Your Guide To Transitioning Into Product Management: Essentials For New PMs
Your Guide To Transitioning Into Product Management: Essentials For New PMsYour Guide To Transitioning Into Product Management: Essentials For New PMs
Your Guide To Transitioning Into Product Management: Essentials For New PMs
Aggregage
?
How To Manage Multiple Product Teams For Successful Development & Delivery
How To Manage Multiple Product Teams For Successful Development & DeliveryHow To Manage Multiple Product Teams For Successful Development & Delivery
How To Manage Multiple Product Teams For Successful Development & Delivery
Aggregage
?
How to Optimize Call Monitoring: Automate QA and Elevate Customer Experience
How to Optimize Call Monitoring: Automate QA and Elevate Customer ExperienceHow to Optimize Call Monitoring: Automate QA and Elevate Customer Experience
How to Optimize Call Monitoring: Automate QA and Elevate Customer Experience
Aggregage
?
Scan to Success: How to Leverage QR Codes for Offline and Online Marketing Power
Scan to Success: How to Leverage QR Codes for Offline and Online Marketing PowerScan to Success: How to Leverage QR Codes for Offline and Online Marketing Power
Scan to Success: How to Leverage QR Codes for Offline and Online Marketing Power
Aggregage
?
Product Strategy Agility: How to Use Experiments and Options to Create Produc...
Product Strategy Agility: How to Use Experiments and Options to Create Produc...Product Strategy Agility: How to Use Experiments and Options to Create Produc...
Product Strategy Agility: How to Use Experiments and Options to Create Produc...
Aggregage
?
Leading the Development of Profitable and Sustainable Products
Leading the Development of Profitable and Sustainable ProductsLeading the Development of Profitable and Sustainable Products
Leading the Development of Profitable and Sustainable Products
Aggregage
?
How To Craft Your Perfect Retail Tech Stack
How To Craft Your Perfect Retail Tech StackHow To Craft Your Perfect Retail Tech Stack
How To Craft Your Perfect Retail Tech Stack
Aggregage
?
How To Cultivate Community Affinity Throughout The Generosity Journey
How To Cultivate Community Affinity Throughout The Generosity JourneyHow To Cultivate Community Affinity Throughout The Generosity Journey
How To Cultivate Community Affinity Throughout The Generosity Journey
Aggregage
?
Secrets of a Successful Sale: Optimizing Your Checkout Process
Secrets of a Successful Sale: Optimizing Your Checkout ProcessSecrets of a Successful Sale: Optimizing Your Checkout Process
Secrets of a Successful Sale: Optimizing Your Checkout Process
Aggregage
?
The Rules Do Apply: Navigating HR Compliance
The Rules Do Apply: Navigating HR ComplianceThe Rules Do Apply: Navigating HR Compliance
The Rules Do Apply: Navigating HR Compliance
Aggregage
?
Understanding User Needs and Satisfying Them
Understanding User Needs and Satisfying ThemUnderstanding User Needs and Satisfying Them
Understanding User Needs and Satisfying Them
Aggregage
?
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
?
Unlocking Employee Potential with the Power of Continuous Feedback
Unlocking Employee Potential with the Power of Continuous FeedbackUnlocking Employee Potential with the Power of Continuous Feedback
Unlocking Employee Potential with the Power of Continuous Feedback
Aggregage
?
The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...
The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...
The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...
Aggregage
?
From Awareness to Action: An HR Guide to Making Accessibility Accessible
From Awareness to Action:  An HR Guide to Making Accessibility AccessibleFrom Awareness to Action:  An HR Guide to Making Accessibility Accessible
From Awareness to Action: An HR Guide to Making Accessibility Accessible
Aggregage
?
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
Aggregage
?
How to Leverage Behavioral Science Insights for Direct Mail Success
How to Leverage Behavioral Science Insights for Direct Mail SuccessHow to Leverage Behavioral Science Insights for Direct Mail Success
How to Leverage Behavioral Science Insights for Direct Mail Success
Aggregage
?
Sales & Marketing Alignment_ How to Synergize for Success.pptx.pdf
Sales & Marketing Alignment_ How to Synergize for Success.pptx.pdfSales & Marketing Alignment_ How to Synergize for Success.pptx.pdf
Sales & Marketing Alignment_ How to Synergize for Success.pptx.pdf
Aggregage
?

More from Aggregage (20)

Do's and Don'ts: How to Pitch Your Press Release to Journalists
Do's and Don'ts: How to Pitch Your Press Release to JournalistsDo's and Don'ts: How to Pitch Your Press Release to Journalists
Do's and Don'ts: How to Pitch Your Press Release to Journalists
?
Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...
Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...
Don’t Get Left Behind: Leveraging Modern Product Management Across the Organi...
?
Your Guide To Transitioning Into Product Management: Essentials For New PMs
Your Guide To Transitioning Into Product Management: Essentials For New PMsYour Guide To Transitioning Into Product Management: Essentials For New PMs
Your Guide To Transitioning Into Product Management: Essentials For New PMs
?
How To Manage Multiple Product Teams For Successful Development & Delivery
How To Manage Multiple Product Teams For Successful Development & DeliveryHow To Manage Multiple Product Teams For Successful Development & Delivery
How To Manage Multiple Product Teams For Successful Development & Delivery
?
How to Optimize Call Monitoring: Automate QA and Elevate Customer Experience
How to Optimize Call Monitoring: Automate QA and Elevate Customer ExperienceHow to Optimize Call Monitoring: Automate QA and Elevate Customer Experience
How to Optimize Call Monitoring: Automate QA and Elevate Customer Experience
?
Scan to Success: How to Leverage QR Codes for Offline and Online Marketing Power
Scan to Success: How to Leverage QR Codes for Offline and Online Marketing PowerScan to Success: How to Leverage QR Codes for Offline and Online Marketing Power
Scan to Success: How to Leverage QR Codes for Offline and Online Marketing Power
?
Product Strategy Agility: How to Use Experiments and Options to Create Produc...
Product Strategy Agility: How to Use Experiments and Options to Create Produc...Product Strategy Agility: How to Use Experiments and Options to Create Produc...
Product Strategy Agility: How to Use Experiments and Options to Create Produc...
?
Leading the Development of Profitable and Sustainable Products
Leading the Development of Profitable and Sustainable ProductsLeading the Development of Profitable and Sustainable Products
Leading the Development of Profitable and Sustainable Products
?
How To Craft Your Perfect Retail Tech Stack
How To Craft Your Perfect Retail Tech StackHow To Craft Your Perfect Retail Tech Stack
How To Craft Your Perfect Retail Tech Stack
?
How To Cultivate Community Affinity Throughout The Generosity Journey
How To Cultivate Community Affinity Throughout The Generosity JourneyHow To Cultivate Community Affinity Throughout The Generosity Journey
How To Cultivate Community Affinity Throughout The Generosity Journey
?
Secrets of a Successful Sale: Optimizing Your Checkout Process
Secrets of a Successful Sale: Optimizing Your Checkout ProcessSecrets of a Successful Sale: Optimizing Your Checkout Process
Secrets of a Successful Sale: Optimizing Your Checkout Process
?
The Rules Do Apply: Navigating HR Compliance
The Rules Do Apply: Navigating HR ComplianceThe Rules Do Apply: Navigating HR Compliance
The Rules Do Apply: Navigating HR Compliance
?
Understanding User Needs and Satisfying Them
Understanding User Needs and Satisfying ThemUnderstanding User Needs and Satisfying Them
Understanding User Needs and Satisfying Them
?
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
?
Unlocking Employee Potential with the Power of Continuous Feedback
Unlocking Employee Potential with the Power of Continuous FeedbackUnlocking Employee Potential with the Power of Continuous Feedback
Unlocking Employee Potential with the Power of Continuous Feedback
?
The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...
The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...
The Key to Sustainable Energy Optimization: A Data-Driven Approach for Manufa...
?
From Awareness to Action: An HR Guide to Making Accessibility Accessible
From Awareness to Action:  An HR Guide to Making Accessibility AccessibleFrom Awareness to Action:  An HR Guide to Making Accessibility Accessible
From Awareness to Action: An HR Guide to Making Accessibility Accessible
?
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
?
How to Leverage Behavioral Science Insights for Direct Mail Success
How to Leverage Behavioral Science Insights for Direct Mail SuccessHow to Leverage Behavioral Science Insights for Direct Mail Success
How to Leverage Behavioral Science Insights for Direct Mail Success
?
Sales & Marketing Alignment_ How to Synergize for Success.pptx.pdf
Sales & Marketing Alignment_ How to Synergize for Success.pptx.pdfSales & Marketing Alignment_ How to Synergize for Success.pptx.pdf
Sales & Marketing Alignment_ How to Synergize for Success.pptx.pdf
?

Recently uploaded

Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
dizzycaye
?
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
45unexpected
?
Celonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptxCelonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptx
AnujaGaikwad28
?
MUMBAI MONTHLY RAINFALL CAPSTONE PROJECT
MUMBAI MONTHLY RAINFALL CAPSTONE PROJECTMUMBAI MONTHLY RAINFALL CAPSTONE PROJECT
MUMBAI MONTHLY RAINFALL CAPSTONE PROJECT
GaneshGanesh399816
?
Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
doctorzlife786
?
Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...
6459astrid
?
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
janvikumar4133
?
the unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithmthe unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithm
huseindihon
?
Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...
chetankumar9855
?
Maruti Wagon R on road price in Faridabad - CarDekho
Maruti Wagon R on road price in Faridabad - CarDekhoMaruti Wagon R on road price in Faridabad - CarDekho
Maruti Wagon R on road price in Faridabad - CarDekho
kamli sharma#S10
?
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
gargnatasha985
?
M44.pdf dairy management farm report of an
M44.pdf dairy management farm report of anM44.pdf dairy management farm report of an
M44.pdf dairy management farm report of an
ManjuBv2
?
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
satpalsheravatmumbai
?
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
44annissa
?
?? Premium Girls Call Bangalore ??000XX00000 ? Choose Best And Top Girl Serv...
?? Premium Girls Call Bangalore  ??000XX00000 ? Choose Best And Top Girl Serv...?? Premium Girls Call Bangalore  ??000XX00000 ? Choose Best And Top Girl Serv...
?? Premium Girls Call Bangalore ??000XX00000 ? Choose Best And Top Girl Serv...
bhupeshkumar0889
?
BDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDelivery
erynsouthern
?
BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...
fatima shekh$A17
?
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).docbai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
PhngThLmHnh
?
Nipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma TranscriptNipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma Transcript
zyqedad
?
the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...
huseindihon
?

Recently uploaded (20)

Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
?
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
?
Celonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptxCelonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptx
?
MUMBAI MONTHLY RAINFALL CAPSTONE PROJECT
MUMBAI MONTHLY RAINFALL CAPSTONE PROJECTMUMBAI MONTHLY RAINFALL CAPSTONE PROJECT
MUMBAI MONTHLY RAINFALL CAPSTONE PROJECT
?
Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
?
Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai ??9920725232 ??? Provide Best And Top Girl Ser...
?
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
?
the unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithmthe unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithm
?
Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...
?
Maruti Wagon R on road price in Faridabad - CarDekho
Maruti Wagon R on road price in Faridabad - CarDekhoMaruti Wagon R on road price in Faridabad - CarDekho
Maruti Wagon R on road price in Faridabad - CarDekho
?
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
?
M44.pdf dairy management farm report of an
M44.pdf dairy management farm report of anM44.pdf dairy management farm report of an
M44.pdf dairy management farm report of an
?
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
?
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
?
?? Premium Girls Call Bangalore ??000XX00000 ? Choose Best And Top Girl Serv...
?? Premium Girls Call Bangalore  ??000XX00000 ? Choose Best And Top Girl Serv...?? Premium Girls Call Bangalore  ??000XX00000 ? Choose Best And Top Girl Serv...
?? Premium Girls Call Bangalore ??000XX00000 ? Choose Best And Top Girl Serv...
?
BDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai ? 9820252231 ? Cash Payment With Room DeliveryDelivery
?
BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi ??9711199171 ??? Provide Best And Top Girl Service And ...
?
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).docbai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
?
Nipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma TranscriptNipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma Transcript
?
the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...
?

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

  • 6. statsig.com Statsig is a modern experimentation and feature flagging platform. We help companies like Notion, OpenAI, Figma, and Atlassian manage feature rollouts and compute experimental results. Statsig Cloud ? >200B events a day ? >20k total experiments across >1B unique user identifiers. Statsig Warehouse Native ? Full power of Statsig Cloud but raw data never leaves your data warehouse.
  • 7. Overview Review of Experimentation 101 Experimentation 201 1. CUPED 2. Holdouts 3. The Peeking Problem and Sequential Testing 4. Stratified Sampling 5. Switchback Experiments 6. Multiarmed Bandits 7. Heterogeneous Treatment Effects 8. Experimental Meta Analysis statsig.com 1. AB Testing Basics
  • 8. Experimentation 101: Why A/B Test? Building products is hard Scientific gold standard for measuring causality Ideas are evaluated by causal user data not opinions Product development becomes a scientific, evidence-driven process
  • 9. How Does Testing Work? POPULATION ASSIGNMENT TREATMENT ANALYSIS Control Test 17% 25%
  • 10. ? Start with a hypothesis ? Power Analysis (tradeoff between sample size, statistical power, and time) ? Standardized methodology ? Use 95% confidence intervals by default ? Don’t fret about interaction effects Experimentation Best Practices
  • 12. Stats Engines Don’t Build Culture Experimentation should be easy and automatic Experimentation is a team sport, the entire product team is on the field Experiment Review Optimize for velocity
  • 14. Controlled Experiment Using Pre-Experimental Data (CUPED) Can reduce confidence intervals by 30-60%, resulting in more statistical power in less time. Craig Sexauer https://www.statsig.com/blog/cuped
  • 15. Problem: The Winner’s Curse ! Definition The phenomenon where estimates from AB tests do not hold up to their expectations.
  • 16. Problem: The Winner’s Curse ! Possible Causes 1. Long-term sustainability 2. Underpowered experiments Actual Effect
  • 17. Problem: The Winner’s Curse ! Possible Causes 1. Long-term sustainability 2. Underpowered experiments 3. False positives No Actual Effect
  • 18. Problem: The Winner’s Curse ! Possible Causes 1. Long-term sustainability 2. Underpowered experiments 3. False positives 4. Over-estimations Underwhelming Effect
  • 19. Problem: The Winner’s Curse ! Possible Causes 1. Long-term sustainability 2. Underpowered experiments 3. False positives 4. Over-estimations 5. Biased Decision Making Negative Effect
  • 20. Solution: Holdouts Definition A small % of users who are intentionally withheld from a feature or features after rollout, for a longer-than-normal period. Several Types ? Team-wide ? Feature-specific ? Hypothesis-based ? Powerful ? Deceptively expensive
  • 21. Problem: The Peeking Problem !
  • 22. Solution: Sequential Testing Tradeoffs ? Statistical Power ? Sensitivity ? Speed What about multiple metrics? Maggie Stewart https://www.statsig.com/blog/sequential-testing-on-statsig
  • 23. Problem: Randomization is Random $5.78 $2.32 !
  • 25. Solution: Stratified Sampling B2B Experimentation ? High heterogeneity ? High variance users, by orders of magnitude ? Subgroups are important to track and compare ? Impact on whales are very important to accurately track ? Limited sample size
  • 26. Problem: Fixed Allocation Examples ? Holiday Sale periods ? Non-durable goods (eg. news) ? Low statistical power ! Learning can be expensive—Experiments take awhile to reach “certainty” Inferior options are given equal traffic for a lengthy period More variants markedly impact statistical power and experiment duration Non-stationary effects
  • 27. Solution: Multiarmed Bandit Pros ? Automated decision making ? Good in situations with multiple options ? Great at eliminating “bad” options Cons ? Learning opportunities are limited ? Cannot handle nuanced decision-making
  • 28. Problem: Network Effects Experimental groups can affect each other ? Eg. Social networks, two-sided marketplaces, messaging apps ? Violation of independence assumption ? Cannot accurately measure individual impact of change, nor project total impact. !
  • 29. Solution: Switchback Tests ? Testing the entire network, by switching states over different time periods. ? Interval Selection is critical ? Assumes long-term impact and residual effects are minimal.
  • 30. Heterogeneous Treatment Effects Average Treatment Effect vs Heterogeneous Treatment Effects Detection ? Hypothesis-driven ? Automation across multiple attributes
  • 32. Conclusion Experiments take too long Winner’s Curse Peeking Problem Randomization Sucks Network Effects No Average User Non-generalizable Findings CUPED Holdouts Sequential Testing Stratified Sampling Switchback Testing Heterogeneous Effects Detection Experimental Meta Analysis Limitations Solution ? ? ? ? ? ? ?

Editor's Notes

  1. Statsig was founded 3 years ago… We’re a scrappy but growing team based out of the beautiful Pacific Northwest. We are famous for our 100% in-person office culture. We’re also a dog-friendly office. I’m sitting in the front row with my dog Parker who lucked out and came on a day where we took our company photo.
  2. Statsig is an experimentation and… feature flagging platform that powers companies like Notion, Figma, and OpenAI. We help teams manage their feature rollouts, setup experiments and provide results. In general, we make it easy to be data-driven. We have 2 products Statsig Cloud Process>200B events a day 20k experiments across >1B unique user identifiers. Statsig WHN You get the full power of Statsig Cloud, but the compute happens within our customers’ data warehouses as the raw data never leaves.
  3. In this talk I’ll review what is A/B Testing, why it’s becoming best practices in product development, and how experimentation is the foundation of a data-driven culture. Experimentation 201 - covers the more popular advanced techniques.
  4. Why A/B Test? There are many tools in the Product analytics toolbox, but most of them are correlation-based analyses. Experimentation however measures causation. Experimentation is the basis of the scientific method, and is the gold standard for measuring causality. This is where data-driven decision making starts. If we do A, B will happen and by about X%. Data can trump opinions, but you have to collect it first. Skeptics will criticize this, and say they don’t need AB testing. They hire smart people who have good product sense and intuition. But the truth is, among top tech companies, only 1/3rd of all ideas work (published data). Turns out intuition is often wrong. This is because building products is fundamentally hard. But even if your intuition is good, experimentation still lets you quantify the impact while producing richer insights. If you’re still not convinced, then talk to Sean Taylor who’s here at Data Council…
  5. How it Works You have a heterogeneous population of users, and the first thing you do is is randomly assign the users into two groups Randomization is the secret sauce With enough users, it produces two equal and comparable groups. If your user base contains 10% power users, it will make sure that about half of those are in test and the other half are in control. Same thing all other user traits: Android/iOS splits, new users, and gender. But the really cool thing is that it not only controls for all known confounding variables, it also controls all unknown factors. For example, what if your competitors are experimenting on your users and offering a competing promo? With these groups, we then subject them to two different experiences, called the Test and Control. This is done over the same time period so that seasonal effects affect both equally. Any difference we observe in behavior between the two groups can be attributed to the difference in their experience This is causality, we gave a coupon to one group and not only were first-time purchases up 5%, total monthly revenues were up 2%.
  6. There are a lot of ways to do experimentation wrong Here’s a list of things to watch out for First, you should always start with a hypothesis. You should know what you’re changing and what you expect to happen. Focus on the primary effect, the first observable metric you expect to change. If you shorten your signup flow, maybe you expect more signups to happen. But also ask yourself, what else can happen? More timespent, more invites? What can go wrong? What critical business metrics might change. All of this should be included in your scorecard. Next, you’ll also want to run a power analysis. This let’s you estimate how many users you need to detect the results you expect. This ensures your experiments have a reasonable chance of succeeding. It’ll also guide how many users and for how long you need to run the experiment for. I also recommend standardizing the methodology across your experimentation program. You want to use the same statistics, the same metrics, and the same decision-making framework. This ensures results are comparable between experiments, across teams and people are speaking the same language. I’m a big fan of using 95% confidence intervals by default. While there are certainly reasons to increase or decrease it, unless you’re able to clearly articulate these reasons BEFORE an experiment, stick with 95% please. It is a practical threshold that makes running successful experiments achievable while maintaining a reasonably low false positive rate. Lastly, don’t fret about interaction effects. This is when experiments collide and interfere with each other. There’s research that says interaction effects are fairly rare. And even when they do occur, they often won’t result in different decisions. Unfortunately to eliminate interaction effects, people will often dividing up their user base, or run experiments sequentially. This is poison. This reduces their pace of experimentation, and slows down their rate of innovation.
  7. I want to introduce a character the Hippo. It’s short for Highest paid person’s opinion and it’s how many businesses make decisions. It’s what you do when you aren’t data-driven. I’ve learned that experimentation is great for producing concrete and simple facts like This feature increased retention by 2%. That recommendation model reduced revenue by 3%. It’s hard to ignore data like this because it’s a causal statement. This helps companies become grounded in data.
  8. I’ll talk a lot in the later section about the importance of advanced statistics. But don’t get distracted. It’s far more important to focus on culture rather than fancy stats. Focus should be on the people and processes. It should be trivial for an engineer to set up, execute and analyze a simple AB test. Focus on democratizing data. Everyone on the product team… PMs, Engineers, and Data scientists should be involved. Experiment review is a critical part of a company’s data culture. The scientific method was designed to invite questions. This is where discussions can take place, where assumptions are challenged, and where knowledge and best practices are shared. Lastly, your company should be optimizing for velocity. Find ways to remove friction so that more ideas are tested.
  9. Welcome to experimentation 201 - We’ll cover some of the more popular ways to address the limitations of standard AB testing.
  10. CUPED Popularized in online experimentation by Microsoft (Kohavi paper published in 2013) variance reduction technique Not all variance is purely random. User-level variance comes from pre-existing factors! Example of high variance situations car purchases based on prior purchase Benefits 30-60% reduction in variance across real Statsig experiments Faster experiments, lower sample sizes, more precise decision making Considerations Less effective for new users. Must use user attributes
  11. The next challenge is sometimes referred to as the Winner’s Curse Winner’s in A/B testing don’t live up to the hype. We’ve had a customer who’s old experimentation platform told them they were up 40% on revenue. Great right? But the problem is overall revenue was only up 10%. This is a great way to lose trust in your experimentation tooling.
  12. There are several reasons this may happen: First is long-term sustainability. Are the metric lifts you observe going to hold up over time? Or are these just novelty effects? Run longer or wait for metrics to stabilize Next is underpowered experiments. If you’re short on users and time, you may be tempted to underpower your experiments. This can lead to a large amount of statistical error in your lift estimates. Null hypothesis is in black: this is if there were no effect. We set the threshold for rejecting the null hypothesis at a p-value of 0.05. If you have a good experiment, that’s confidently above this threshold, you’ll get a probabilistic value from this distribution, that's in green. For example…
  13. Another reason is false positives: You don’t actually have an experimental lift and the test group is the same as control. In this case, you have a 5% chance of finding a statistically significant result when there isn’t actually any lift.
  14. Next is over-estimations. If your experimental effect is not quite above the threshold, you can still have a chance of declaring this a winner. This isn’t that bad, it’s still a lift. But you’ll overestimate this for sure.
  15. And lastly, biased decision making. This is human error. We tend to look at results with rose-colored glasses and can sometimes cherry pick results. This means sometimes an experiment is just bad… the results are bad. But we find ways to ignore these, and ship anyways. All of this can lose trust in experimentation. And people can start to game the system.
  16. Solution: Holdouts definition: small subset of users intentionally withheld from a treatment after a full rollout. Typically long-term (>3 months). Usages: Accuracy in long-term measurement Meta notifications holdout. [MORE RESEARCH] Cumulative estimates measurement of a team or an experimentation program. How good are the wins really? Are we making proper decisions? Reroll of randomization. A “second” opinion Can be used in performance reviews, for resourcing, and for keeping teams honest. Also powerful as a debugging tool. I won’t get into this, but sudden outages and metrics movements can be quickly isolated to a set of features using a network of holdouts. Downside: Holdouts are deceptively expensive. Engineering teams have to maintain two branches. And if they don’t do it right, you’ll contaminate the holdout. My advice: Make sure you have top-level buy-in. Make sure you readout holdouts at a fixed cadence (visibility/utility)
  17. Now, let’s talk about the peeking problem The standard hypothesis test is based on statistics that generate a 5% false positive rate. This is based on a single observation, at the full duration of the experiment. Problem is that we’ve been telling PMs and engineers to be monitoring their product dashboards. The data-driven ones want to see experimental results as soon as they launch. Watching how your experiment is going is simply human nature. There are other reasons one may want to peek: Finding wins and locking them in early. Detecting regressions and aborting experiments Finding issues and fixing them. There are practical considerations when trying to solve the peeking problem and all of this makes the stats hard. How often will you peek? What’s the schedule? Are you going to make a decision? How will you ensure you’re optimizing for long-term effects and not just overreacting to novelty effects? How do you adjust for multiple metrics and their tradeoffs?
  18. The solution is called Sequential Testing It’s a generic term for statistical methods that account for continuous or periodic monitoring. There are many methods All of them pose tradeoffs between factors like: Statistical power Sensitivity (can they detect effects early?) Speed We ended up selecting mSPRT after careful evaluation using real data across hundreds of experiments and thousands metrics. We found that 60% of experimental effect were detected at the halfway mark while guaranteeing the false positive rate of 5%. mSPRT has the added advantage of not requiring a tuning parameter. There’s another problem: What if your new ranking model is generating a ton of clicks early and sequential testing says this is statistically significant… should you ship early? Well what about guardrail metrics? What if revenue is between -5% and +5%… is that okay? Our recommendation is to use sequential testing to identify regressions that are worthy of aborting. And wait for the full duration of the experiment to fully evaluate all metrics. This gives you full statistical power across your scorecard. I personally really like model. It’s human nature to be rooting for an experiment, and we are giving you permission to abort early, but you must be patient for the win.
  19. Now let’s talk about randomization Everyone familiar with Canadian coins? Well it’s just like US money, but there’s a $1 coin called a loonie, and a BIG bimetallic coin called the toonie because it’s worth $2. And technically the penny is no longer, but I’ve kept it here for this example. If I take a pile of Canadian coins and randomly split it into two groups, it’s very likely i’ll end up with an imbalance like this. Why? Well some coins are worth orders of magnitude more than other coins, and randomization doesn’t work so well here.
  20. Instead what if we carefully balanced the two groups? We can generate two equivalent groups. This is comparable. This is a real problem in B2B experiments.
  21. B2B typically suffers from skew and small sample sizes. For example, Statsig has skew. If you’re us, what if Atlassian and OpenAI are in the same group? To balance an experiment, we apply a technique called Stratified Sampling. We can further analyze the experiment by subgroups to understand impact across the entire user base.
  22. Another limitation of AB testing is fixed allocation. This is where we run a 50/50 test, and hold that constant. But what if one of your groups is doing better than the other? Don’t you want to shift traffic? And what if time is in really short supply and there’s an urgency to strike while the iron’s hot? This sort of situation happens for things like Black Friday sales where you cannot afford to wait a week to measure the impact… you want to get to the winning variant within hours if not minutes.
  23. One solution is multiarmed bandits The example I’ll use is sporting websites that may want to test 8 different video thumbnails or headlines for last night’s games. They want to converge on the best variant while the game is still recent and people are still interested, rather than a week from now. This example is great, because multiarmed bandits automatically allocate traffic without manual decision making. It’s also great for situations with lots of variants as it will eliminate the poor performers fairly quickly. One major downside though, is that this isn’t great for creating generalized learnings, nor situations where decision-making is complicated and tradeoffs between metrics are unknown.
  24. Network Effects can be a big problem. This is where the test and control groups can affect each other. It violates our assumption of independence. This is most commonly found in two-sided marketplaces like social networks, online marketplaces, and communication platforms. Imagine you’re a ride sharing company and want to test a different rider matching algorithm. This can causes riders in one group to book more rides, depleting the supply of drivers, which negatively impacts the control group. The results you’ll get from such a test, difference between the test and control groups, will not be indicative of what happens when you fully roll out this feature.
  25. Companies like Lyft and Uber pioneered switchback testing as a way to solve this. Instead of splitting your user base 50/50, it switches time blocks randomly. Now the entire ecosystem can have a rider algorithm applied. This sort of testing is also ideal for infra and backend experiments. There are some important considerations here: Picking your time interval is critical. You want frequent switches so you have a lot of observations, but you want a time interval that allows the ecosystem to fully stabilize and be measured.
  26. Finally, let’s talk about heterogeneous effects In AB testing, we measure the average treatment effect. But we all know there’s no such thing as the average user. A major software company shared a story with me of how they ran a signup flow test that generated a small up-lift, say 0.4%. But when they split by gender, they found it was divergent. Interesting… this is still positive though so they still made the right decision. But they then looked prior 10 experiments that were shipped and they were all biased towards men. This was pretty bad because cumulatively, this means their product is now less successful for half the population. The area that this is more common though is looking for technical bugs. What if your new feature doesn’t work on specific browser types? Or Android devices with small screens? To do this right, you’ll want to run automated detection across a set of attributes, correcting for Type I errors due to the multiple comparison problem.
  27. Lastly let’s talk about Experimental Meta Analysis After you’ve ran dozens or hundreds of experiments, you now have a small dataset containing causal observations. There’s a LOT you can do with this and this is an area we’re exploring and don’t have all the answers. But we want to surface data like: relationship between metrics identifying proxy metrics Understanding what movements are possible
  28. Conclusion I did a whirlwind tour through a bunch of solutions to specific problems in AB testing. The key takeaway here is there’s a lot you can do. But I do want to leave you with a lasting thought: While these are really cool, stats engines don’t build culture. I don’t think folks should overthink experimentation, it’s better to run an experiment than to talk yourself out of it.
  29. This concludes my talk. I want to thank you for watching. If you want to hear more, please follow me on LinkedIn. If you want to get in touch, my email and twitter accounts are here as well if you prefer. And if you don’t care about experimentation but like dogs, visit statsig.com/pets Thank you very much.
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news
news