Trustworthy Experiments

What it does

Trustworthy Experiments is a framework designed to guide you through running controlled experiments (A/B tests) that yield reliable and actionable results. It addresses common pitfalls in experimentation, aiming to minimize false positives and ensure the validity of findings. The skill draws on industry best practices and insights from experts like Ronny Kohavi, who has identified significant rates of experiment failure and invalid results due to factors like sample ratio mismatch.

When to use it

• Designing A/B tests for product features or improvements.
• Calculating appropriate sample sizes and experiment runtime for statistical power.
• Validating the results of experiments before making decisions about implementation.
• Diagnosing unexpected or suspicious experimental outcomes (Twyman's Law).
• Selecting metrics that balance short-term gains with long-term value (OEC).

Key capabilities

• Provides a framework for designing and executing reliable A/B tests.
• Helps determine appropriate sample sizes and experiment duration.
• Offers guidance on validating experimental results to avoid false positives.
• Addresses the importance of choosing balanced metrics (OEC).
• Supports diagnosing issues with experiment outcomes using Twyman's Law.

Example prompts

• "Help me design an A/B test for [feature name] and determine the necessary sample size."
• "I ran an experiment on [metric], but the results seem suspicious. Can you help me diagnose what might be wrong?"
• "What metrics should I use to evaluate the success of [product change]?"

Tips & gotchas

• Requires a sufficient user base (tens of thousands minimum) for valid results.
• Controlled experiments are not appropriate for one-time decisions or situations lacking user choice.
• Experiments require time; they aren't suitable for immediate decision-making.