Ab Test Calculator

The ab-test-calculator skill enables AI agents to perform statistical analysis for A/B testing scenarios. It calculates key metrics such as sample size, conversion rates, and statistical significance based on user-provided data points.

When to use it

• Determine the required sample size needed to detect a specific lift in conversion rates with a defined confidence level.
• Analyze current test results to calculate the probability that one variant outperforms another statistically.
• Estimate the minimum detectable effect (MDE) for an experiment given a fixed budget or timeline.
• Validate whether observed differences between control and treatment groups are significant enough to drive decisions.

Key capabilities

• Computes sample size requirements based on baseline conversion rates and desired statistical power.
• Calculates p-values and confidence intervals for A/B test comparisons.
• Determines the minimum detectable effect (MDE) required to achieve significance within a set timeframe.
• Processes inputs including baseline rate, lift percentage, confidence level, and power.

Example prompts

• "Calculate the sample size needed to detect a 5% increase in conversion rate with 95% confidence and 80% power, assuming a baseline of 10%."
• "Analyze these A/B test results: Control had 20 conversions out of 1000 visits, Treatment had 24 out of 1000. Is the difference statistically significant?"
• "What is the minimum detectable effect for a test running for 7 days with 500 users per variant and a baseline conversion rate of 2%?"

Tips & gotchas

Ensure you provide accurate baseline conversion rates, as errors here propagate through all subsequent calculations. This tool assumes independent Bernoulli trials and may not account for complex dependencies like seasonality or user cohort effects without manual adjustment.