Exact A/B test statistics for AI assistants: sample size for two proportions, test duration, two-proportion z-test and chi-square, Fisher exact and Fisher-Freeman-Halton r x c tests, and the real false positive rate of peeking. Version 1.0.0.
calculate_sample_size | Users per variant to detect a lift in a conversion rate: baseline, absolute or relative MDE, alpha, power, one or two-sided, multiple variants with Bonferroni, unequal splits. Also shows the Evan Miller figure. |
estimate_test_duration | Days and full weeks an A/B test must run given daily traffic, the share of traffic in the test and the number of variants. |
test_ab_significance | Two-proportion z-test, chi-square (with and without Yates), Fisher exact, confidence interval, relative lift and a sample ratio mismatch check. |
fisher_exact_test | Exact p-value for a 2x2 table (Fisher, one or two-sided) or any r x c table (Fisher-Freeman-Halton), next to Pearson chi-square and an expected-count check. |
check_peeking_risk | Real false positive rate when you check a test several times, and Pocock and O'Brien-Fleming stopping boundaries for up to 10 looks. |
Hosted endpoint (Streamable HTTP, no token, no sign-up). Paste it into a Claude.ai custom connector or ChatGPT developer mode:
https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp
Claude Code:
claude mcp add --transport http ab-test-calculator https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp
Local stdio (Node 20+):
npx -y github:theluckystrike/ab-test-sample-size-calculator-mcp
Source and install notes for Cursor and VS Code: github.com/theluckystrike/ab-test-sample-size-calculator-mcp
Interactive calculators (sample size, Bayesian A/B, CUPED, sequential testing) are on abwex.com (Fisher-Freeman-Halton calculator).
Part of the Zovo MCP collection: https://mcp.zovo.one