How to automate A/B test analysis without manufacturing winners

Automating experiment analysis is straightforward and carries a specific risk: it makes checking a test cheap, and checking repeatedly is the most reliable way to produce a result that is not real.

Stopping the moment a test crosses significance — which is what everybody does when the number is in front of them — inflates the false positive rate substantially above whatever threshold you thought you set. So the design question is not how to compute a p-value on a schedule. It is how to report progress without inviting the decision that invalidates it.

What we'll cover

• The peeking problem
• Decide the horizon before starting
• The checks that come before the result
• Building it with CodeWords
• Guardrail metrics
• Reporting a result honestly
• What a non-result means
• Keeping a record of what was tried
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

The peeking problem

A test's significance threshold assumes one look, at a predetermined point.

Each additional look is another chance to cross it by accident. Check daily for two weeks and the probability of seeing a spurious significant result at some point is far higher than the threshold suggests.

Stopping when it looks good is the specific failure, because you stop on the random high points and continue through the random low ones — which biases every result upward.

Automation makes this worse by putting the number in front of people continuously.

Two honest responses: fix the horizon in advance and do not act before it, or use a method designed for continuous monitoring — sequential testing or a Bayesian approach with a pre-agreed decision rule.

What does not work is looking daily with a fixed-horizon method and promising to be disciplined.

Decide the horizon before starting

Calculate the sample size from the effect you would care about, your baseline rate, and your thresholds — before launching, not after.

An effect too small to act on is not worth detecting. Powering a test to find a half-percent improvement you would never ship is an expensive way to learn nothing.

Convert it to a duration, and round up to whole weeks — because day of week affects behaviour substantially and a partial week biases the result.

Write the horizon and the decision rule down, before the test starts, where everybody can see it.

Run at least one full cycle of whatever periodicity your business has. Two weeks is the common minimum for that reason rather than by convention.

Then do not stop early except on a guardrail breach, which is covered below.

The checks that come before the result

Run these first, every time, and report a failure loudly rather than reporting a result.

Sample ratio mismatch. If you assigned fifty-fifty and observe fifty-two to forty-eight on large numbers, something is broken in assignment or tracking, and the result cannot be trusted regardless of how good it looks. This is the single most important check and the least implemented.

Assignment consistency. Somebody seeing both variants pollutes both arms — usually a logged-out, cross-device, or caching problem.

Tracking parity. If one variant is losing events to an ad blocker or a slower page, its apparent performance is measurement rather than behaviour.

Novelty and primacy. A change performing spectacularly in the first two days and returning to baseline is a reaction to newness, not an improvement.

Contamination. Other experiments or a marketing campaign running simultaneously on the same population.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the experiment data, the guardrails, and the report can be joined without anybody exporting anything.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which experiments, which metrics, and what should be reported when.
3. Authorize the connections — the experiment data, analytics, and wherever the report goes.
4. Describe the exceptions: a sample ratio mismatch, a guardrail breach, a test with too little data to report.
5. Run it against a completed experiment whose outcome you already know.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Guardrail metrics

The one legitimate reason to look early, and the one thing worth alerting on continuously.

Guardrails are things that must not get worse — errors, latency, checkout completion, unsubscribes — regardless of what the primary metric does.

Monitor them continuously and alert on breach, which is a different activity from evaluating the result.

Stop the test on a serious breach. A variant causing errors should not run to its horizon for statistical tidiness.

Set them per experiment, including anything the change could plausibly harm rather than a generic list.

Report them alongside the result, always, because a win on the primary metric that cost something on a guardrail is a trade rather than a victory.

This is the only early-stopping rule that does not bias the result, because it is not a decision about whether the change worked.

Reporting a result honestly

State the effect size with an interval, not just significance. "Between one and four per cent better" is information; "significant" is not.

Give the absolute alongside the relative. A forty per cent improvement on a two per cent baseline is eight tenths of a percentage point.

Report the guardrails, every time.

Say what the test could not detect. A test powered for a five per cent effect that returned nothing has not shown the change does nothing.

Name the segments you did not analyse. Slicing after the fact until something is significant is the other common way to manufacture a result, and pre-registering the segments you care about is the defence.

Include the decision rule you set in advance, so the reader can see the result is being judged against the rule rather than the reverse.

What a non-result means

The most common outcome, and the most commonly mishandled.

Most tests do not produce a winner, which is normal and is not a failed experiment.

"No significant difference" is not "no difference". It usually means the effect, if any, is smaller than the test could detect.

That is still useful. Knowing a change does not matter much frees you to choose on other grounds — simplicity, maintenance, or preference.

Do not re-run hoping for a better result. Repeating a test until it wins is peeking with extra steps.

Do not slice until something is significant. If you must explore segments, say clearly that it is exploratory and requires its own test to confirm.

Record it. A register of what was tried and did not work is one of the more valuable artefacts a team can have and almost nobody keeps one.

Keeping a record of what was tried

The artefact that compounds, and almost nobody maintains one.

Register every test: the hypothesis, the horizon, the decision rule, the result, and what was done about it.

Record the failures particularly. A team that does not know what has already been tried retries it, which is one of the more expensive forms of institutional forgetting.

Note what the test could not detect, so a future "we tested that" can be checked against what the test was actually powered for.

Link the analysis, so somebody can see how the conclusion was reached rather than taking it on trust.

Review the register periodically. A pattern of tests in one area producing nothing is itself a finding about where the leverage is not.

Keep it where product decisions are made, not in the experimentation tool, since the audience is the people choosing what to try next.

Making it survive

Run the validity checks before computing anything, and refuse to report a result when one fails.

Suppress the result until the horizon, showing progress and guardrails only. This is the design decision that prevents peeking rather than discouraging it.

Handle the experiment with too little data by saying so rather than reporting a wide, meaningless interval.

Keep the analysis code in version control, since it encodes statistical decisions that deserve review.

Report the outcome. Experiments running, days remaining, guardrail status, and any validity check that failed.

Limits worth knowing about

Assignment and analysis must agree on the unit — user, session, or account — and a mismatch invalidates everything quietly.

Small samples cannot detect small effects, and no analysis method changes that.

Multiple simultaneous tests on one population interact, and the interaction is rarely accounted for.

Segment analysis after the fact multiplies the false positive rate, however tempting the segment looks.

Statistical significance is not business significance. A real and tiny improvement may not justify the maintenance cost of the change that produced it.

What to build first

The sample ratio mismatch check, running on every experiment, blocking the result when it fails. It is a few lines of arithmetic, it is the most reliable indicator that an experiment is broken, and it is almost never implemented — which means teams routinely act on results from tests where assignment or tracking was faulty.

Two habits make the difference. Show progress and guardrails during the test but suppress the result until the horizon, since designing against peeking works and asking people to resist it does not. And report the effect size with an interval rather than significance alone, because "between one and four per cent" supports a decision and "significant" invites one.

Frequently asked questions

Why is checking the result daily a problem?

Because each look is another chance to cross the threshold by accident, and people stop on random high points while continuing through low ones. It inflates false positives well above the threshold you thought you set.

Can I look at the test while it runs?

At the guardrails, yes, and you should. At the result, no — unless you are using a method designed for continuous monitoring with a pre-agreed rule. Showing progress without the result is the practical compromise.

What is sample ratio mismatch?

Observed traffic not matching the split you assigned. It means assignment or tracking is broken, it invalidates the result regardless of how good it looks, and it is the most important check and the least implemented.

How long should a test run?

To a sample size calculated in advance from the effect you would actually act on, rounded up to whole weeks, covering at least one full cycle of your business's periodicity. Not "until it reaches significance".

What does "no significant difference" mean?

Usually that the effect is smaller than the test could detect, not that there is none. That is still useful — it frees you to choose on simplicity or maintenance grounds — and it is not a reason to re-run.

Is it acceptable to analyse segments?

Only if you pre-registered them. Slicing after the fact until something is significant multiplies the false positive rate, and any interesting segment found that way needs its own test to confirm.

What should a result report contain?

Effect size with an interval, the absolute alongside the relative, guardrail status, what the test could not detect, and the decision rule set in advance — so the result is judged against the rule rather than the other way round.

Why keep a register of experiments?

Because teams retry what they have already tried. Record the hypothesis, horizon, decision rule, result, and what the test could not detect — the last one lets a future "we tested that" be checked against what was actually measurable.

What does a run of null results tell you?

Often that the leverage is not in that area. A pattern across the register is a finding in its own right, and it is only visible if somebody has been recording the tests that did not produce a winner.

Related reading

• Google Analytics and CodeWords
• Mixpanel and CodeWords
• Automating report generation