Workflow automation for growth teams: automating what you are testing

Growth teams automate faster than anybody else in a company, because the work is repetitive and the results are measured. That speed produces a specific failure: a test that looked promising gets automated, becomes part of how the product works, and nobody ever establishes whether it did anything.

Six months later the team is maintaining fourteen automations, several of which are doing nothing, one of which is actively harmful, and none of which anybody wants to turn off because nobody can say what would happen.

What we'll cover

• Automating a test makes it permanent
• Instrumentation before automation
• Holdouts, which are not optional
• Building it with CodeWords
• What is worth automating
• Reading a result honestly
• Retiring what stops working
• The limits of what growth can automate
• Working with the teams whose systems you are touching
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Automating a test makes it permanent

A manual experiment ends by default. Somebody stops doing it.

An automated one continues by default, which inverts the burden of proof.

Nobody turns off something that might be working, and without a measurement nobody can say it is not.

The stack accumulates. Each automation is small and the maintenance is collective.

Interactions appear. Fourteen automations touching the same users produce effects none of them was tested for.

Decide the retirement condition before building it, which is the only reliable prevention.

Instrumentation before automation

Build the measurement first, always, and confirm it records what you think.

Define the success metric in advance, precisely, including the window.

Check the baseline. A surprising number of experiments are compared against a figure nobody validated.

Make the metric reachable. If measuring the outcome requires a quarter, the experiment is a quarter long and should be planned as one.

Instrument the counterfactual too, meaning you can identify who did not receive the treatment.

No instrumentation, no automation. It is a rule that costs a day and prevents most of what goes wrong here.

Holdouts, which are not optional

Hold back a proportion permanently, not just during the test.

Platform and tool attribution overstates, universally, and a holdout is the only correction available.

People who would have converted anyway are the largest category in most growth measurements.

A long-running holdout answers decay, which is the question nobody asks — whether something that worked still works.

Small holdouts are cheap. A few per cent costs little and is the only evidence you will ever have.

Report against the holdout, not against the platform's number, in everything that reaches a decision-maker.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the trigger, the treatment, the holdout, and what is measured.
3. Authorize the connections — the product analytics, the messaging platform, the CRM and the data warehouse.
4. Describe the exceptions: a user in the holdout, somebody recently contacted, an account in an active sales process, a user who opted out.
5. Run it with the treatment disabled for a week first and confirm the measurement records what you expect.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What is worth automating

The experiment mechanics: assignment, delivery, measurement and reporting, which is the repetitive part.

Data assembly, so the analysis starts from a table rather than from four exports.

Alerting on a result reaching significance, or on a guard metric moving the wrong way.

The lifecycle triggers that already proved themselves against a holdout.

Reporting to the rest of the company, since growth work is invisible otherwise and invisible work gets deprioritised.

Not the idea generation, which is where the value is and which is not a throughput problem.

Reading a result honestly

Peeking inflates false positives. Deciding to stop when the numbers look good is the most common error in this work.

Set the duration in advance and respect weekly cycles, since a test that ends on a Tuesday measures a different population from one that ran two full weeks.

Segment effects are usually noise unless the segment was specified beforehand.

Watch the guard metrics, since a lift in one number bought with a loss in another is a trade rather than a win.

Most experiments do not work, and a programme reporting mostly wins is measuring badly rather than performing well.

Record the ones that failed somewhere findable, or the same idea returns every eighteen months.

Retiring what stops working

Effects decay. An audience habituates, a channel saturates, a competitor copies it.

Re-test periodically rather than assuming a past result holds.

Run a reverse test for anything long-running and significant: turn it off for a segment and see whether anything happens.

Report the age of every automation alongside its last validated result.

Turning something off is a legitimate outcome and should be as reportable as launching something.

A stack that only grows is not a growth programme, it is an accumulation.

The limits of what growth can automate

Automation increases throughput of a process, which helps when the process is the constraint and not otherwise.

A weak product is not a growth problem. More traffic through a funnel that does not convert produces more people who did not convert.

Positioning is not testable at the margin, and a series of small experiments will not find it.

Retention is mostly product, and lifecycle messaging influences it at the edges rather than determining it.

Channel saturation is real, and automating more volume into a saturated channel accelerates its decline.

Know which constraint you are working against, since automating around the wrong one is the most expensive mistake available here.

Working with the teams whose systems you are touching

Growth automation reaches into the product, the CRM and the email platform, none of which growth usually owns. How that relationship is handled decides how much you can actually ship.

Agree the boundaries once — which surfaces growth can change, which need review, which are off limits — rather than negotiating per experiment.

Do not write to production data casually. A growth experiment mutating customer records is the thing that ends the arrangement.

Use the same suppression and consent rules as everybody else, since an exemption granted to move fast is the exemption that produces a complaint.

Tell support what is running. An experiment nobody warned them about arrives as a confusing ticket they cannot explain.

Share the results back, including the failures, which is what makes the next request easier.

Clean up after yourself. Abandoned experiment code and orphaned fields accumulate in systems other people maintain, and that bill arrives as reluctance.

Making it survive

Make assignment sticky. A user moving between treatment and control invalidates the measurement.

Log every assignment and exposure, since post-hoc reconstruction is not reliable.

Respect suppression and frequency caps shared with the rest of the company.

Make it reversible. Anything you cannot turn off quickly should not go out.

Report the outcome: experiments running, their age, results against holdout, and what has not been validated recently.

Limits worth knowing about

Sample size limits what you can detect. Small user bases cannot resolve small effects, which is a mathematical fact rather than a methodological one.

Novelty effects fade, so an early result overstates.

Network effects break assignment, since treated users influence untreated ones.

Seasonality confounds anything compared across periods.

Tooling assumptions vary, and the statistics an experiment platform reports rest on assumptions worth reading once.

What to build first

The instrumentation and the holdout, with the treatment switched off. A week of that confirms the measurement works and gives you the baseline, and it costs a fraction of what re-running an inconclusive experiment costs.

Two habits make the difference. Decide the retirement condition before building anything, since an automation continues by default and nobody will ever turn off something that might be working. And hold back a permanent control group, because platform attribution overstates everywhere and the holdout is the only evidence you will ever have that any of this did something.

Frequently asked questions

Why is automating an experiment risky?

Because a manual experiment ends by default and an automated one continues by default. Without a measurement nobody can say it is not working, so nobody turns it off, and the stack accumulates interactions none of them was tested for.

What should be built before the automation?

The instrumentation, with the success metric and its window defined in advance, the baseline validated, and the ability to identify who did not receive the treatment.

Why are holdouts necessary?

Because platform attribution overstates universally, and people who would have converted anyway are the largest category in most growth measurements. A permanent holdout is also the only way to detect decay.

What is the most common analysis error?

Peeking — deciding to stop when the numbers look good. Set the duration in advance and respect weekly cycles, since a test ending mid-week measures a different population.

Should a growth programme report mostly wins?

No. Most experiments do not work, and a programme reporting mostly wins is measuring badly rather than performing well. Record the failures somewhere findable or the same idea returns every eighteen months.

How do I know whether something still works?

Re-test, or run a reverse test: turn it off for a segment and see whether anything happens. Effects decay as audiences habituate and channels saturate, and a past result is not evidence about the present.

What can automation not fix?

A weak product, positioning, or a saturated channel. More traffic through a funnel that does not convert produces more people who did not convert, and automating volume into a saturated channel accelerates its decline.

When can small teams not run experiments?

When the sample cannot resolve the effect size. That is arithmetic rather than methodology, and the right response is to make larger changes and measure them qualitatively rather than to run underpowered tests.

How do growth teams keep access to other teams' systems?

By agreeing boundaries once rather than per experiment, using the same suppression and consent rules as everybody else, telling support what is running, sharing failures as well as wins, and cleaning up abandoned experiment code.

Why does support need to know about experiments?

Because an experiment nobody warned them about arrives as a confusing ticket they cannot explain, and that experience is what turns other teams against the arrangement.

How long should a holdout run?

Permanently for anything you intend to keep. A holdout that ends with the experiment answers whether something worked once; a standing one answers whether it still works, which is the question nobody asks.

What is the cost of a small holdout?

A few per cent of treated users, which is almost always less than the cost of maintaining an automation that is doing nothing and that nobody can justify turning off.

Related reading

• Automating user behaviour analytics
• Automating lead scoring
• Workflow automation for product teams