Workflow automation for research teams: provenance and reproducibility

Automation in research has an unusual requirement attached to it. In most functions, a process that produces the right answer is good enough. In research, a result that cannot be reproduced — by somebody else, or by you in two years — is not a result, whatever number it produced.

That makes provenance the design constraint rather than an afterthought. An automated pipeline that transforms data without recording what it did is worse than a manual process with good notes, because it produces outputs faster and explains them less.

What we'll cover

• Reproducibility is the requirement
• What provenance means in practice
• Where automation hides the method
• Building it with CodeWords
• Data collection and cleaning
• Ethics, consent and sensitive data
• The administrative load
• Collaboration across institutions
• Where research automation is genuinely risky
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Reproducibility is the requirement

Somebody else must be able to arrive at the same result from the same inputs.

Including you, later, which is the case that actually arises and the one people underestimate.

The environment counts: versions of tools, libraries and parameters change results in ways that are not obvious.

Manual steps break it. A step somebody performs by hand, even carefully, is the step that cannot be repeated exactly.

Automation helps here, which is the strongest argument for it in this setting — a scripted transformation is reproducible in a way a spreadsheet edit is not.

Provided it is recorded. An automated step that is not versioned and logged has traded one reproducibility problem for another.

What provenance means in practice

Where each dataset came from, with a version or a retrieval date, since sources change without notice.

Every transformation applied, in order, as code rather than as a description.

The parameters used, which are frequently the difference between two runs producing different numbers.

The environment, captured, so a rerun uses the same versions rather than whatever is current.

Who ran it and when, which matters for collaboration and for anything that will be examined.

A link from every figure back to what produced it, which is the thing reviewers ask for and teams scramble to reconstruct.

Where automation hides the method

A pipeline becomes opaque once nobody currently present wrote it.

Silent cleaning is the danger. Dropping rows, filling missing values or excluding outliers inside a script changes the result and leaves no trace in the output.

Log every exclusion with a count and a reason, since the number excluded is part of the method.

Defaults are decisions. A library's default handling of missing data is a methodological choice somebody should have made deliberately.

Review the pipeline like a method section, because that is what it is.

Write it to be read, since the code is the record of how the work was done and will outlive the person who wrote it.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the sources, the transformations, what is logged, and what needs review.
3. Authorize the connections — the data sources, the storage, and wherever outputs are published.
4. Describe the exceptions: a source that changed, a validation failure, a result outside an expected range, an incomplete record.
5. Run it against a dataset you have already analysed and confirm it reproduces the published figures.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Data collection and cleaning

Collection is repetitive and error-prone by hand, and it is where automation is least controversial.

Validate on arrival: ranges, types, completeness, duplicates — and report rather than correcting silently.

Keep the raw data immutable, always, with cleaning as a separate versioned step.

Never overwrite. A cleaning step that modifies the source destroys the ability to try a different approach later.

Record the cleaning decisions as part of the method, since they are.

Watch for source drift. A feed whose format or meaning changed quietly is the failure that produces confidently wrong analysis.

Ethics, consent and sensitive data

Approvals govern the data, including how it may be stored, processed and shared, and an automation must operate inside them.

Consent is specific to a purpose, and reusing data for a new question may require going back.

Anonymisation is harder than removing names, and re-identification through combined fields is a real risk with rich datasets.

Keep identifiable data separate from analysis data, with the link held under stricter control.

Cross-border transfer is constrained in several regimes, which affects where an automation may run.

Check before building. The approval conditions are the specification here, not a compliance step afterwards.

The administrative load

Grant reporting against funder categories, tracked as work happens rather than reconstructed.

Deadline tracking across applications, ethics renewals, and reporting obligations.

Reference and citation management, which is pure administration.

Data management plan compliance, including deposit and retention obligations that are conditions of funding.

Collaboration logistics — scheduling across institutions and time zones, access requests, shared document housekeeping.

None of it is research, which is why it is worth removing.

Collaboration across institutions

Access is the constraint. Different institutions have different rules about what may be shared and where it may be stored.

Agree the data arrangements first, in writing, since discovering a conflict mid-project is expensive.

Version shared datasets properly, or two sites analyse slightly different data and spend a month reconciling.

Automate the boring coordination — status, deadlines, who is waiting on whom — which is where multi-site projects lose time.

Keep one canonical protocol that everybody works from.

Expect institutional systems not to talk to each other, and plan for export-based exchange rather than integration.

Where research automation is genuinely risky

Some things automation makes easy are things that were previously difficult for good reasons, and the ease is the problem rather than the automation.

Running the analysis repeatedly until something is significant, which inflates false positives and is now a matter of re-running a command.

Trying many specifications without recording how many, which makes any reported result impossible to interpret.

Scraping at scale, which may breach terms of service, overload a source, or collect data you have no basis to hold.

Automated literature summarisation, which misrepresents with some regularity and is hardest to check in fields you know least well.

Generating figures without inspecting them, since a plot that looks wrong is frequently the fastest way to catch an error in the pipeline.

Pre-register or log the analysis plan, which is the practical defence against the first two and costs almost nothing.

Making it survive

Version everything: code, data, parameters and environment.

Make runs reproducible from a single command, which is the practical test of everything above.

Fail on validation rather than proceeding with suspect data, since a pipeline that continues quietly produces analysis nobody should trust.

Archive outputs with their inputs, so a figure can be traced without rerunning anything.

Report the outcome: runs, validation failures, records excluded with reasons, and sources whose content changed.

Limits worth knowing about

Automation cannot judge whether a method is appropriate, which is the part that matters most.

Statistical assumptions must be checked by somebody who understands them, not by a pipeline that runs regardless.

Repeated analysis inflates false positives, and an automated pipeline makes repeated analysis trivially easy.

Institutional systems are frequently closed, so exports are often the only route.

Reproducible is not correct. A pipeline can reproduce a flawed result perfectly, which is why the review of the method still matters.

What to build first

The provenance record: for every output, the data sources with versions, the transformations as code, the parameters, the environment, and a link from each figure to what produced it. It changes no analysis and it is the difference between a result you can defend and one you can only assert.

Two habits make the difference. Keep raw data immutable and treat cleaning as a separate versioned step, since a pipeline that modifies its source removes the ability to try a different approach later. And log every exclusion with a count and a reason, because the number of records dropped is part of the method and a silent drop inside a script is how an analysis becomes unreproducible without anybody noticing.

Frequently asked questions

Why does automation matter more for reproducibility?

Because a scripted transformation is repeatable and a manual one is not, even when performed carefully. The caveat is that an automated step which is not versioned and logged has simply moved the reproducibility problem rather than solved it.

What does a provenance record need to contain?

Data sources with versions or retrieval dates, every transformation as code, the parameters used, the captured environment, who ran it and when, and a link from every figure back to what produced it.

How can automation hide the method?

Through silent cleaning. Dropping rows, filling missing values or excluding outliers inside a script changes results and leaves no trace in the output. Log every exclusion with a count and a reason.

Should raw data ever be modified?

No. Keep it immutable and treat cleaning as a separate versioned step, because a pipeline that overwrites its source removes the ability to try a different approach later.

What should happen when validation fails?

The pipeline should stop. Proceeding quietly with suspect data produces analysis that looks complete and should not be trusted, which is worse than an obvious failure.

How do consent and ethics approvals affect automation?

They are the specification rather than a later compliance step. Consent is specific to a purpose, storage and transfer are constrained, and identifiable data should be held separately from analysis data under stricter control.

Is anonymisation just removing names?

No. Re-identification through combinations of fields is a real risk with rich datasets, and the more variables a dataset holds the harder genuine anonymisation becomes.

Does a reproducible pipeline mean the result is correct?

No. A pipeline can reproduce a flawed result perfectly. Automation removes one class of error and does not substitute for review of whether the method was appropriate.

What does automation make too easy in research?

Re-running an analysis until something is significant, and trying many specifications without recording how many. Both inflate false positives, and both become trivial once the pipeline runs from one command. Logging the analysis plan in advance is the cheap defence.

Is automated scraping a reasonable data collection method?

Only with the terms, the load on the source, and your basis for holding the data all checked first. The ease of collecting is not a basis for having collected it.

Related reading

• Automating PDF generation from data
• Workflow automation for education
• Automating user behaviour analytics