Workflow automation for research teams: provenance and reproducibility

Workflow automation for research teams must record how every result was produced, because a result that nobody can reproduce, including you in two years, is not a result. Provenance (the record of where data came from and what was done to it) is the design constraint, not an afterthought.

An automated pipeline that changes data without recording what it did is worse than a manual process with good notes. The pipeline produces outputs faster and explains them less.

What we'll cover

• Reproducibility is the requirement
• What provenance means in practice
• Where automation hides the method
• Building it with CodeWords
• Data collection and cleaning
• Ethics, consent and sensitive data
• The administrative load
• Collaboration across institutions
• Where research automation carries risk
• Making it survive
• Limits of research automation
• What to build first
• Frequently asked questions

Key takeaways

• Record the data sources, transformations, parameters and environment for every output, so each figure traces back to what produced it.
• Keep raw data immutable and treat cleaning as a separate, versioned step.
• Log every exclusion with a count and a reason, because the number of dropped records is part of the method.
• Stop the pipeline when validation fails, instead of continuing with suspect data.
• Remember that a reproducible result can still be wrong, so keep reviewing whether the method is appropriate.

Reproducibility is the requirement

Somebody else must be able to arrive at the same result from the same inputs.

Including you, later. This is the case that arises most often and the one people underestimate.

The environment counts: versions of tools, libraries and parameters change results in ways that are not obvious.

Manual steps break it. A step somebody performs by hand, even carefully, is the step that cannot be repeated exactly.

Automation helps here. A scripted transformation is reproducible in a way a spreadsheet edit is not, which is the strongest argument for automation in research.

Only if it is recorded. An automated step that is not versioned and logged has traded one reproducibility problem for another.

What provenance means in practice

Where each dataset came from, with a version or a retrieval date, because sources can change without notice.

Every transformation applied, in order, as code rather than as a description.

The parameters used, because a different parameter can make two runs produce different numbers.

The environment, captured, so a rerun uses the same versions instead of whatever is current.

Who ran it and when, which matters for collaboration and for any work that will be examined.

A link from every figure back to what produced it. Reviewers ask for this, and teams that did not record it must reconstruct it.

Where automation hides the method

A pipeline becomes opaque once nobody on the team wrote it.

Silent cleaning is the danger. Dropping rows, filling missing values or excluding outliers inside a script changes the result and leaves no trace in the output. Log every exclusion with a count and a reason.

Defaults are decisions. A library's default handling of missing data is a methodological choice, and somebody should make it deliberately.

Review the pipeline like a method section, because that is what it is.

Write the code to be read. The code is the record of how the work was done and will outlive the person who wrote it.

Building it with CodeWords

CodeWords connects to 3,000+ integrations, and each connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the sources, the transformations, what is logged, and what needs review.
3. Authorize the connections: the data sources, the storage, and wherever outputs are published.
4. Describe the exceptions: a source that changed, a validation failure, a result outside an expected range, an incomplete record.
5. Run it against a dataset you have already analysed and confirm it reproduces the published figures.

You describe the outcome; Cody builds it, connects it, and deploys it. The Free plan covers light use. Pro costs $39 per month ($33 per month billed yearly) and Business costs $100 per month (CodeWords pricing page, 2026); details are on the pricing page.

Data collection and cleaning

Collection is repetitive and error-prone by hand, so it is where automation is least controversial.

Validate on arrival: ranges, types, completeness and duplicates. Report problems instead of correcting them silently.

Keep the raw data immutable, with cleaning as a separate versioned step.

Never overwrite the source. A cleaning step that modifies the source removes the ability to try a different approach later.

Record the cleaning decisions as part of the method, because they are part of it.

Watch for source drift. A feed whose format or meaning changed without notice produces confidently wrong analysis.

Ethics, consent and sensitive data

Approvals govern the data, including how it may be stored, processed and shared, and an automation must operate inside those conditions.

Consent is specific to a purpose, and reusing data for a new question can require going back to participants.

Anonymisation is harder than removing names. Combined fields can re-identify people in rich datasets.

Keep identifiable data separate from analysis data, with the link held under stricter control.

Cross-border transfer is restricted under some data protection rules, which affects where an automation may run. Check the rules that apply to your data.

Check before building. The approval conditions are the specification, not a compliance step afterwards.

The administrative load

Grant reporting against funder categories, tracked as work happens rather than reconstructed.

Deadline tracking across applications, ethics renewals, and reporting obligations.

Reference and citation management, which is administration, not research.

Data management plan compliance, including deposit and retention obligations that are conditions of funding.

Collaboration logistics: scheduling across institutions and time zones, access requests and shared document housekeeping.

None of this work is research, which is why it is worth automating.

Collaboration across institutions

Access is the constraint. Different institutions have different rules about what may be shared and where it may be stored.

Agree the data arrangements first, in writing, because discovering a conflict mid-project is expensive.

Version shared datasets properly. Otherwise two sites analyse slightly different data and spend time reconciling the results.

Automate the routine coordination: status, deadlines and who is waiting on whom. Multi-site projects lose time here.

Keep one canonical protocol that everybody works from.

Expect institutional systems not to connect to each other, and plan for export-based exchange instead of integration.

Where research automation carries risk

Automation makes some practices easy that were once difficult for good reasons. The ease is the problem, not the automation.

Running the analysis repeatedly until something is significant inflates false positives, and it now takes one command.

Trying many specifications without recording how many makes any reported result impossible to interpret.

Scraping at scale can breach terms of service, overload a source, or collect data you have no basis to hold.

Automated literature summarisation can misrepresent a paper, and the errors are hardest to catch in fields you know least well.

Generating figures without inspecting them removes a useful check, because a plot that looks wrong is often the fastest way to catch an error in the pipeline.

Pre-register or log the analysis plan. This is the practical defence against the first two risks and costs almost nothing.

Making it survive

Version everything: code, data, parameters and environment.

Make runs reproducible from a single command. This is the practical test of everything above.

Fail on validation instead of proceeding with suspect data, because a pipeline that continues quietly produces analysis nobody should trust.

Archive outputs with their inputs, so a figure can be traced without rerunning anything.

Report the outcome: runs, validation failures, records excluded with reasons, and sources whose content changed.

Limits of research automation

Automation cannot judge whether a method is appropriate, and that judgement matters most.

Statistical assumptions must be checked by somebody who understands them, not by a pipeline that runs regardless.

Repeated analysis inflates false positives, and an automated pipeline makes repeated analysis trivially easy.

Institutional systems are often closed, so exports are frequently the only route.

Reproducible is not correct. A pipeline can reproduce a flawed result perfectly, which is why the review of the method still matters.

What to build first

The provenance record: for every output, the data sources with versions, the transformations as code, the parameters, the environment, and a link from each figure to what produced it. The record changes no analysis, and it is the difference between a result you can defend and one you can only assert.

Two habits matter most. Keep raw data immutable and treat cleaning as a separate versioned step, because a pipeline that modifies its source removes the ability to try a different approach later. And log every exclusion with a count and a reason, because the number of records dropped is part of the method, and a silent drop inside a script makes an analysis unreproducible without anybody noticing.

Frequently asked questions

Why does automation matter more for reproducibility?

A scripted transformation can be repeated exactly, and a manual one cannot, even when performed carefully. The caveat is that an automated step which is not versioned and logged has moved the reproducibility problem instead of solving it, so record every script, input and parameter.

What does a provenance record need to contain?

A provenance record needs the data sources with versions or retrieval dates, every transformation as code, the parameters used, the captured environment, who ran it and when, and a link from every figure back to what produced it. Together these let someone else repeat the run.

How can automation hide the method?

Automation hides the method through silent cleaning. Dropping rows, filling missing values or excluding outliers inside a script changes results and leaves no trace in the output. Log every exclusion with a count and a reason, so the cleaning stays visible as part of the method.

Should raw data ever be modified?

No. Keep raw data immutable and treat cleaning as a separate versioned step. A pipeline that overwrites its source removes the ability to try a different approach later, and it leaves you unable to show what the data looked like when it arrived.

What should happen when validation fails?

The pipeline should stop and report the failure. Proceeding quietly with suspect data produces analysis that looks complete but cannot be trusted, which is worse than an obvious failure because nobody notices the problem until a reviewer or a later rerun exposes it.

How do consent and ethics approvals affect automation?

Approvals are the specification, not a later compliance step. Consent is specific to a purpose, storage and transfer can be restricted, and identifiable data should be held separately from analysis data under stricter control. Check the approval conditions before you build the pipeline.

Is anonymisation just removing names?

No. Combinations of fields can re-identify people in rich datasets, and the more variables a dataset holds, the harder genuine anonymisation becomes. Removing names is a first step only, so assess the re-identification risk of the whole dataset before you share or reuse it.

Does a reproducible pipeline mean the result is correct?

No. A pipeline can reproduce a flawed result perfectly. Automation removes one class of error, the unrepeatable manual step, and does not replace review of whether the method was appropriate or whether the statistical assumptions hold. Keep a person responsible for that review.

What does automation make too easy in research?

Automation makes it easy to re-run an analysis until something is significant, and to try many specifications without recording how many. Both inflate false positives once the pipeline runs from one command. Logging or pre-registering the analysis plan in advance is the cheap defence.

Is automated scraping a reasonable data collection method?

Only after you check the source's terms, the load your requests place on it, and your basis for holding the data. The ease of collecting data is not a basis for having collected it, so confirm each of those points before scraping at scale.

Related reading

• Automating PDF generation from data
• Workflow automation for education
• Automating user behaviour analytics