What is data orchestration, and why does a scheduler stop being enough?

Data orchestration is coordinating the jobs that move and transform data, so each runs when what it depends on is ready. That last clause is the whole subject, because the alternative — running things at times chosen to be probably late enough — is what everybody starts with and what eventually stops working.

The transition is recognisable. Somebody adds fifteen minutes to a job's schedule because it sometimes started before its input was ready, and a few months later the schedule is a set of superstitions nobody can safely change.

What we'll cover

• Scheduling and orchestrating
• Where cron stops working
• What a dependency graph buys
• Task-based and asset-based
• Backfills and reruns
• A job ran is not data is ready
• Who gets told when it breaks
• Where the pipeline code lives
• Orchestrating with CodeWords
• What to get right first
• Frequently asked questions

Scheduling and orchestrating

A scheduler runs things at times. Cron is a scheduler: at this minute, run this.

An orchestrator runs things in an order, with each step waiting for what it needs, and the time is only the trigger for the first one.

The difference shows up on a slow day. A scheduler runs the downstream job whether or not the upstream one finished; an orchestrator waits.

Retries differ too. A scheduler retries the whole thing or nothing; an orchestrator retries the failed step and continues from there.

So does visibility. A scheduler knows a command exited; an orchestrator knows which step of which pipeline failed and what is blocked behind it.

Where cron stops working

The progression is consistent, and recognising which stage you are at is useful.

One job, one schedule. Fine, indefinitely.

A second job needing the first's output, scheduled to run half an hour later because that is usually long enough.

A slow day, and the second job reads yesterday's data and produces a report nobody notices is wrong.

Padding accumulates. Each job's schedule grows a margin, and the whole pipeline finishes hours after it could.

A failure in the middle leaves everything downstream running against stale inputs, all of it succeeding.

Nobody dares change the timings, because the reasoning behind each offset is lost.

At that point the schedules encode dependencies implicitly, badly, and the fix is to state them explicitly.

What a dependency graph buys

Correct ordering by construction. A task runs when its dependencies have succeeded, not when the clock says.

Parallelism for free. Independent branches run simultaneously without anybody coordinating it, which usually shortens the whole pipeline considerably.

Failure that propagates sensibly. A failed task blocks what depends on it and does not block what does not.

Partial recovery. Fix the failure, rerun from that task, and everything downstream follows — rather than rerunning from the start.

A picture somebody can read. The graph is documentation of how the data actually flows, which is frequently the first time anybody has had one.

The cost is defining the dependencies, which is real work and is work that surfaces relationships nobody had written down.

Task-based and asset-based

Two ways of expressing the same thing, and the distinction affects how you think.

Task-based orchestration defines operations and their order. Run this, then that. The graph is a graph of work.

Asset-based orchestration defines the data that should exist and how each piece is produced. The graph is a graph of tables and files, and the work is derived.

Asset-based suits data work better in most cases, because the question people ask is "is the customer table up to date", not "did job seventeen run".

It makes freshness a first-class concept, since each asset has a state and a last-materialised time rather than only a job history.

Task-based is more general, which is why it remains common and why it suits pipelines that are not purely about producing data.

Backfills and reruns

The capability that most justifies an orchestrator, and the one people discover they need urgently.

Reprocessing a date range is normal — a transform bug, a source that was wrong for a week, a new column.

Parameterise by date from the beginning. A pipeline hard-coded to "yesterday" cannot be backfilled without editing it, which is exactly the wrong thing to be doing under pressure.

Make tasks idempotent, so rerunning a date produces the same result rather than duplicating.

Run backfills separately from the scheduled pipeline, so a long one does not delay tonight's run.

Limit concurrency deliberately. A backfill running two years of dates in parallel will overwhelm whatever it reads from.

Record what was reprocessed. Figures that change retrospectively look like a broken pipeline to anybody holding last week's number.

A job ran is not data is ready

The distinction that separates orchestration from data reliability.

A successful task means the code did not error. It does not mean the output is correct, complete, or non-empty.

Check the output, not the exit code. Row counts, freshness, null rates on columns that should not be null, and referential integrity between tables.

Make the checks part of the graph, so a failed check blocks downstream work rather than being a separate report nobody reads.

An empty result is the dangerous case, because it usually succeeds. A source that returned nothing produces a downstream table of zeros and a dashboard that quietly says business stopped.

Publish freshness. Consumers of a table should be able to see how current it is without asking, which removes a recurring question and a class of wrong decisions.

Who gets told when it breaks

Alert the owner of the asset, not a central channel. A pipeline failure in a shared channel belongs to nobody.

Say what is affected downstream. "The revenue table failed" is less useful than naming the three dashboards now stale.

Distinguish a failure from a delay. A task retrying is not the same as a pipeline that has given up, and alerting identically on both trains people to ignore the first.

Tell the consumers, not only the engineers. Whoever reads the dashboard should know the data is stale before they act on it, which is the part almost nobody builds.

Report what is stale, on a schedule, rather than only alerting at the moment of failure — because the failure was last Thursday and the dashboard is still wrong today.

Where the pipeline code lives

An orchestrator introduces a question that a set of cron entries never raised: what is the relationship between your pipeline definitions and your ordinary software practices?

Definitions are code and belong in version control, reviewed like anything else. A pipeline changed by editing something in a running system has no history and no review.

Keep the orchestration thin. The graph should describe dependencies; the transformation logic belongs in the warehouse or in libraries the graph calls, where it can be tested independently.

Test the tasks, not the graph. A task is a function with inputs and outputs and tests ordinarily; the graph's correctness is mostly about dependencies, which a structural check can verify.

Have environments. A development orchestrator reading development data is what lets somebody change a pipeline without gambling on production.

Pin your dependencies. An orchestrator's own upgrades are frequently disruptive, and a pipeline estate is unusually sensitive to them.

Deploy pipelines like software, through the same route as everything else, rather than by copying files onto a server.

Orchestrating with CodeWords

CodeWords connects to more than 3,000 integrations, and the dependency and retry behaviour is part of describing the automation rather than infrastructure to operate.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: what depends on what, and how often.
3. Describe the exceptions: a source that is unavailable, a result that is empty, a figure outside the expected range.
4. Connect the systems involved once and reuse the connections.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to get right first

Parameterise every pipeline by date and make each task idempotent. It costs almost nothing at the start and it is what makes backfilling possible — and you will need to backfill, usually urgently, usually because something was wrong for a week and somebody has just noticed.

Two habits make the difference. Put data quality checks inside the graph rather than beside it, so a bad result blocks downstream work instead of flowing into a dashboard. And treat an empty result as a failure until proven otherwise, because it is the one bad outcome that reliably reports success.

Frequently asked questions

When does cron stop being enough?

When one job depends on another finishing. At that point the schedule is encoding a dependency by guessing at a duration, and the guess is wrong on any slow day — which produces downstream jobs reading stale data and succeeding.

What does an orchestrator give me that a scheduler does not?

Ordering by dependency rather than by clock, parallelism for independent branches, retries of the failed step rather than the whole pipeline, and the ability to rerun from a point of failure.

What is the difference between task-based and asset-based?

Task-based describes operations and their order; asset-based describes the data that should exist and how each piece is produced. Asset-based tends to suit data work better, because people ask whether a table is current rather than whether a job ran.

Why does a successful pipeline produce wrong numbers?

Because a task succeeding means the code did not error, not that the output is correct. Row counts, freshness, and null rates belong inside the graph so a bad result blocks downstream work.

What is the most dangerous failure?

An empty result, because it usually succeeds. A source returning nothing produces downstream tables of zeros and a dashboard that appears to show business stopping, with nothing reporting an error anywhere.

How should backfills be handled?

Parameterise by date from the start, make tasks idempotent, run backfills separately from the scheduled pipeline, and limit concurrency so a two-year backfill does not overwhelm the source it reads from.

Who should be alerted?

The owner of the affected data, with the downstream dashboards named, plus the consumers — who need to know the data is stale before they act on it. That last part is almost never built and is the one that prevents wrong decisions.

Should pipeline definitions be in version control?

Yes, and deployed through the same route as your other software. A pipeline edited inside a running system has no history, no review, and no way to reproduce what it looked like when a number was wrong.

What belongs in the orchestrator and what does not?

Dependencies and scheduling belong there; transformation logic belongs in the warehouse or in libraries the graph calls. Logic embedded in orchestration is logic that cannot be tested without running the whole pipeline.

Related reading

• ETL pipelines explained
• Data lineage
• What is a DAG in data engineering