AI workflow automation: when the model is actually needed

Putting a model into a workflow changes its character. A deterministic step either works or fails visibly; a model step produces an answer every time, including when it is wrong, and the wrongness is plausible rather than obvious.

That is worth the trade for a small number of steps and is a bad trade for most of them. The design question in this whole subject is which is which, and it is usually answered by default — by reaching for a model wherever the input is text.

What we'll cover

• Two kinds of step
• Where a model earns its place
• Where it does not
• Building it with CodeWords
• Designing around uncertainty
• Confidence, and what to do with it
• Testing something non-deterministic
• Cost and latency in the path
• Prompts are code, and should be treated as such
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Two kinds of step

Deterministic steps produce the same output for the same input, fail loudly, and are testable by assertion.

Probabilistic steps produce a plausible output for any input, fail silently, and are testable only statistically.

Mixing them is normal and the proportion matters — a workflow of twelve steps where one is probabilistic is a different object from one where nine are.

Errors compound differently. Two deterministic steps either work or do not; two probabilistic steps multiply their error rates.

Debugging differs entirely. A deterministic failure has a stack trace; a probabilistic one has an output that looks fine and is not.

Prefer deterministic wherever the logic can be expressed as a rule, which is more often than it first appears.

Where a model earns its place

Unstructured input that genuinely varies — free text, documents in inconsistent formats, speech.

Classification with fuzzy boundaries, where a rule list would need hundreds of cases and still miss some.

Summarisation and extraction from language, where the shape of the input is not known in advance.

Generation of a draft that a person will review, which is the safest application because the review is built in.

Matching things that are the same but written differently, which rules handle poorly and which is a large share of real data work.

Where the cost of an occasional error is low and the volume makes a rule-based approach impractical.

Where it does not

Anything expressible as a rule. If a condition can be written down, write it down.

Arithmetic, which models do unreliably and code does exactly.

Anything requiring the same answer every time, such as a pricing decision or a compliance check.

Data lookups, which should query the source rather than asking a model what it remembers.

Routing with a fixed set of destinations and clear criteria, which is a rule.

Anything irreversible. A probabilistic step deciding to delete, pay or send is the design most likely to produce a memorable failure.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the steps, which need judgement, and what should happen when confidence is low.
3. Authorize the connections the workflow needs.
4. Describe the exceptions: an ambiguous input, a low-confidence result, an output outside an expected range, a step that failed.
5. Run it over inputs where you already know the right answer and compare.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Designing around uncertainty

Constrain the output. A step that must return one of five values is far more reliable than one returning free text.

Validate afterwards, deterministically, since checking a result against a rule is cheap and catches the obvious failures.

Give it an escape. "Cannot determine" must be an available answer, or the model will invent one.

Put the uncertainty where it is cheap. A probabilistic step early, feeding a human review, costs less than one late in the path acting on its own conclusion.

Never chain two probabilistic steps without a check between them, since the second amplifies the first's errors and nothing observes it.

Make the failure mode visible, because the characteristic problem here is a wrong answer that looks right.

Confidence, and what to do with it

Ask for it and use it, since a system that discards confidence has discarded the mechanism for handling its own uncertainty.

Set a threshold empirically by looking at what was wrong below various levels, not by choosing a round number.

Route low confidence to a person, which is the whole point of having it.

Monitor the low-confidence rate as a health signal — a rise means the inputs have changed even when nothing else has.

Expect poor calibration. Reported confidence and actual accuracy correlate imperfectly, so the threshold needs validating rather than trusting.

Report both: what was processed automatically and what needed review, since the second number is the honest measure of how much was automated.

Testing something non-deterministic

Build an evaluation set of inputs with known correct outputs, which is tedious and is the only real option.

Measure accuracy per category, not overall, since the average hides the category that is badly wrong.

Re-run it on every change to a prompt, a model version or the surrounding logic.

Watch for silent drift. A provider updating a model can change behaviour without any change on your side.

Pin versions where the provider allows it, and treat an upgrade as a change requiring re-evaluation.

Keep failures as test cases, which is how the evaluation set becomes useful rather than theoretical.

Cost and latency in the path

Model calls are slow relative to everything else in a workflow, which matters for anything a person is waiting on.

Cost scales with volume in a way deterministic steps do not, and a workflow that is cheap at a hundred a day may not be at a hundred thousand.

Batch where the workflow allows it, since per-item calls are the expensive pattern.

Cache aggressively for repeated inputs, which in most real workflows is a substantial share.

Use the smallest model that passes evaluation, rather than the best available, since the difference in cost and latency is large and the difference in outcome frequently is not.

Set a budget alarm, because runaway cost in this category is usually a loop rather than growth.

Prompts are code, and should be treated as such

A prompt determines behaviour, which makes it part of the system rather than a setting. Most of the operational problems in this area come from treating it as configuration.

Put it in version control, so a change is reviewable and a regression is traceable to a diff.

Review changes the way you would review a conditional, since a reworded instruction can alter behaviour on a whole class of inputs.

Never edit one in production without running the evaluation, which is the equivalent of a hot fix with no tests.

Record which version produced which output, or investigating a past result is impossible.

Keep them readable. A prompt accumulated over a year of fixes is as unmaintainable as any other code nobody refactored.

Expect them to need maintenance as models change underneath them, which is a running cost rather than a one-off.

Making it survive

Handle provider outages and rate limits, which happen, by queuing or degrading rather than failing the workflow.

Set timeouts and decide what a timeout means, since a slow response and no response need different handling.

Log the input and output of every probabilistic step, which is the only way to investigate a wrong result afterwards.

Never let a model call block a critical path without a fallback.

Report the outcome: items processed, confidence distribution, escalations, and the cost.

Limits worth knowing about

Plausible wrongness is the defining hazard, and no amount of prompting removes it entirely.

Behaviour changes with model versions, sometimes in ways that affect only a subset of inputs.

Sensitive data leaving your environment is a decision with contractual and regulatory dimensions.

Evaluation is expensive to build and to maintain, and it is the step most often skipped.

Deterministic code is usually the better answer, and the discipline is remembering to ask.

What to build first

The evaluation set: a few dozen real inputs with the correct outputs written down. It is unglamorous, it is the thing that lets you change a prompt or a model without guessing, and it converts every later decision in this subject from an argument into a measurement.

Two habits make the difference. Ask whether each step could be a rule before making it a model call, since deterministic code is testable, cheap, fast and honest about failing. And never chain two probabilistic steps without a deterministic check between them, because the second amplifies the first's errors and nothing in the workflow will notice.

Frequently asked questions

When does a workflow actually need a model?

For genuinely unstructured input, classification with fuzzy boundaries, summarisation and extraction from language, draft generation a person will review, and matching things written differently. Not for anything expressible as a rule.

What should never be a model step?

Arithmetic, data lookups, anything requiring the same answer every time such as a pricing or compliance decision, routing with fixed destinations and clear criteria, and anything irreversible.

What is the defining hazard?

Plausible wrongness. A deterministic step fails visibly; a probabilistic one produces an answer every time including when it is wrong, and the wrongness looks reasonable.

How do I design around that?

Constrain the output to a small set of values, validate the result deterministically afterwards, make "cannot determine" an available answer, and put the uncertainty early where it feeds a human review rather than late where it acts.

Why not chain several model steps?

Because errors multiply and nothing observes it. A deterministic check between them is what stops the second step amplifying the first's mistakes.

How do I test something non-deterministic?

An evaluation set of real inputs with known correct outputs, measured per category rather than overall, re-run on every prompt or model change. Provider-side model updates can change behaviour with no change on your side.

How should confidence scores be used?

To route low-confidence items to a person, with the threshold set empirically rather than chosen as a round number. Reported confidence correlates imperfectly with accuracy, so it needs validating.

Which model should I use?

The smallest that passes your evaluation. The cost and latency difference between model sizes is large and the difference in outcome frequently is not.

Should prompts be version controlled?

Yes — they determine behaviour, so they are code rather than configuration. Review changes as you would a conditional, run the evaluation before any production edit, and record which version produced which output.

Do prompts need ongoing maintenance?

Yes. Models change underneath them, so behaviour can drift with no change on your side. It is a running cost rather than a one-off piece of work.

Related reading

• AI workflow automation software: how to evaluate one
• AI workflow patterns worth knowing
• AI integration software