MLflow and CodeWords: automating the model lifecycle

MLflow records what happened. Which run, which parameters, which metrics, which artifacts, which model version is in which stage. What it does not do is act on any of it, which leaves the interesting parts — deciding a model is good enough, promoting it, noticing it has stopped being good enough — to people with other work to do.

That gap is where automation belongs, and the most valuable automation here is not the promotion step. It is the checking that happens before and after.

What we'll cover

• Reproducibility, which decays quietly
• What the MLflow API reaches
• Connecting it to CodeWords
• Seven automations worth building
• Promotion gates that mean something
• Monitoring after deployment
• Where your tracking server lives, and why it matters
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Reproducibility, which decays quietly

A tracked run is not automatically a reproducible one, and the difference shows up months later when somebody asks how a model in production was built.

The code version. A run without a recorded commit is a run you cannot rebuild, and the commit is easy to log and easy to forget.

The data version. The dataset a run used, identified precisely enough to fetch again. This is the most commonly missing piece and the one that makes an old run irreproducible in practice.

The environment. The dependency versions the run executed under, since a library's default behaviour changing between versions will change results.

The seed, where randomness is involved, which is most of the time.

An automation that checks runs for these and reports what is missing costs little and is worth more than most of what teams build here, because it catches the omission while the context still exists rather than when somebody needs it.

What the MLflow API reaches

Experiments and runs can be created, searched, and read, including parameters, metrics, tags, and artifacts.

Metrics with history — a metric logged repeatedly through training has a full series, not only a final value, which matters for anything detecting a run that diverged.

The model registry holds registered models and versions, with stages, descriptions, and tags.

Stage transitions can be requested, made, and read, which is the mechanism promotion automation uses.

Artifacts can be listed and downloaded — plots, evaluation outputs, and the model itself.

Tags are the extension point for anything the schema does not cover, including approvals and evidence links.

Webhooks exist on Databricks-managed MLflow; open-source deployments generally poll instead, which is fine at the frequency model events occur.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which registered model, what conditions matter, and who should hear about it.
3. Authorize the connection to your tracking server, with credentials that can read runs and, where needed, transition stages.
4. Describe the exceptions: a run missing a metric, an evaluation on the wrong dataset, a metric that improved suspiciously.
5. Test against a scratch experiment before anything touches the registry.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Reproducibility checking. Flag runs missing a code version, a data version, an environment record, or a seed, while the person who ran them still remembers.

Promotion gating. When a version is proposed for staging or production, check it against the criteria and report a verdict with evidence, rather than letting the transition depend on somebody remembering to look.

Regression detection against the incumbent. Compare the candidate with what is currently deployed on the same evaluation data, including the segments that matter rather than only the aggregate.

Training failure alerting. Runs that errored, that diverged, or that finished with metrics far outside the usual range. Failed training jobs are otherwise noticed when somebody wonders where the model got to.

Experiment digests. What was tried this week, what improved, and what the current best is, so people not reading the tracking interface still know.

Drift monitoring. Compare live input distributions and, where labels arrive, live performance against what the model was validated on.

Registry hygiene. Versions stuck in staging for months, models with no owner, and stages that no longer reflect what is deployed.

Promotion gates that mean something

A gate that always passes is a formality. A few things make the difference.

Compare against the incumbent, not a threshold. The question is whether this is better than what is deployed, which a fixed threshold does not answer.

Evaluate on the same data. A candidate evaluated on a different split is not comparable, and this is the commonest way a gate is passed by something that should not have passed.

Check the segments. An aggregate improvement can conceal a serious regression on a subgroup, which is the failure most likely to cause real harm and least likely to be caught by a single number.

Check cost as well as quality. Latency, size, and inference cost are part of whether a model is deployable.

Record the evidence on the version. Tag it with what was checked and what the numbers were, so the decision is auditable later without reconstruction.

Report, do not decide, for anything consequential. For production promotion, a gate that assembles the evidence and asks a person is more useful than one that transitions automatically. The work saved is the gathering, not the judgement.

Monitoring after deployment

The thing that goes wrong is rarely the deployment. It is the slow divergence afterwards.

Input distributions shift relative to training data, which is usually detectable long before performance visibly degrades.

Labels arrive late. Where the outcome is known eventually, measure performance on it, accepting the delay, since it is the only direct evidence.

Prediction distributions move, which is available immediately and is a useful proxy when labels are slow.

Volume by segment changes, which explains a good deal of apparent drift and is trivial to check.

Log the results back as metrics against the model version, so production performance lives in the same place as validation performance rather than in a separate dashboard nobody correlates.

Where your tracking server lives, and why it matters

MLflow runs in several configurations, and an automation written against one may not work against another.

A local file store works for a single person and cannot be reached by anything else. Any shared automation implies a remote tracking server.

A remote tracking server with a database backend is the usual arrangement, and it is what automation should point at.

Managed MLflow — on Databricks or another platform — adds capabilities, notably webhooks on registry events, and adds its own authentication model.

Artifact storage is separate from tracking. The tracking server records where an artifact is; the bytes live in object storage. An automation downloading artifacts needs credentials for that store as well, which is the most common reason a script that reads metrics fine cannot fetch a model.

Authentication varies by deployment, from none on an internal server to full platform credentials on a managed one, so confirm what yours expects before building around an assumption.

Building it so it survives

Make transitions idempotent. Check the current stage before requesting a change, since a retried automation should not produce a second transition.

Search runs with filters rather than listing an experiment and filtering afterwards, since experiments accumulate thousands of runs.

Handle the missing metric. A run lacking the metric you gate on should be treated explicitly, not as a zero.

Keep credentials scoped. Reading runs and reporting needs far less privilege than transitioning production stages, and most of what you build only reads.

Report the outcome. Which version, which checks, which passed, and what the numbers were.

Limits worth knowing about

Open-source MLflow has no webhooks, so polling is the mechanism unless you are on a managed deployment that adds them.

Search syntax is limited compared with a general query language, and complex selection is easier done by filtering the results you retrieve.

Artifact storage is separate from the tracking server, so an automation downloading artifacts needs access to the underlying store as well as to MLflow.

Stages are a convention, not enforcement. Nothing stops a version being transitioned by anybody, and nothing guarantees the production stage matches what is actually serving traffic — a gap worth an automation of its own.

Metric history can be large for long runs logging frequently, so retrieve what you need rather than everything.

What to build first

Reproducibility checking: a scheduled pass over recent runs reporting which lack a code version, a data version, an environment record, or a seed. Read-only, quick, and it catches the omission while it is still cheap to fix.

Two habits make the difference. Report it to the person who ran the run rather than to a channel, since only they can fill the gap. And check the data version specifically, because it is the one most often missing and the one that makes an old model impossible to rebuild.

Frequently asked questions

Should promotion to production be fully automatic?

Rarely. Gather the evidence automatically, check it automatically, and let a person approve the production transition. The automation removes the assembling, which is the tedious part; the judgement is the part worth keeping human.

How do I compare a candidate against the deployed model?

Evaluate both on the same held-out data and compare the metrics that matter, including by segment. Comparing a candidate's numbers against a threshold instead is the usual shortcut and it answers a different question.

Can I get notified when a model is registered?

On managed MLflow with webhook support, directly. On open-source deployments, poll the registry on a schedule — model registration happens infrequently enough that a few minutes of latency is immaterial.

What is the best drift signal when labels arrive slowly?

Input distribution shift and prediction distribution shift, both available immediately. Treat them as early warnings rather than as measurements of performance, and measure performance properly when labels arrive.

Why is a run from last year impossible to reproduce?

Almost always a missing data version. Code and environment are usually recoverable; the exact dataset as it stood is not, unless it was identified at the time. That is the argument for checking runs while they are recent.

Should automations transition stages directly?

For staging, reasonably. For production, prefer requesting and reporting. Also worth building: a check that the production stage matches what is actually serving traffic, because nothing in MLflow guarantees that it does.

How do I stop the registry filling with abandoned versions?

Report versions that have sat in staging beyond a threshold, with the person who registered them, and archive on their say-so. Automatic archiving is a poor idea, since some of those versions are deliberately parked.

Why can my automation read metrics but not download artifacts?

Because artifact storage is separate from the tracking server. The tracking server tells you where the artifact is; fetching it needs credentials for the underlying object store, which are often granted separately and forgotten.

Does open-source MLflow support webhooks?

Not in the way managed deployments do. Poll the registry on a schedule instead — model registration and stage transitions happen infrequently enough that polling every few minutes is entirely adequate.

How do I know the production stage matches what is deployed?

You check, because nothing in MLflow guarantees it. Compare the version marked production in the registry against the version your serving infrastructure reports, and alert on a mismatch. It is a short automation and it catches a genuinely dangerous drift.

Related reading

• Hugging Face and CodeWords
• BigQuery and CodeWords
• CodeWords integrations and templates