Register your interest: Tag @Cody, get an agent
BlogEngineering

Terraform and CodeWords: automating around state, not through it

State is the thing to protect, drift is what you cannot see, and apply should stay gated. The reporting and plan-time automation that is worth building, and the automation that is not.

Osman RamadanOsman Ramadan11 min read

Summarize with AI

Terraform and CodeWords: automating around state, not through it
On this page

Terraform is already automation, so the question is not how to automate it. It is what should surround it — and the answer is shaped by one thing above all others: state is the record of what exists, and a damaged state file is a considerably worse day than a failed apply.

That makes most of the useful work here read-only. Reporting on plans, detecting drift, and checking what has changed are all safe and all valuable. Running apply from an automation is a narrower proposition and deserves more scepticism than it usually gets.

What we'll cover

State, and why it governs everything

State maps configuration to real resources. Without it, Terraform does not know that a resource in your code is the one running in your account.

Concurrent operations corrupt it, which is why locking exists and why an automation that bypasses locking is dangerous rather than merely inefficient.

State contains secrets. Values marked sensitive in the configuration are stored in plain form in state, so the backend holding it needs encryption and restricted access.

Losing state does not delete infrastructure, and it does mean rebuilding the mapping by importing resources one at a time, which is slow and error-prone.

Never edit state by hand except through Terraform's own commands, and even then with a backup taken first.

The practical rule for automation: read state and plans freely, and treat anything that writes state as an operation requiring the same care as a deployment.

Drift, which you cannot see by default

Drift is the gap between what the configuration says and what actually exists, and it accumulates for entirely ordinary reasons.

Somebody changed something in the console. Usually during an incident, usually correctly, and usually without going back to codify it.

A provider changed a default, so a resource now has an attribute nobody set.

Another automation modified a resource that Terraform also manages.

A resource was created outside Terraform in a namespace Terraform believes it owns.

Nothing reports this. A scheduled plan does, which is why drift detection is the most valuable thing on this list: it converts an invisible divergence into a short weekly list, while somebody still remembers why the change was made.

Where the automation belongs

In CI, for anything triggered by a change to the configuration. Plan on the pull request, apply after merge and approval. This is well-trodden and it is where the core loop belongs.

Outside, for anything scheduled or cross-cutting. Drift detection across many workspaces, cost reporting, provider version tracking, and compliance checks are not triggered by a commit and do not belong in a pipeline attached to one repository.

Outside, for anything connecting to other systems — the ticket, the change calendar, the cost platform, the notification.

The division is the same as for any CI-shaped tool. If a commit triggers it and the repository is the context, it belongs in the pipeline.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

  1. Open CodeWords and start a new automation.
  2. Describe what should happen in plain language to Cody, the automation builder: which workspaces, what to check, and who should be told.
  3. Authorize the connection to your Terraform platform and to any cloud provider you read from, with read-only credentials for anything that only reports.
  4. Describe the exceptions: a plan that fails, a workspace already locked, a change involving a destroy.
  5. Run it against a non-production workspace before anything looks at production.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Scheduled drift detection. A plan run on a schedule across workspaces, reporting where reality has diverged from the configuration. The highest-value automation here.

Plan summaries in plain language. Terraform plan output is long and technically precise, and a summary naming what is created, changed, and destroyed is what actually gets read on a pull request.

Destroy warnings. Any plan involving destruction of a stateful resource — a database, a storage bucket, a volume — flagged prominently rather than buried in output.

Provider and module version reporting. Which workspaces are pinned to what, and what is outdated. Version drift across workspaces is invisible and awkward when it matters.

Cost reporting from plans. What a change will cost before it is applied, attached to the pull request where the decision is being made.

Compliance checking. Resources lacking required tags, buckets that are public, security groups that are too open, reported per workspace.

Change records. An entry in your change management system when infrastructure changes, assembled from the plan rather than typed by somebody afterwards.

Reading a plan usefully

Plan output in machine-readable form is what makes most of the above possible, and a few habits make the results worth reading.

Count by action. Created, changed, destroyed, replaced. Four numbers at the top of a comment do most of the work.

Call out replacements separately. A replacement destroys and recreates, which for a database is a very different event from an in-place change, and the plan output presents both similarly.

Name the stateful resources involved, since those are the ones where a mistake is not recoverable.

Highlight changes to security-relevant attributes — network rules, public access, permissions — because those are what a reviewer should look at first.

Keep the full output available and do not lead with it. The summary is for deciding whether to look closer.

Apply, and why it should stay gated

Automated apply is common and it deserves more scrutiny than it gets.

Apply after merge with approval is reasonable for most changes, and this is the standard arrangement.

Automatic apply on a schedule is not. Infrastructure changing without a person deciding is how an unnoticed configuration change becomes an outage at a time nobody expected.

Destroys should require explicit approval, always, and preferably from somebody other than the author.

Production should have a different gate from everywhere else. The same pipeline with the same approvals across all environments removes the safety that having environments provides.

Never apply from an automation that cannot be stopped. Whatever runs apply needs a person able to cancel it, and a clear record of what it did.

The reasonable position: automate the plan, the summary, the checks, and the notification. Let a person press apply for anything consequential. The time saved by removing that step is small and the failure it prevents is not.

Workspaces, and the sprawl that follows

Any organisation past a certain size has more workspaces than anybody can name, and the reporting that helps is unglamorous.

Inventory them. Which workspaces exist, which repository and directory each maps to, and who owns it. This list almost never exists and is the prerequisite for everything else here.

Find the abandoned ones. Workspaces not applied in a year, still holding state for resources still costing money.

Find the ones with no recent plan, which are the ones where drift will have accumulated undetected.

Track variable and credential age, since long-lived provider credentials in workspaces are a standing risk nobody reviews.

Report per team, not centrally. A central list of ninety workspaces is a document; the eleven belonging to one team is a task.

The inventory is dull to build and it is the thing that turns every other automation here from a per-workspace script into something that covers the estate.

Building it so it survives

Use read-only credentials for reporting. Drift detection needs to read the provider and the state, not to change anything.

Respect locking. An automation running plan against a workspace while somebody applies is asking for trouble; check the lock and skip rather than wait indefinitely.

Never write state. If an automation needs to import or remove resources from state, that is a person's job with a backup taken first.

Handle plan failure as a result. A plan that cannot run is itself a finding — usually a credential, a provider version, or a resource removed outside Terraform.

Report the outcome. Workspaces checked, drift found, plans that failed, and what was skipped.

Limits worth knowing about

Plan is not free. It calls provider APIs to refresh state, which takes time and counts against provider rate limits, so a scheduled plan across many workspaces needs pacing.

State can contain secrets in plain form, so anything reading state must be treated as handling credentials.

Provider rate limits apply during refresh, and a large workspace refreshing everything can hit them.

Some resources always show a difference due to provider quirks or computed attributes, and a drift report that does not account for them cries wolf until people ignore it.

Terraform Cloud and self-managed workflows differ in what is available through an API, so check which you are building against.

What to build first

Scheduled drift detection: a plan run weekly across your workspaces, reporting where reality differs from configuration. It is read-only, it needs no changes to your pipeline, and it catches the console change made during an incident while somebody still remembers why they made it.

Two habits make the difference. Exclude the known-noisy resources from the first run, or the report is ignored by its third week. And route each workspace's drift to the team that owns it, since a central list of drift across forty workspaces belongs to nobody.

Frequently asked questions

Should automation run terraform apply?

After merge and approval, reasonably. On a schedule with no person involved, no. Destroys should always require explicit approval, and production should be gated differently from other environments.

How do I detect drift?

Run plan on a schedule and report any non-empty result. Nothing else surfaces a change made in a cloud console, and those changes are the normal outcome of incidents rather than an exception.

Is it safe for an automation to read state?

Reading is safe; the caveat is that state contains sensitive values in plain form, so anything reading it is handling secrets and needs to be treated that way. Writing state should stay a human operation with a backup.

Why does drift detection report changes that are not real?

Provider quirks and computed attributes make some resources always show a difference. Identify those early and exclude them, or the report loses credibility and people stop opening it.

What should a plan summary include?

Counts by action, replacements called out separately, stateful resources named, and security-relevant changes highlighted. The full output should be available and should not be what a reviewer reads first.

Can I get a cost estimate before applying?

Yes, from the plan, attached to the pull request. It is most useful as a delta rather than a total — what this change adds — which is the number the person deciding actually wants.

Should each environment share a pipeline?

Share the mechanism, not the gates. Identical approvals everywhere removes the point of having separate environments, and production deserves a stricter path than a development workspace.

How do I find workspaces nobody is maintaining?

Inventory every workspace with its last apply, last plan, and owner. Workspaces untouched for a year usually still hold state for resources still being billed, and nothing surfaces them until somebody audits the cloud account instead.

Should drift reports go to a central channel?

No — route each workspace's drift to the team that owns it. A combined list across the estate belongs to nobody and gets skimmed; eleven items belonging to one team is a task somebody completes.

Get started today

Your first workflow is free to build.

Describe what you need. Cody handles the build, the connections, and the deployment.