MongoDB and CodeWords: automating against documents that drift

The flexibility that makes MongoDB pleasant to develop against is the thing that makes automating against it interesting. Documents in the same collection can have different shapes, fields appear and disappear across application versions, and nothing rejects a document because it is missing something your automation expects.

That is not a flaw so much as a trade, and it means an automation written against the documents you sampled will eventually meet the ones written by a version of the application from two years ago.

What we'll cover

• Schema drift, which is guaranteed
• What the MongoDB API reaches
• Change streams instead of polling
• Connecting it to CodeWords
• Seven automations worth building
• Aggregation, which is the reporting tool
• Atlas or self-hosted, which changes the options
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Schema drift, which is guaranteed

Four shapes of drift, all of which an automation will eventually encounter.

Fields that were added. Documents written before the field existed simply lack it, and a query filtering on it silently excludes them rather than erroring.

Fields that changed type. A value stored as a string in one version and a number in another, which makes comparisons behave unpredictably and sorts produce surprising orders.

Fields that were renamed. Both names exist in the collection, each covering a period of history.

Nested structure that moved. A value that used to sit at the top level now living inside a sub-document, or the reverse.

None of these produces an error. They produce results that are quietly incomplete, which is the characteristic failure here and the reason the first automation worth building is one that measures the drift rather than assuming it away.

What the MongoDB API reaches

Documents can be found, inserted, updated, replaced, and deleted, individually or in bulk, with a rich query language over nested structures.

Aggregation pipelines transform and summarise, and they are where any real reporting belongs rather than pulling documents out and processing them elsewhere.

Indexes can be listed and created, and whether one exists decides whether a query is instant or a collection scan.

Change streams deliver a durable, resumable feed of changes, which is the right mechanism for reacting to anything.

Bulk write batches operations, which matters at volume.

Collection statistics report sizes, document counts, and index usage, which supports capacity reporting and the index question below.

Atlas Search provides full-text search where you are on Atlas, which avoids standing up a separate search system for straightforward cases.

Change streams instead of polling

Change streams are durable and resumable, which is the important property. Each event carries a resume token, and an automation that stores the token can pick up exactly where it stopped after a restart rather than missing what happened while it was down.

They see deletions, which polling on an updated_at field does not. Deletion invisibility is the classic gap in polled synchronisation and it produces downstream systems holding records that no longer exist.

They require a replica set, which every production deployment and every Atlas cluster is, so this is rarely a constraint in practice.

Filter at the source. A pipeline on the change stream means you receive only the operations and fields you care about rather than everything, which matters on a busy collection.

Store the resume token durably after processing, not before. Storing it first means a crash mid-processing skips the event permanently.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which collection, which documents, and what to do with them.
3. Authorize the connection with a connection string pointing at a secondary for read-heavy work, using a user scoped to the specific database and collections.
4. Describe the exceptions: a document missing an expected field, a field of an unexpected type, a change stream that cannot resume.
5. Run it against a restored snapshot before pointing anything at production.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Schema drift reporting. Sample the collection and report which fields exist on what proportion of documents, with their types. This is the first thing to build and it routinely surprises the team that owns the application.

Data quality checks. Documents missing required fields, values outside expected ranges, orphaned references, and states that should not persist. Read-only, reporting findings.

Change-driven sync. Feed a warehouse, a search index, or another service from a change stream, including deletions.

Operational reporting. The aggregation somebody runs by hand each week, delivered before they ask.

Reconciliation against other systems. Counts and sampled contents compared against whatever else holds the same entities.

Index usage reporting. Which indexes are used, which are not, and which queries are scanning collections. Unused indexes cost write performance and storage for nothing.

Growth reporting. Collection sizes, document counts, and growth rates, so scaling is planned rather than urgent.

Aggregation, which is the reporting tool

Pulling documents out and summarising them in your automation is the common approach and usually the wrong one.

Aggregate in the database. Matching, grouping, and computing happen close to the data rather than after transferring all of it, which is faster and moves far less over the network.

Match first and early. A pipeline that filters before it does anything else can use an index; one that filters late cannot.

Project only what you need. Dropping fields early in the pipeline reduces what every subsequent stage handles.

Watch memory. Stages have memory limits, and pipelines exceeding them fail unless permitted to use disk, which is slower. Filtering earlier usually resolves it more cleanly than raising limits.

Run analytical pipelines against a secondary where you have one, so heavy aggregation does not compete with application traffic.

Keep the pipeline readable and stored somewhere people can see it. Aggregation pipelines become dense quickly, and one buried inside an automation nobody opens is untrustworthy the moment somebody questions a number.

Atlas or self-hosted, which changes the options

The deployment shapes what is available more than most people expect.

Atlas provides triggers, which run functions on database events or on a schedule without any external scheduler. For small reactions they are simpler than anything external, and they keep the logic inside the database platform where your automation tooling cannot see it — a trade worth making deliberately.

Atlas Search avoids standing up a separate search cluster, which removes an entire system from anything where search is the point.

Network access lists are explicit on Atlas, so an automation connecting from elsewhere needs its egress addresses added before anything will work.

Smaller tiers have real connection limits, low enough that a handful of concurrent automation runs can exhaust them.

Self-hosted deployments give you the oplog window and the replica topology directly, which matters for change stream resumption, and hand you the responsibility for both.

Whichever you are on, use a database user scoped to the specific database rather than a cluster-wide one, and check whether the tier you are on can absorb the read load you are about to add.

Building it so it survives

Handle missing fields explicitly. Decide whether absent means zero, unknown, or excluded, per field, because the default behaviour will not be what you want for all of them.

Make writes idempotent. Upsert on a stable identifier rather than insert, so a retried run does not duplicate.

Store change stream resume tokens after processing. This is the difference between a restart resuming correctly and silently skipping events.

Scope credentials to the database and collections. An automation with cluster-wide privileges can drop collections, which is far more reach than reporting requires.

Report the outcome. Documents read, written, and skipped, with the reason for skipping. The skip count is where drift becomes visible.

Limits worth knowing about

Document size is capped, which matters for anything appending to an array indefinitely — a pattern that works until it abruptly does not.

Aggregation stages have memory limits, and exceeding them fails unless disk use is allowed.

Connection limits apply, particularly on smaller Atlas tiers, so bursty automation runs need pooling.

Change streams need a replica set and the oplog window determines how far back a resume token remains valid. An automation down longer than that window cannot resume and needs a full reconciliation instead.

Indexes cost writes. Adding indexes to make an automation's query fast slows down every insert, which is a trade to make deliberately rather than by accident.

What to build first

Schema drift reporting: sample the collection and report which fields appear on what proportion of documents, and with which types. Read-only, quick to build, and it tells you what every subsequent automation has to handle. In most collections of any age it finds at least one field that is two types.

Two habits make the difference. Sample across the whole history rather than recent documents, since the drift lives in the old ones. And share the output with whoever owns the application, because some of what it finds is a bug rather than a quirk.

Frequently asked questions

How do I handle documents with different shapes?

Decide explicitly what a missing field means for each field you use, and report the proportion affected rather than silently excluding them. A query filtering on a field that only half your documents have will return half your data without any indication.

Should I poll or use change streams?

Change streams, in almost every case. They are durable, resumable, and they see deletions, which polling on a timestamp does not. Polling remains reasonable for low-frequency checks where missing a deletion does not matter.

Why is my query slow?

Usually a missing index, and the explain output will say whether it scanned the collection. The second most common cause is an aggregation that filters late, which prevents an index being used even though one exists.

Can automations write to the application's database?

Cautiously, and preferably not to collections the application owns, since you bypass whatever validation its code performs. Giving the automation its own collections to write into avoids the question entirely and is usually straightforward.

What happens if my automation is down for a long time?

Change streams resume from a token as long as the oplog still covers that point. Beyond the oplog window the token is invalid and you need a reconciliation pass instead, which is worth building before you need it rather than during an incident.

Should reporting run against a secondary?

Yes, for anything analytical. Secondaries are slightly behind, which is immaterial for reporting and matters if you are checking whether something just written is there. Reading your own recent writes should go to the primary.

How do I stop an array growing until the document breaks?

Cap it deliberately, or move the items into their own collection. An array appended to on every event works perfectly until the document reaches the size limit, at which point writes fail on exactly the documents belonging to your most active users.

Should I use Atlas triggers or an external automation?

Triggers for small reactions entirely inside the database, external automation for anything touching other systems or needing to be visible alongside the rest of your workflows. The risk with triggers is logic living somewhere nobody looks when they are auditing what runs.

Why does the automation fail to connect to Atlas?

Usually the network access list. Atlas requires source addresses to be permitted explicitly, so confirm what your automation egresses from and have those added, rather than assuming credentials alone are enough.

Related reading

• PostgreSQL and CodeWords
• Elasticsearch and CodeWords
• CodeWords integrations and templates