Elasticsearch and CodeWords: keeping the index honest

The most consequential thing to understand about Elasticsearch is that it is a search index rather than a system of record. Everything in it is a copy of something that lives somewhere else, which means the interesting failure is not data loss but divergence: the index confidently returning results that no longer match reality.

Most automation worth building around it exists to prevent that divergence, detect it when it happens anyway, and make reindexing something you can do on a Tuesday rather than something you schedule for a weekend.

What we'll cover

• Why divergence is the real risk
• What the Elasticsearch API reaches
• Connecting it to CodeWords
• Seven automations worth building
• Reindexing without downtime
• Detecting drift
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Why divergence is the real risk

Deletes are the classic. A record removed from the database, and the indexing pipeline handles creates and updates but not deletions. The item is gone and still searchable, and somebody finds it by clicking a result that leads nowhere.

Failed updates leave stale documents. An indexing call that failed and was not retried produces a document reflecting a previous state, indefinitely.

Mapping changes are not retroactive. Changing how a field is analysed affects new documents. Everything indexed under the old mapping keeps its old behaviour, so the same query returns different results depending on when a document was written.

Partial reindexes. A reindex that stopped halfway leaves an index that is part new and part old, which is worse than either.

None of these produces an error anybody sees. They produce search results that are quietly wrong, which is why detection has to be built deliberately.

What the Elasticsearch API reaches

Documents can be indexed, updated, deleted, and retrieved by ID, individually or in bulk.

The bulk API is the one that matters for anything at volume, since indexing documents one at a time is dramatically slower.

Search covers the query language, aggregations, and the scoring behaviour you are presumably here for.

Index management covers creating indices, updating mappings and settings, and reading what currently exists.

Aliases point a name at one or more indices, and they are the mechanism behind zero-downtime reindexing.

Reindex copies documents from one index to another, optionally transforming them, and runs as a task you can monitor.

Tasks report on long-running operations such as a reindex, so an automation can wait properly rather than guessing.

Index lifecycle management handles rollover and retention for time-series data, which is configuration rather than something your automation should reimplement.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: what source, what index, and how changes should propagate.
3. Authorize the connection with credentials scoped to the specific indices involved rather than cluster-wide privileges.
4. Describe the exceptions: a document that fails validation, a bulk request with some items rejected, a source record that was deleted.
5. Run it against a copy of the index before pointing anything at the one serving queries.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Change-driven indexing. Source changes propagate to the index, including deletions, which is the one most pipelines omit. Build the delete path at the same time as the create path rather than afterwards.

Reconciliation against the source. Compare document counts and sample contents between the index and the system of record, and report differences. This is the automation that finds divergence before a user does.

Reindex orchestration. Create the new index, reindex into it, verify, and swap the alias, as one described process rather than a sequence somebody performs by hand under pressure.

Failed-document reporting. Bulk indexing partially fails more often than people expect, and the response reports which items failed. An automation that does not read that is silently losing documents.

Query performance monitoring. Slow queries, by pattern, over time. Search degrades gradually as data grows and is noticed only when somebody complains.

Index size and growth reporting. Which indices are growing, how fast, and when they will become a problem, so scaling is planned rather than urgent.

Zero-result query reporting. Searches returning nothing, grouped by term. This is a product signal as much as a technical one, and it is frequently the most useful report the search team produces.

Reindexing without downtime

Worth describing precisely, because doing it by hand under pressure is where mistakes happen.

Never point your application at a concrete index. Point it at an alias. If your application knows the index name, zero-downtime reindexing is not available to you and fixing that is the prerequisite for everything below.

Create the new index with the new mapping, named with a version or a date.

Reindex from the old to the new, monitoring the task rather than guessing when it finished.

Keep writes flowing to both during the reindex, or capture and replay what changed while it ran. This is the step people skip and it is what produces a new index missing a day of updates.

Verify before swapping. Compare counts, run a handful of known queries against both, and check the results match expectations.

Swap the alias atomically, which points the name at the new index in one operation.

Keep the old index for a while. Rolling back is then changing an alias rather than reindexing again in a hurry.

Detecting drift

Compare counts first. Cheap, and it catches the large failures. A difference in total documents between source and index is unambiguous.

Sample and compare content. Counts agreeing does not mean documents match. Pull a sample by ID from both and compare the fields that matter.

Check for orphans specifically. Documents in the index whose source record no longer exists. This is the deletion failure, and counting alone will not find it if creates and deletes happen to balance.

Look at the oldest indexed timestamp. If every document carries when it was last indexed, the oldest one tells you whether anything has been missed entirely.

Run it on a schedule and report the differences rather than correcting them silently, at least at first. The differences tell you which part of the pipeline is failing, and silently fixing them hides the cause while treating the symptom indefinitely.

Building it so it survives

Use the bulk API and read its response. A bulk request can partially succeed, and the per-item results say which failed. An automation ignoring that loses documents quietly.

Make indexing idempotent. Use the source record's identifier as the document ID, so reindexing the same record overwrites rather than duplicating.

Handle version conflicts deliberately. Two updates to the same document arriving out of order should resolve predictably, which usually means an external version number from the source.

Scope credentials to indices. An automation with cluster-wide privileges can delete indices, which is more reach than an indexing job needs.

Report the outcome. Documents indexed, failed, and the current count. A pipeline that silently stops is discovered by a user searching for something that exists.

Limits worth knowing about

Bulk request size has practical limits, and very large batches are slower and more likely to fail than moderately sized ones.

Mapping changes are mostly not modifiable in place, so changing a field's type means reindexing. Designing mappings deliberately at the start saves this.

Field count limits exist, and dynamic mapping on unpredictable data can create a mapping explosion that degrades the cluster.

Refresh is not instantaneous. A document indexed is not immediately searchable, which catches automations that index then immediately query for the result.

Deep pagination is expensive, and anything walking a large result set should use the appropriate scrolling or search-after mechanism rather than paging by offset.

What to build first

Reconciliation against the source, read-only: compare counts, sample contents, and look for orphaned documents. No write access, and it usually finds something on the first run, most commonly deletions that never propagated.

Two habits make the difference. Report the differences rather than correcting them, because the differences tell you which part of the pipeline is broken and correcting them silently means fixing the same symptom forever. And run it often enough that the divergence is small when you find it, since a month of drift is far harder to explain than a day of it.

Frequently asked questions

Can Elasticsearch be my system of record?

Treat it as a search index over data that lives elsewhere. It has no transactions in the sense a database does, its consistency model is different, and it is optimised for query rather than for being authoritative. Using it as a primary store is a decision people regret at the moment they need to reindex.

How do I reindex without downtime?

Through an alias: create the new index, reindex into it while writes continue to both, verify, then swap the alias atomically. The prerequisite is that your application queries the alias rather than a concrete index name, which is worth fixing before you need it.

Why are deleted records still showing in search?

Because the indexing pipeline handles creates and updates but not deletions, which is the single most common gap. Build the delete path at the same time as the rest, and run an orphan check to find what was missed before you fixed it.

What is the most common silent failure?

Partial bulk failures. A bulk request returns success at the HTTP level while reporting per-item errors in its body, and an automation that checks only the status code loses those documents without any indication.

How do I know the index matches the database?

Reconcile on a schedule: compare counts, sample documents by ID and compare fields, and check for documents whose source record no longer exists. Counts alone can agree while the contents diverge, so the sampling matters.

Should the automation create indices and mappings?

Creating indices as part of a reindex is reasonable and well-defined. Changing mappings on a live index is mostly not possible anyway, and where it is, it deserves a deliberate decision rather than an automation doing it because a field appeared.

Why did my search results change after a mapping update?

Because mapping changes apply to newly indexed documents rather than retroactively, so the same query behaves differently depending on when each document was written. Getting consistent behaviour means reindexing everything under the new mapping.

How do I handle a mapping change on a large index?

Reindex into a new index with the new mapping and swap the alias, which is the only reliable route since most mapping changes cannot be applied in place. On a large index, plan for the reindex to take hours and for writes to continue to both during it.

What should I do about zero-result queries?

Report them grouped by term, weekly. They are a product signal as much as a technical one: people searching for something you do not have, or using words your content does not use. It is frequently the most actionable report the search stack produces and almost nobody runs it.

How often should reconciliation run?

Daily for anything customer-facing, weekly for internal search. The principle is that the drift should be small when you find it, since a day of divergence is a quick fix and a month of it is an investigation into which part of the pipeline was failing and when.

Should the automation write directly or through a queue?

Through a queue for anything at volume, because it absorbs bursts, makes retries straightforward, and stops a slow cluster becoming a slow application. Direct writes are fine for low volume and become the thing you regret during your first traffic spike.

Related reading

• CockroachDB and CodeWords
• BigQuery and CodeWords
• CodeWords integrations and templates