Datadog and CodeWords: automating around the alerts

Two problems dominate Datadog in practice, and neither is a gap in the product. The first is that the bill grows in ways nobody can trace to a decision. The second is that monitors accumulate until the alerts stop meaning anything, at which point the expensive observability platform has become a source of noise.

Both are addressable with automation, and both are mostly about reporting on Datadog rather than reacting to it.

What we'll cover

• Where the bill comes from
• What the Datadog API reaches
• Connecting it to CodeWords
• Seven automations worth building
• Alerts that mean something
• Enrichment, which belongs outside
• Tags, which decide whether any of this works
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Where the bill comes from

Worth understanding because the automation that pays for itself fastest is the one that reports on this.

Custom metric cardinality. A metric tagged with something high-cardinality — a user identifier, a request identifier, a full URL — multiplies into an enormous number of distinct time series. One careless tag can dominate a bill, and the person who added it had no idea.

Log ingestion versus indexing are charged separately, and logs can be ingested for processing without being indexed for search, which is the lever most teams do not know they have.

Host and container counts move with autoscaling, so a spike in load has a cost tail.

Retention settings apply per index and per metric, and defaults are rarely revisited.

APM span volume grows with traffic and with instrumentation added and forgotten.

A weekly report on custom metric cardinality, by metric and by tag, is the single most valuable automation here and it is read-only.

What the Datadog API reaches

Metrics can be queried and submitted, including custom metrics from your own systems.

Monitors can be created, read, updated, muted, and their state inspected, which is what any monitor hygiene work uses.

Events can be posted and queried, and posting deployment and change events into Datadog is the cheapest way to make graphs interpretable.

Logs can be searched and aggregated, with the usual retention caveats.

Dashboards can be created and modified programmatically, which suits generating a consistent dashboard per service.

Incidents can be created and updated where you use Datadog's incident management.

Downtimes can be scheduled, so planned work does not generate alerts.

Usage endpoints report what you are consuming, which is the basis of any cost reporting.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which monitors or metrics, and what should happen when something fires.
3. Authorize the connection with an API key and an application key scoped appropriately.
4. Describe the exceptions: a monitor with no owner, an alert during a deployment, a metric that has stopped reporting.
5. Test against a non-critical monitor before anything touches the on-call path.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Cardinality and cost reporting. Which custom metrics have the most series, which tags are driving it, and what changed this week. Read-only and it routinely finds one metric responsible for a startling share of the total.

Monitor hygiene reporting. Monitors with no owner, monitors that have never fired, monitors that fire constantly, and monitors muted indefinitely. All four are signals that something should be deleted or fixed.

Alert enrichment. When a monitor fires, gather what the responder would otherwise gather — recent deployments, related alerts, the relevant dashboard link, whether this is a recurrence — and attach it to the notification.

Deployment events into Datadog. Post an event on every deploy so graphs show what changed. The cheapest possible improvement to incident investigation.

Downtime scheduling from your change calendar. Planned maintenance should not page anybody, and doing this by hand means somebody forgets.

Dashboard generation per service. A consistent dashboard created from a template when a new service appears, so observability arrives with the service rather than after the first incident.

Silent metric detection. A metric that has stopped reporting is invisible to threshold monitors, because no data is not a breach. This is the failure mode that costs most and alerts least.

Alerts that mean something

Alert fatigue is a design problem, and the design decisions are few.

Alert on symptoms, not causes. Users experiencing errors is an alert. A single host at high CPU is usually not.

Every alert needs an owner and an action. If nobody knows what to do when it fires, it is a dashboard item rather than an alert.

Alert on absence. A monitor on a threshold cannot fire when data stops arriving, and a pipeline that silently stopped is the classic undetected outage. Datadog supports no-data conditions; configure them deliberately rather than leaving the default.

Set recovery thresholds apart from alert thresholds, or a metric hovering at the boundary produces a stream of notifications.

Review what fired. A monthly report of alert frequency by monitor, with how many led to action, is uncomfortable reading and the fastest route to a quieter on-call rotation.

Enrichment, which belongs outside

Datadog knows about your infrastructure. It does not know about your customers, your deployments, or your business calendar, and that is where external automation adds something Datadog cannot.

Which customers are affected. Error rates are abstract; three named accounts are actionable and change how an incident is handled.

What changed recently. Deployments, feature flag changes, and configuration updates, gathered from the systems that hold them.

Whether this is a recurrence. Previous incidents with the same signature, and what resolved them, which is the single most useful thing to put in front of a responder.

What it is costing. For anything customer-facing, an estimate of affected volume helps prioritisation.

Who to tell. Support and account teams usually learn about incidents from customers, and they should not.

Tags, which decide whether any of this works

Datadog's tagging model is the thing that makes cross-cutting queries possible, and it degrades without anybody noticing.

Unified service tagging — service, environment, and version on everything — is what lets metrics, traces, and logs line up for the same request. Without it you have three separate tools that happen to share a login.

Version tags make deployments visible in the data, which is how you tell a regression from a coincidence.

Team or owner tags decide who gets told. An alert that cannot be routed automatically gets routed by whoever is watching, which is not a system.

Cost allocation depends on tags. Attributing spend to a team is impossible without them, and impossible to retrofit accurately.

Tag drift is constant. New services arrive without the standard tags, and a report listing what is missing keeps it from becoming permanent.

An automation checking tag conformance across services and reporting the gaps is dull, quick, and makes everything else on this list possible.

Building it so it survives

Do not alert on your alerting. An automation reacting to monitors should degrade quietly rather than generating its own page when it fails.

Handle at-least-once delivery. Webhooks from monitors can repeat, so key on the monitor and the transition rather than reacting blindly.

Respect downtimes. Automation reacting to alerts should check whether a downtime is active, or planned maintenance produces exactly the noise the downtime existed to prevent.

Scope keys carefully. Application keys carry the permissions of the user who created them, which is a trap when that user is an administrator and the automation only needs to read.

Report the outcome. What fired, what was enriched, and what the automation skipped.

Limits worth knowing about

Rate limits apply per endpoint and differ between them, with log and metric queries generally tighter than configuration endpoints.

Log search is bounded by retention and indexing. Logs ingested but not indexed are not searchable, which is a deliberate cost trade and a surprise when you first hit it.

Metric queries are subject to rollup. Over long windows, points are aggregated, so a query over a month is not the same resolution as one over an hour.

Application keys inherit their creator's permissions, so an automation may quietly have far more access than intended.

Some data has a delay before it is queryable, which matters for automation that reacts immediately and then asks for the numbers.

What to build first

Custom metric cardinality reporting: which metrics have the most time series, which tags drive them, and what changed week on week. It writes nothing, it needs only read access, and it very commonly identifies a single tag added months ago that accounts for a large share of the metrics bill.

Two habits make the difference. Report the change as well as the total, since a new high-cardinality tag matters more than a longstanding one. And send it to the team that owns the service rather than to a central channel, because only they can decide whether the tag is worth its cost.

Frequently asked questions

Why is our Datadog bill rising with no obvious change?

Usually custom metric cardinality — a tag with many distinct values multiplying into large numbers of time series. Autoscaling, retention defaults, and added instrumentation account for most of the rest. A weekly cardinality report makes the cause visible.

How do I detect a metric that has stopped reporting?

With a no-data condition, deliberately configured. Threshold monitors cannot fire on absence, which is why a pipeline that silently stopped is the outage nobody gets paged for.

Should automation acknowledge or resolve alerts?

No. Acknowledgement is a statement that a person is handling it, and automating it removes the signal that matters. Automate the gathering of context instead, which is the part that wastes a responder's time.

What context is most useful to attach to an alert?

Recent deployments, which customers are affected, and whether this has happened before with what resolution. All three come from systems outside Datadog, which is exactly why external automation is the right place for this.

Why does my automation have more access than expected?

Application keys carry the permissions of the user who created them. A key created by an administrator can do everything that administrator can. Create keys with an account scoped to what the automation actually needs.

How do I stop planned maintenance paging people?

Schedule downtimes from whatever holds your change calendar, and have any alert-reacting automation check for an active downtime before doing anything. Doing this by hand works until the one time somebody forgets.

What should a monitor review report include?

How often each monitor fired, how many firings led to action, and which have no owner. It is uncomfortable reading and it is the fastest route to an on-call rotation people can sustain.

Why do our metrics, traces, and logs not line up?

Almost always inconsistent tagging. Service, environment, and version applied uniformly are what connect the three, and services that arrived without them stay disconnected until somebody notices. A conformance report catches it while it is still one service.

Can we attribute Datadog spend to teams?

Only if the tags support it. Ownership tags applied consistently make allocation straightforward; without them it is guesswork, and it cannot be reconstructed after the fact for data already collected.

Should we send logs we never search?

Sometimes, and know that you are. Ingestion and indexing are billed separately, so logs can be processed into metrics without being indexed for search. Deciding that deliberately per log source is one of the larger cost levers available and one of the least used.

Related reading

• PagerDuty and CodeWords
• Sentry webhooks
• CodeWords integrations and templates