How to automate API health checks that catch real failures

Most API health checks answer a question nobody asked. A endpoint returning 200 means the process is running and accepting connections. It does not mean the database is reachable, the third party you depend on is responding, or that the thing your customers actually do still works.

So the check passes through outages. The dashboard is green, the alerts are quiet, and the first report comes from a customer — which is the specific failure worth designing against.

What we'll cover

• What a shallow check misses
• Assert on the journey, not the endpoint
• Where to check from
• Building it with CodeWords
• Seven checks worth building
• Thresholds that survive a bad afternoon
• Checking your dependencies
• Who the alert wakes, and what it says
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

What a shallow check misses

A liveness endpoint that returns a constant tells you a process is up. That is genuinely useful to an orchestrator deciding whether to restart something, and almost useless as monitoring.

The database being unreachable does not stop that endpoint returning 200, unless it checks.

A dependency timing out likewise — the service is running, it simply cannot do anything.

A deployment that broke authentication leaves health checks passing while every real request is rejected.

An empty response is the quietest failure of all: the endpoint answers, the status is fine, and the body contains nothing because a downstream query returned no rows.

Certificate expiry takes everything down at a moment nobody is watching, and no status-code check anticipates it.

Assert on the journey, not the endpoint

The change that turns a check into monitoring.

Pick the thing customers actually do — sign in, search, add to basket, submit a form — and check that.

Assert on the body, not the status. A response containing the field you expect, with a plausible value, is evidence; a 200 is not.

Check a value, not just a shape. A search returning zero results with a 200 is a broken index, and only a check that asserts "more than zero" catches it.

Include authentication. A check that skips the auth layer skips the thing most likely to break after a deployment.

Measure the duration and assert on it, since a response that takes eleven seconds is a failure to a user even when it eventually arrives.

Keep it read-only where you can. A check that creates data needs cleaning up, and a check that cleans up badly leaves test records in production.

Where to check from

Inside the network tells you the service is healthy. Useful, and it cannot see what a customer sees.

Outside exercises DNS, certificates, the load balancer, the CDN and whatever else sits in front — which is where a meaningful share of outages live.

From more than one region where you serve more than one, because a regional failure looks like nothing at all from the region that still works.

From where your customers are. A check that runs from the same continent as the service will not notice latency that makes the product unusable elsewhere.

Both is the right answer, with different alerting: an internal failure is an engineering problem and an external-only failure is usually networking, certificates, or a provider.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so a check can assert against your own systems and report wherever the responder actually is.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which journey to exercise, what a correct response looks like, and how often.
3. Authorize the connections it needs — the service, and wherever the alert should go.
4. Describe the exceptions: a slow response, an empty result, a check that fails during a known deployment.
5. Run it against a staging environment and confirm it fails when you break something deliberately.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven checks worth building

The critical journey, end to end, asserting on the content of the response. The one check to have if you only have one.

Authentication specifically, since it breaks on deployment more than anything else and fails in a way that looks like a client problem.

Certificate expiry, with warning at thirty days rather than an alert on the day. This is the most preventable outage there is.

Each third-party dependency, checked independently, so you find out it is their problem before spending an hour assuming it is yours.

A write path, where one exists, in a way that is safe to repeat — because read-only checks pass through a database that has gone read-only.

Scheduled jobs reporting in. A job that stopped running produces nothing, and nothing looks exactly like a quiet period.

The error rate from your own traffic, which catches what synthetic checks cannot: a failure affecting one customer segment, one client version, or one region.

Thresholds that survive a bad afternoon

Alert on consecutive failures, not on one. A single timeout is normal on any network, and alerting on it teaches people to dismiss alerts.

Two or three in a row, from more than one location, is a signal.

Use percentiles for latency, not averages. The median hides a problem affecting a minority; the ninety-fifth percentile does not.

Compare against the same hour on the same weekday, since traffic patterns make a flat threshold wrong twice a week.

Have a different threshold for degraded and down. Slow and broken need different responses, and collapsing them means the urgent one arrives with the same weight as the routine one.

Suppress during deployments deliberately, with a window, rather than letting every release produce alerts people learn to ignore.

Checking your dependencies

The half that saves the most time during an incident.

Check each third party you depend on directly, with your own credentials, rather than relying on their status page.

Status pages lag. They are updated by a person after the incident is understood, which is frequently an hour after you noticed.

Check the specific operation you use, not their homepage. A provider can be up while the one endpoint you depend on is failing.

Report it as their outage, distinctly, so the responder does not start by reading your own logs.

Track how often each one fails, because that number is the argument for a fallback, a cache, or a different provider — and nobody has it when the conversation happens.

Who the alert wakes, and what it says

A check that fires at three in the morning is only useful if the message lets somebody act.

Say what failed, in the journey's terms. "Checkout search returned zero results" is actionable; "probe 14 failed" is a prompt to go and read a dashboard.

Include the assertion that failed and the actual value, so the responder knows whether it is a slow response or an empty one.

Say whether it is internal, external, or a dependency, since those three point at different first steps.

Name the recent deployments. A check that started failing eleven minutes after a release has told you most of what you need.

Route by what it is. A dependency outage goes to whoever manages that relationship as well as to on-call; a certificate expiry goes to whoever owns DNS.

Say when it recovers. A channel that reports failures and never recoveries leaves everybody assuming things are still broken.

Making it survive

Make the check itself reliable. A monitoring automation that fails silently is worse than none, so it should report that it ran even when everything passed.

Do not run the check from the thing being checked, which is the classic mistake — when the service is down, so is its monitoring.

Rotate the credentials it uses like any other, and alert distinctly when the check fails on authentication rather than on the service.

Keep the check's own runtime short, so a slow check does not look like a slow service.

Report the outcome. Checks run, passed, failed, and the latency distribution.

Limits worth knowing about

Synthetic checks see what they test. They will not find a bug affecting one customer's data shape, which is what real-traffic error rates are for.

Rate limits apply to your own API too, and a check running every ten seconds against an endpoint with a tight limit will consume budget that matters.

A check that writes needs cleanup, and cleanup that fails leaves debris that eventually confuses somebody.

Third-party checks consume their quota, which for a paid API is a real cost at high frequency.

Certificates renew automatically until they do not, so the expiry check is worth having even where renewal is automated.

What to build first

One check that exercises your most important customer journey, asserting on the response body, from outside your network, alerting after two consecutive failures. It is an afternoon's work and it catches the category of outage that shallow checks pass straight through.

Two habits make the difference. Assert on content rather than status code, since a 200 with an empty body is the failure that gets missed. And break something deliberately to confirm the check fails — a check that has never failed is usually a check that cannot.

Frequently asked questions

Why did monitoring stay green during an outage?

Because the check asserted on a status code rather than on what the response contained. A liveness endpoint returns 200 while the database is unreachable, authentication is broken, or a query returns nothing.

What should a health check assert?

The content of a response from a real customer journey, including authentication, with a plausible value rather than just a shape — and a duration threshold, since a response that takes eleven seconds has failed for the user.

Should checks run inside or outside the network?

Both. Internal tells you the service is healthy; external exercises DNS, certificates, and everything in front of it, which is where a substantial share of outages actually are. Alert on them differently.

How many failures before alerting?

Two or three consecutive, from more than one location. A single timeout is normal on any network, and alerting on one teaches people that alerts can be ignored.

Should I trust a provider's status page?

Not as your first signal. They are updated by a person after an incident is understood, often an hour after you noticed. Check the specific operation you depend on, with your own credentials.

What is the most preventable outage?

Certificate expiry. Warn at thirty days rather than alerting on the day, and keep the check even where renewal is automated — automated renewal fails quietly often enough to justify it.

How do I know the check itself is working?

Have it report that it ran even when everything passed, and break something deliberately to confirm it fails. A check that has never failed is usually one that cannot.

What should the alert message contain?

The journey that failed in plain terms, the assertion that failed with the actual value, whether it was internal, external, or a dependency, and any deployment in the preceding minutes. That combination usually identifies the cause without opening anything.

Should recoveries be announced?

Yes. A channel that reports failures and never recoveries leaves people assuming the problem is ongoing, which produces duplicated investigation and a lot of unnecessary checking.

Related reading

• Automating website uptime monitoring
• Automating synthetic monitoring
• Automating Slack alerts from monitoring tools