API rate limiting explained: both sides of the limit

Rate limiting caps how much a caller may ask for in a period. From the service's side it is protection: one badly behaved client cannot degrade the service for everybody else. From the caller's side it is a constraint that determines how an integration has to be written.

Most guidance covers one side. It is worth understanding both, because the reason a limit behaves the way it does explains what your client should do about it — and the commonest client behaviour, retrying immediately, makes the situation worse for everyone including the client.

What we'll cover

• Why services limit
• The algorithms, and what they feel like
• What the limit is counted against
• Being a good client
• The headers worth reading
• Designing a limit for your own API
• Rate limits across an automation estate
• Throttling and quotas are not the same thing
• Handling limits with CodeWords
• What to get right first
• Frequently asked questions

Why services limit

To protect capacity. One caller in a loop can consume resources that every other caller needs.

To contain the blast radius of a bug. A client with a retry loop and no backoff is indistinguishable from an attack, and usually is a bug rather than malice.

To make cost predictable, since serving requests costs money and unbounded consumption makes that unmanageable.

To enforce commercial tiers, which is legitimate and is the reason limits often vary by plan rather than by technical capacity.

To defend against abuse, including credential stuffing and scraping.

Understanding which of these a limit serves tells you something useful: a capacity limit may be negotiable with a conversation, a commercial one requires a different plan, and an abuse limit will not move.

The algorithms, and what they feel like

Fixed window counts requests per clock interval and resets. Simple, and it permits a burst at the boundary — twice the limit across two adjacent seconds — which is why it is less common.

Sliding window counts over a moving period, which removes the boundary burst and is more expensive to compute.

Token bucket refills at a steady rate up to a maximum. It permits bursts up to the bucket size and then throttles to the refill rate, which is why an integration can make fifty rapid requests and then find itself limited to a few per second.

Leaky bucket smooths output to a constant rate, queueing rather than rejecting, so requests are delayed instead of refused.

Concurrency limits cap simultaneous in-flight requests rather than requests per period, which is a different constraint and one that parallelism has to respect.

Recognising which you face explains otherwise confusing behaviour — particularly the token bucket's pattern of working fine and then abruptly not.

What the limit is counted against

Frequently more than one thing at once, which is why a client under the documented limit still gets rejected.

Per application, across all its users, which means your own users compete with each other.

Per user or per account, so one customer's heavy use does not affect another's.

Per endpoint, since an expensive endpoint may have a tighter limit than a cheap one.

Per organisation, shared with every other integration your company runs against that service — which is the one nobody accounts for.

By cost rather than count. Some APIs meter complexity, so a deeply nested query consumes more than a simple one even as a single request.

Several simultaneously. A request can pass the per-user limit and fail the per-application one, which is confusing until you know it.

Being a good client

Respect the retry hint. When the response says how long to wait, waiting that long is correct and ignoring it makes the limiting more aggressive.

Exponential backoff with jitter when there is no hint. The jitter matters: without it, every client that was limited retries in unison and the recovery is immediately overwhelmed again.

Never retry immediately. It is the default behaviour of naive code and it converts a brief limit into a sustained one.

Limit your own concurrency rather than discovering the service's limit empirically.

Cache what does not change. A large share of rate limit problems are the same data fetched repeatedly.

Batch where the API supports it, since one batched call is dramatically cheaper against your budget than fifty individual ones.

Spread scheduled work. Every integration in the world running at the top of the hour is a real phenomenon, and adding a few minutes of offset costs nothing.

The headers worth reading

Most APIs tell you the state of your budget and most clients ignore it.

The limit and the remaining count, which let you slow down before being refused rather than after.

The reset time, which tells you when the budget refreshes.

The retry hint on a rejection, which is the authoritative answer to how long to wait.

Read them proactively. A client that slows as it approaches the limit never experiences a rejection, which is a better outcome than handling one well.

Log them. When somebody asks why an integration is slow, knowing you have been running at ninety per cent of budget for a week answers the question immediately.

Designing a limit for your own API

Limit by account, not by address, since addresses are shared and spoofed.

Return a clear status and a retry hint, and document both. A rejection with no indication of how long to wait produces clients that guess, which is worse for you.

Publish the limit. Undocumented limits produce clients that behave badly because they cannot behave well.

Tier by cost of the operation rather than applying one number to everything, since a search and a lookup are not equivalent.

Provide a burst allowance. Rigid per-second limits are hostile to normal usage patterns, which are bursty.

Have an escape valve. A way for a legitimate caller to request a higher limit turns an adversarial relationship into a conversation.

Rate limits across an automation estate

The organisational version of the problem, and the one that catches growing teams.

Limits are usually per organisation, so every automation, script, and tool you run shares one budget.

Nobody owns the budget. Each team adds an integration reasonably, and the aggregate exceeds the limit with no single culprit.

A new automation can break an existing one, which is a confusing failure because nothing about the older one changed.

Inventory what calls what. Knowing which systems have integrations against them, and roughly how chatty each is, is the precondition for managing this.

Share the connection rather than duplicating it, where your tooling supports it, so the consumption is visible in one place.

Monitor consumption against the limit as a trend, so you find out before something fails rather than after.

Throttling and quotas are not the same thing

Two constraints that get discussed together and behave completely differently.

A rate limit is about speed. Wait and you can proceed, usually within seconds.

A quota is about total volume over a longer period — a day, a month — and waiting does not help once it is exhausted.

The retry logic that works for one is useless for the other. Backing off against a monthly quota simply delays the same failure until the end of the month.

Detect the difference from the response, which usually distinguishes them, and handle a quota by alerting a person rather than by retrying.

A quota approaching exhaustion deserves warning at eighty per cent rather than a failure at a hundred, because the remedies — buying more, deferring work, changing what you call — all take time.

Some limits are both. A per-second rate on top of a monthly quota is common, and a client needs to handle each appropriately rather than treating every rejection identically.

Handling limits with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused across automations, which keeps the consumption against each service visible in one place.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder.
3. Describe the exceptions: a rate limit, a service that is slow, a burst of work that needs pacing.
4. Connect each system once and reuse the connection.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to get right first

Backoff with jitter, and honouring the retry hint. It is a small amount of code, it is frequently absent, and its absence is what turns a momentary limit into a sustained outage as every client retries in unison against a recovering service.

Two habits make the difference. Read the remaining-budget headers and slow down before you are refused, since never being rejected is better than handling rejection well. And log your consumption as a proportion of the limit, because the moment somebody asks why an integration got slow, that number answers it without an investigation.

Frequently asked questions

Why am I rate limited when I am under the documented limit?

Because limits usually apply at several levels at once — per application, per user, per endpoint, and per organisation. You can be under one and over another, and the per-organisation budget shared with every other integration your company runs is the one nobody counts.

What should a client do when it is limited?

Honour the retry hint if there is one; otherwise back off exponentially with jitter. Never retry immediately — it converts a brief limit into a sustained one and it is the default behaviour of naive code.

Why does jitter matter?

Without it, every client limited at the same moment retries at the same moment, and the recovering service is immediately overwhelmed again. Randomising the delay spreads the recovery.

Why does my integration work and then suddenly throttle?

Probably a token bucket: it permits a burst up to the bucket size and then throttles to the refill rate. Fifty rapid requests succeed and the fifty-first waits, which is confusing until you recognise the pattern.

How do I avoid limits entirely?

Cache what does not change, batch where the API supports it, limit your own concurrency, and read the remaining-budget headers to slow down before being refused. Most rate limit problems are the same data fetched repeatedly.

How should I design a limit for my own API?

By account rather than address, tiered by operation cost, with a burst allowance, a clear retry hint, published documentation, and a route for legitimate callers to request more. Undocumented limits produce clients that cannot behave well.

Why did a new automation break an old one?

They share an organisation-level budget. Nothing about the old one changed; the aggregate consumption crossed the limit. This is why an inventory of what calls what is worth keeping before you need it.

Is a quota the same as a rate limit?

No. A rate limit is about speed and waiting resolves it; a quota is total volume over a longer period and waiting does not help. Backing off against an exhausted monthly quota just delays the same failure.

How should a quota be handled?

Alert a person rather than retry, and warn at around eighty per cent rather than failing at a hundred — buying more, deferring work, or reducing consumption all take longer than the remaining budget usually lasts.

Related reading

• API authentication methods
• API gateways
• OpenAI API rate limits