OpenAI API limits: context, output, spend, and model access
The limits that are not rate limits: context windows and what fills them, output token ceilings, budget caps, and which models your organization can actually reach.
Aymeric Zhuo10 min read
On this page
Four separate things get called "the OpenAI API limit", and they fail in completely different ways. Confusing them is why a developer spends an afternoon implementing retry logic for a problem that retrying cannot solve.
This page covers three of them: the context window, which governs how much you can send in one request; the output ceiling, which governs how much comes back; and the spend and access limits, which govern what your organization is permitted to do at all. The fourth, requests and tokens per minute, is a genuinely different problem with a genuinely different solution, and it is covered in OpenAI API rate limits.
What we'll cover
The four limits and how to tell them apart
The error tells you which one you have met, and the response to each is different.
A context window error arrives as a 400 saying the request exceeds the model's maximum. It is a property of the single request, it is deterministic, and retrying produces the identical error. The fix is to send less.
An output ceiling is not usually an error at all, which is what makes it insidious. The response simply stops, with a finish reason of length, and a truncated answer that may look complete.
A spend limit arrives as a 429 with a message about quota or billing rather than about rate. Retrying will not help until somebody changes a setting or adds funds. This is the one most often mistaken for a rate limit, because it shares the status code.
A rate limit arrives as a 429 about requests or tokens per minute. Waiting genuinely does resolve it, which is what separates it from the other three.
Read the message body rather than the status code. Two of these four return 429 and only one of them is worth retrying.
The context window
The context window is the total number of tokens a single request may involve, and the critical detail is that it covers input and output together. A model with a large window does not offer that much room for your prompt plus that much again for the answer. Everything shares the budget.
What consumes it, in rough order of how often people forget:
Conversation history. Chat applications resend the whole exchange with every turn, so a conversation grows its own token cost quadratically. This is the single most common cause of a context error appearing in an application that worked fine for weeks.
Retrieved documents. Anything pulled in from a search or vector store, which is easy to underestimate because the retrieval count is a configuration number rather than a token count. Ten chunks of twelve hundred tokens is twelve thousand tokens before the question is even asked.
System instructions. Usually modest, though elaborate instruction sets with examples can run to thousands of tokens on every single call.
Tool and function definitions. These are serialized into the request. A dozen tools with detailed schemas is a meaningful, fixed cost on every request whether the tools are used or not.
The reserved output. Space must remain for the answer. A request filling the window entirely with input leaves nothing to respond with.
As a working estimate, English text runs around four characters per token, so a thousand words is roughly thirteen hundred tokens. Code, structured data, and non-Latin scripts are considerably less efficient. For anything where the estimate matters, count properly with a tokenizer library rather than guessing, because the guess is always low.
When a request will not fit, the approaches are to summarize older conversation turns rather than resending them verbatim, to retrieve fewer and better chunks rather than more, to process long documents in sections with a final pass over the section summaries, or to move to a model with a larger window and accept the cost.
Output token ceilings
Separately from the window, each request carries a maximum number of tokens the model may generate, and each model has its own ceiling on that figure regardless of window size.
The failure mode deserves emphasis: truncation is silent. The API returns success. The response contains a well-formed beginning and simply stops. A truncated JSON object fails to parse, which at least announces itself; a truncated report reads as a finished report that ends slightly abruptly, and nothing flags it.
Check the finish reason on every response. A value of stop means the model finished. A value of length means it ran out of room and the content is incomplete. This one check catches an entire class of quiet data corruption, and it is two lines of code.
Where output genuinely needs to be long, ask for a structured outline first and then generate each section in its own request. This produces better results than one enormous generation in any case, since quality tends to degrade across very long outputs.
Spend limits and budgets
These are the limits your organization sets on itself, and they exist to stop a runaway loop producing a memorable invoice.
A monthly budget caps what the organization can spend in a billing period. Requests are refused once it is reached, with a 429 whose message refers to quota rather than rate.
Notification thresholds send an email at a chosen level of spend, which is worth configuring even when the cap is comfortable, because the first signal of a broken loop is usually a cost graph rather than an error.
Project-level limits allow a budget per project rather than per organization, which is how to stop an experiment consuming the production allowance. If several teams share an organization, this is the setting that prevents an awkward conversation.
Prepaid credit on some account types simply runs out, and the API stops until more is added.
The practical advice is to set a cap you are comfortable with rather than leaving it open, and to set the notification well below it. A cap with no notification means the first thing you learn is that production has stopped.
Model access and usage tiers
Not every model is available to every account. Access is governed by usage tiers, which an organization moves through based on cumulative spend and the time since its first payment.
Higher tiers bring larger rate limits and, at times, earlier access to newly released models. Newly created organizations start at the bottom, which is why a working example from a mature account can fail on a new one with a message about the model not existing.
A 404 or "model not found" on a model you know exists usually means one of three things: the organization's tier does not include it, the model requires separate access that has not been granted, or the identifier is wrong. Listing available models through the API settles which, and takes a moment.
When planning an application around a specific model, verify your organization can actually reach it before building. The exact tier thresholds and model availability change often enough that OpenAI's own documentation is the only reliable source, rather than any figure quoted in an article.
Limits on the endpoints nobody plans for
Chat completions get all the attention, and the limits that cause production surprises are usually elsewhere.
Embeddings carry their own input ceiling per request, and batching many texts into one call is the standard optimization until one oversized item fails the entire batch. Validate individual lengths before batching, and handle the oversized ones separately rather than letting one item fail a thousand.
File uploads are capped per file and per organization in total. Applications that upload documents for retrieval accumulate storage indefinitely unless something deletes them, and the ceiling arrives without warning because nobody was watching a number that only ever went up.
Audio transcription has a hard file size limit, which means longer recordings must be split before sending. The split needs to respect sentence boundaries or the transcript degrades at every seam.
Image generation and vision carry their own dimension and size constraints, and images count toward token usage in ways that are easy to underestimate when a workflow processes photographs of documents.
Structured output schemas have complexity limits. Deeply nested schemas with many optional fields can be rejected outright, which surfaces as a confusing validation error rather than as a limit.
The general lesson is that each endpoint has its own constraints and they are not documented in one place. When adding a new endpoint to a workflow, reading its specific limits section first costs five minutes and saves an afternoon.
Designing so limits do not bite
Count tokens before sending. A pre-flight check that measures the assembled request against the model's window, and trims history or retrieval when it will not fit, turns a production error into an automatic adjustment.
Check the finish reason every time. Treat length as a failure needing handling, rather than as a successful response.
Cap history explicitly. Decide how many turns to carry, or summarize beyond a threshold. Letting conversations grow without bound guarantees the error eventually.
Budget retrieval in tokens, not documents. Set the limit as a token allowance and fill it with the best chunks, rather than a fixed count whose size you do not control.
Separate projects by budget. Development and production should not share a cap, so that an experiment cannot stop the business.
Alert on spend before the cap. A notification at seventy percent is a Tuesday afternoon problem. The cap itself is an incident.
This is a reasonable amount of defensive plumbing, and it is the same plumbing on every project. Building on a platform where the model access is managed removes most of it: on CodeWords you describe what the automation should do and Cody, the automation builder, builds it, with model access provided rather than requiring your own keys and quota management. Automations connect to more than 3,000 integrations. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.
Frequently asked questions
Does the context window include the response?
Yes. Input and output share one budget, so room must remain for the answer. A request that fills the window entirely leaves the model nothing to reply with.
How do I count tokens accurately?
Use a tokenizer library for the model family you are calling, which gives exact counts. Character-based estimates are useful for rough sizing and consistently underestimate, particularly for code and structured data.
Why did my response stop halfway through?
The output ceiling was reached. The finish reason on the response will be length rather than stop. Either raise the maximum for that request, if the model permits, or restructure the work into several smaller generations.
What is the difference between a rate limit and a spend limit?
Both return 429. A rate limit is temporary and resolves by waiting, since it concerns requests per minute. A spend limit is a budget that has been reached, and waiting changes nothing until the period rolls over or the cap is raised. The message body distinguishes them.
Why can I not access a model that exists?
Most often the organization's usage tier does not include it yet. Tiers advance with cumulative spend and account age. Listing available models through the API confirms what your organization can actually reach.
Do longer contexts cost more?
Yes, directly. Pricing is per token, so a request carrying a large conversation history costs more than the same question asked fresh. Trimming history is a cost optimization as much as a reliability one.
Can I raise the context window?
Not for a given model, since it is a fixed property. The options are to move to a model with a larger window, or to restructure the work so each request carries less.