Best LLM orchestration frameworks: what each is actually for
LangChain, LlamaIndex, DSPy, Haystack and the agent frameworks solve different problems. Which fits retrieval, which fits agents, and when a framework is the wrong answer entirely.
Aymeric Zhuo11 min read
On this page
The frameworks in this space are frequently compared as though they are competing implementations of one idea. They are not. Some are retrieval libraries, some are agent runtimes, some are prompt optimizers, and one of the most common right answers is to use no framework at all.
So the first question is what you are actually building, because that determines which category you need before any comparison within a category becomes useful.
What we'll cover
Four different problems, four kinds of tool
Getting the right context in front of the model. Chunking documents, embedding them, storing them, retrieving the relevant parts, reranking. This is retrieval, and it is where most production value in this space sits.
Composing several model calls into a pipeline. Chaining steps, branching on output, parsing structured results, handling failures. This is orchestration proper.
Improving prompts systematically rather than by trial and error, using examples and a measurable objective.
Running loops where the model decides what to do next, calling tools and deciding when it is finished. This is agents, and it is the category with the widest gap between demonstration and production.
A project can need all four, and most need one. Buying a framework for all four when you need one is the most common source of regret here.
When you need no framework
Worth stating first, because it is underrated and frequently correct.
If your application makes one model call, or a handful of calls in a fixed order, the provider's own SDK is sufficient. You will write perhaps thirty lines that a framework would have written for you, and in exchange you will understand every one of them, you will have no abstraction between you and the API, and you will not inherit a dependency that changes its interface.
The signs that you genuinely want a framework:
You are implementing retrieval and it is getting involved. Chunking strategies, embedding management, hybrid search, reranking. This is real work that libraries do well.
You have many steps with branching and failure handling and you are starting to build the plumbing yourself.
You want to swap models or providers without rewriting your call sites.
You need evaluation and want something that structures it.
The signs you do not:
You want a demo working quickly. Frameworks help here and the cost arrives later, when the abstraction that got you started obscures what is actually being sent to the model.
Because the tutorials use one. The tutorials optimize for a short tutorial.
A useful discipline whichever you choose: be able to see the exact prompt that reaches the model. Frameworks that make this hard will cost you hours during the first debugging session, because prompt problems are the most common problems and you cannot fix what you cannot read.
Retrieval-focused: LlamaIndex and Haystack
LlamaIndex is organized around getting data into a model's context. Loading from many sources, chunking, indexing, retrieving, reranking. If your application is fundamentally "answer questions about our documents", this is the most direct fit, and its abstractions are shaped like the problem rather than like a general pipeline.
Haystack covers similar ground with a more explicit pipeline model, composing components into a graph. Teams who want retrieval with visible structure, and who value production concerns like deployment and evaluation being treated seriously, tend to prefer it.
Both are worth it when retrieval is genuinely the hard part of your application, which it usually is when the answer quality depends more on finding the right passage than on the model's reasoning.
Neither is worth it when you have twenty documents. Embedding twenty documents and doing a similarity search is a hundred lines of code and no dependency, and you will understand it completely.
General composition: LangChain
The broadest of these, and the most argued about.
What it gives you: a very large ecosystem of integrations, a composition model for chaining steps, structured output handling, memory abstractions, and connections to essentially every vector store, model provider and tool you might want. The ecosystem is the genuine advantage and it is a large one.
The recurring criticism: abstraction depth. Several layers between your code and the API call means that when something behaves unexpectedly, understanding why involves reading the framework rather than reading your own code. The interface has also moved considerably over its life, so material written eighteen months ago frequently does not apply.
The fair assessment: it is the most capable option when you want breadth, and it asks you to accept its way of structuring things. Teams that adopt it and stay happy generally use a narrow slice of it deliberately. Teams that adopt it and regret it generally used it as a default rather than a choice.
Worth knowing: its observability tooling is a separate commercial product and is genuinely useful for seeing what reached the model, which partly addresses the abstraction complaint.
Optimization: DSPy
A different idea from the others, and the most interesting if it matches your problem.
Rather than writing prompts by hand and tuning them by intuition, DSPy has you declare what a step should do, supply examples, and define a metric. It then optimizes the prompts against that metric.
Worth it when you have a task with measurable correctness and a set of labelled examples, and where prompt quality materially affects results. Classification and extraction are the natural fits.
Not worth it when you cannot define a metric, which is most open-ended generation. If you cannot score the output automatically, there is nothing for it to optimize against.
The requirement people underestimate: you need real examples with known good answers. Twenty to a hundred, properly labelled. That is an afternoon's work and it is the actual cost of adopting it, not the library.
Agent runtimes
Frameworks for loops where the model chooses the next action: CrewAI, AutoGen, LangGraph, and the agent capabilities inside the major provider SDKs.
What they give you: tool-calling loops, multi-step reasoning, state between steps, and in some cases several models coordinating.
The honest caveat, and it applies across the category: the distance between an impressive demonstration and something you can rely on in production is large. Agents fail in ways that are hard to bound. They loop, they take unexpected actions, they cost unpredictably because the number of model calls is not fixed in advance, and testing them is genuinely difficult because the same input does not produce the same path.
When they earn their place: genuinely open-ended tasks where the steps cannot be known in advance, with a person reviewing the output, and with hard limits on iterations and spend.
When a plain workflow is better: almost every business process. If you can describe the steps, describe the steps. A deterministic pipeline with a model doing the two steps that need judgement is cheaper, faster, testable, and produces the same answer every time — which is what most processes actually require.
That distinction is worth being firm about, because "agent" is doing a great deal of marketing work at the moment. A process with a right answer wants an automation, not an agent. On CodeWords you describe the process in plain language and Cody, the automation builder, builds it, connects it to your systems, and deploys it, with models used for the steps that need judgement rather than for deciding the shape of the work. Automations connect to more than 3,000 integrations. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.
What actually decides it
Can you see the prompt? Whichever you choose, verify early that you can inspect exactly what is sent. This predicts your debugging experience more than any feature.
How much of the framework will you use? Adopting a large framework for one capability means inheriting its upgrade path, its breaking changes, and its opinions, in exchange for a fraction of its value.
Does it version well? Prompts, chains and configurations should live in your repository under review, not in a platform's interface. This matters more than it sounds when someone changes a prompt and quality drops for a fortnight before anyone notices.
Can you evaluate? Whatever you build, you need a set of examples with known answers and a way to run them. The framework that makes this natural will save you more than any convenience feature.
What happens when you want a different model? The abstraction should make this cheap. If it does not, you have taken on the cost without the benefit.
A sensible order to adopt any of this
Most teams arrive at frameworks too early and regret the abstraction, or too late and have rebuilt half of one badly. A workable sequence:
Start with the provider SDK and no framework. Get one call working, see the exact prompt and the exact response, and understand the shape of the problem before choosing an abstraction over it.
Add evaluation before adding capability. Twenty to fifty real examples with known good answers, and a script that runs them. Do this second, not last. Every decision after this point becomes measurable rather than a matter of opinion, and this single step separates teams who improve steadily from teams who change prompts and hope.
Add retrieval only when context is the bottleneck. If the model is answering badly because it lacks information, retrieval helps. If it is answering badly because the task is hard, retrieval will not, and a better prompt or a better model will.
Adopt a framework at the point you are writing its features yourself. When you notice you have built chunking, reranking and a retry loop by hand, a library does those better. Before that point you are buying abstraction you do not need.
Keep the prompts in your repository. Under version control, reviewed like code, whatever else you adopt. A prompt changed outside review is a production change nobody saw.
Frequently asked questions
Do I need a framework at all?
Often no. One model call or a fixed sequence of calls is well served by the provider's SDK and thirty lines of your own code. Reach for a framework when retrieval gets complicated, when you have many branching steps, or when you want to change providers without rewriting.
Which is best for retrieval-augmented generation?
LlamaIndex if retrieval is the centre of the application, Haystack if you want explicit pipelines and production tooling, and neither if you have a small document set, where a direct implementation is simpler and entirely adequate.
Is LangChain still worth using?
Yes, when you want its ecosystem breadth and adopt a deliberate slice of it. The criticism about abstraction depth is fair and mostly bites teams who used it by default rather than by decision. Confirm early that you can see the prompts.
What is the difference between an agent framework and a workflow tool?
An agent decides what to do next; a workflow does what you described. Agents suit open-ended tasks with a person reviewing output. Workflows suit processes with a right answer, which is most business work, and they are cheaper, testable and reproducible.
How do I control cost with agent frameworks?
Cap iterations, cap spend per run, and log every model call with its token usage. The cost problem with agents is that the number of calls is decided at run time rather than by you, so the limits have to be explicit or there are no limits.
Should prompts live in code or in a platform?
In code, under version control and review, with a platform used for observing what happened rather than for storing the source of truth. Prompts are logic, and logic that lives outside your repository changes without anyone seeing a diff.
How do I test any of this?
Build a set of real examples with known good answers, twenty to fifty, including the awkward cases. Run them whenever anything changes: the prompt, the model, the retrieval settings. This is the single highest-value practice in the whole area and it needs no framework.