A multi-agent system beat a single agent by 90.2% on Anthropic’s internal research evaluation while using about 15 times the tokens of a chat interaction. AI agent orchestration is the orchestration layer behind that result: an orchestrator coordinates multiple AI agents across complex workflows. Specialized agents share memory and hand tasks over through agent communication that the coordination layer controls. This guide shows when multi agent orchestration pays off for enterprise systems and which agent orchestration patterns suit your architecture and your risk.
AI agent orchestration: what it is and when a single agent is enough
What AI agent orchestration means
AI agent orchestration is the coordinating of multiple specialized agents so that they complete one workflow together. An orchestrator such as a manager agent decides which agent acts next and what shared context it receives. In multi agent systems it acts as the control plane with human oversight. Centralized orchestration and concurrent orchestration route each request between tools and data in real time for customer support or sales.
Agent to agent messages carry each task. Autonomous agents with no such coordination behave like a group chat with no chair.
How orchestration differs from a workflow and from a single agent with tools
A classic workflow runs a fixed sequence of steps. It suits refunds and approvals where the path is known and each run should behave the same way. Automation of that kind needs no autonomous decision. A single agent with tools chooses its own steps but works alone inside one context window. It can call a database or create a quote through an API without any coordination problem. Its behavior is open to evaluation because one component makes every decision.
Orchestration begins when several agents with distinct responsibilities must cooperate on one task. The workflow automation may still exist inside it and complex workflows often mix both. A fixed flow can handle the payment step while agents handle the open questions. Microsoft’s Azure Architecture Center describes three levels of complexity: a direct model call, a single agent with tools and multi-agent orchestration. Start at the lowest level that solves the problem.
What coordination costs in tokens
Anthropic reported the gain and the bill in the same article. Typically a team sees the first and forgets the second. Its multi-agent research system, with a lead model and smaller subagent models, outperformed a single-agent baseline by 90.2% on an internal evaluation. Agents used about four times the tokens of a chat interaction and multi-agent systems about 15 times. On the BrowseComp benchmark token usage alone explained 80% of the variance in performance.
Did you know?
Anthropic found that multi-agent systems pay off for valuable tasks that parallelize well and exceed one context window. The same article warns that they are a poor fit when agents depend on each other at every step.
Read that result as a pricing signal. More agents buy more reasoning and more parallel work. Every extra agent also adds model calls that you pay for on each run. Orchestration therefore belongs on tasks where a better answer is worth many times the cost of a chat reply.
Why enterprise interest in multi-agent systems is rising
Gartner recorded a 1,445% rise in client inquiries about multiagent systems between the first quarter of 2024 and the second quarter of 2025. It also expects that by 2027 about 70% of multiagent systems will rely on agents with narrow responsibilities. Narrow responsibilities improve accuracy but raise the amount of coordination the system must handle across an entire operation.
Demand and failure rates grow together. Gartner also predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of cost and unclear value. Our guide to agentic AI covers that risk from the strategy side. This guide stays on the architecture that decides whether a multi-agent system survives.
Single agent or multi-agent: a five-question selector
Microsoft calls a single agent with tools "often the right default for enterprise use cases". Multi-agent orchestration adds coordination overhead and latency and new failure modes. It earns its place when one agent fails on prompt complexity or tool overload or when security requirements force separate boundaries per agent. Every requirement should be explicit before you build.
The selector below turns that advice into a decision. Answer five questions about your task. You get a recommended level and a pattern to test first. You also get an indicative token multiplier based on the Anthropic figures.
If the tool recommends a single agent you have saved money and a failure surface. If it recommends orchestration the next sections explain how the coordination works.
How multi-agent orchestration works: roles, routing, shared state and handoff
The orchestrator and the worker agents
The orchestrator plans the task and delegates each part. It may be a manager agent powered by a language model or a deterministic program with its own logic. Worker agents execute one bounded job each. A pricing agent reads the catalog. A compliance agent checks a rule. A drafting agent writes the answer.
Give each worker a single responsibility and a written contract. A structured specification keeps development of each agent independent. The contract states the inputs it accepts and the output format it returns. Typed contracts let you swap one agent for another without retesting the whole system. They also make a failed step visible at the boundary where it happened. Contracts are the lowest-cost way to explore where a coordinating agent loses information.
Routing: how a task reaches the right agent
Routing decides which agent receives a request. Deterministic routing uses rules and queues. A request tagged as a refund always goes to the refund agent. Model-based routing lets a language model read the request and pick the agent. The first is predictable and low in cost. The second handles unclear requests but adds a model call and a new source of error. Each approach should include a default path when no agent matches.
Most production systems mix both. Rules route the common cases and a model route handles the rest. Log every routing decision with its reason. A wrong route is the first place to look when an answer drifts from what the user asked.
Shared state and memory across agents
Agents that cannot see the same facts contradict each other. Shared state is the record of the task that every agent can read and update. It holds the goal and the decisions already made and the results so far. Memory adds what persists beyond one run, such as customer preferences or the history of earlier conversations. Persistence usually relies on a database and often on vector search and retrieval for long documents. Retrieval quality is a feature to test because search errors reach every downstream agent. Teams call this retrieval-augmented generation or RAG and good practice is to log which source each answer used.
Keep state outside the agents. Store it in a persistent layer with versions and checkpoints. The same shared context lets every agent read the same facts in real time. If a worker crashes the orchestrator restores the last checkpoint instead of restarting the task. Context passed only through prompts is lost at the first failure and drifts as the conversation grows.
Handoff policies: what must travel with the work
A handoff passes control from one agent to another or to a person. The receiving side needs more than the last message. A sound policy defines the payload:
- the goal of the task and what is already done
- the relevant shared state and its version
- the access rights the receiving agent may use
- the trace identifier that links the new step to the previous one
Add a rule for refusal. The receiving agent must be able to say that it cannot take the task and return it with a reason. Without that rule a handoff loop can run until the budget is gone.
Agent communication: MCP for tools and A2A for agents
Two open protocols standardize how agents connect. MCP, the Model Context Protocol, links an agent to tools and data sources. In December 2025 the Linux Foundation created the Agentic AI Foundation with MCP as one of its founding projects. A2A, the Agent2Agent protocol, links one agent to another. It became a Linux Foundation project in June 2025 and reached version 1.0 in March 2026, with more than 150 supporting organizations counted at its first anniversary in April 2026.
In short MCP carries the call from an agent to a tool while A2A carries the task between agents. A multi-agent system can use both. Standards reduce custom glue code and software integrations but do not replace state and access control. Agent communication still needs rules about who may call what. Our article on MCP and A2A for enterprise AI search covers the protocol details.
Human in the loop: in, on or out
Three postures describe where people sit. A person in the loop approves each consequential action before it runs. A person on the loop supervises and can interrupt. A person out of the loop reviews results afterward through audit trails.
Choose per action and not per system. Reading a catalog needs no approval. Sending a binding quote may need one. Define approval gates for high-risk actions and give approvers their own access rights. Every human in the loop decision belongs in the trace next to the agent decisions.
Orchestration patterns: centralized, decentralized, hierarchical and beyond
Centralized orchestration with a manager agent
In centralized orchestration one manager agent receives the request and routes work to specialists. It then merges the answers. This pattern gives the clearest audit trail and the simplest governance because every decision passes through one point.
The same point is its weakness. A single point of failure limits throughput and one faulty manager affects every task. Use centralized orchestration when auditability matters more than raw scale because the architecture stays manageable to operate. Add a fallback manager or a restart policy so that one crash does not stop the workflow.
Decentralized and federated orchestration
In decentralized orchestration agents coordinate as peers with no manager. They negotiate who takes a task and share results directly. The design resists the loss of one node but conflicts need explicit resolution rules. Two agents that both claim a task can duplicate work or block each other.
Federated orchestration connects separate systems that keep their own data. Each organization runs its own agents and exposes only what the exchange needs. It suits cases where privacy or contracts forbid pooling information in a distributed environment. Expect more integration work and slower coordination than inside one organization.
Hierarchical orchestration
Hierarchical orchestration layers delegation. A top-level orchestrator splits a goal into domains and passes each domain to a sub-orchestrator. Each sub-orchestrator manages its own workers. The structure mirrors an enterprise that already has departments with their own rules.
It scales better than one manager because no single agent tracks every worker. Management of each domain stays local. The cost is more handoffs and more places where context can thin out. Keep the depth to two or three levels and pass the goal intact at each handoff.
Sequential and concurrent orchestration
Sequential orchestration chains agents in a fixed order. Each one builds on the previous output. It suits staged processing such as draft then review then approval. Avoid it when early errors propagate or when one stage can run without waiting.
Concurrent orchestration sends the same task or independent subtasks to several agents at once and aggregates the results. It cuts latency and brings several viewpoints. Avoid it when agents need each other’s output or when merging conflicting answers has no clear rule. Concurrency also multiplies token use at the same moment.
Group chat and magentic orchestration
Group chat orchestration puts several agents in one shared conversation under a chat manager. A maker agent proposes and a checker agent critiques in a loop. It works for review tasks where discussion improves the result. Cap the number of turns or the conversation will not end.
Magentic orchestration targets open-ended problems with no known plan. A manager agent builds a task list and assigns items to specialists. It revises the plan as results arrive. It is the most flexible pattern and the slowest and costliest to run.
Handoff orchestration
Handoff orchestration lets the active agent decide that another agent should take over. A triage agent receives the request and passes it to the specialist that fits. The specialist may pass it on again. Control moves with the context and no manager sits above.
This pattern suits customer journeys where the right expert only becomes clear during the conversation. It also fits the transfer to a human advisor. Its risk is a handoff loop between two agents. Set a maximum number of transfers and log each one.
How to choose a pattern
Pick the orchestration pattern from the shape of the task and the control you need. Begin with the simplest option that works and add structure only when a test shows a gap. Each approach to coordinating multiple agents has a typically different cost profile.
Running orchestration in production: failures, observability, security and cost
Why multi-agent systems fail: 14 modes in three families
A study titled "Why Do Multi-Agent LLM Systems Fail?" by Cemri and colleagues gives the best map of the problem. The authors analyzed more than 1,600 annotated traces from 7 multi-agent frameworks. They defined 14 failure modes in three families: system design issues and inter-agent misalignment and task verification. Their taxonomy is called MAST and annotators agreed with a kappa of 0.88.
Each family points to a control that the orchestration layer can own. Design flaws call for contracts and clear termination conditions. Misalignment calls for shared memory and explicit handoff payloads. Weak verification calls for a checker agent or a human approval gate. Teams that handle these failures at the orchestration level avoid patching every agent separately. Treat each report of a failure as an event with a root cause in one of the three families.
Did you know?
The taxonomy shows that many failures come from coordination and not from model intelligence. A stronger model does not fix a lost conversation history.
Failure recovery: retries, checkpoints and termination limits
Plan failure recovery before the first deployment. A failed step should retry with the same input and produce the same effect. Each retry is an event the trace must record. That property is called idempotency. A payment call that runs twice is a defect and a lookup that runs twice is harmless.
Save checkpoints after each completed step so that a restart resumes instead of repeating work. Define a fallback agent or a fallback to a person when retries fail. Set a hard cap on iterations and a watchdog that detects coordination deadlocks. An agent that cannot finish should stop and report instead of looping.
Observability: traces, metrics and audit trails
You cannot repair what you cannot see. Observability for a multi-agent system means one trace that follows a task across every agent and tool call. The trace records which agent acted and which context it received and what it returned.
Add metrics for latency and success rate and token use per agent. Keep an audit trail that is hard to alter so that a reviewer can reconstruct who decided what. This record supports debugging and compliance review and it lets you compare two versions of an agent on the same traffic. Operational reports built from traces show which processes cost the most. Incident processes and compliance processes depend on the same records.
Security: least privilege per agent
Each agent should hold only the permissions its job needs. This is the principle of least privilege. It matters more with several agents because one compromised agent can reach whatever it is allowed to reach. Give each agent its own credentials and rotate them. Developers who write agent code and support teams who operate it should never share one credential.
Encrypt traffic between agents and restrict which tools each one may call. Treat the content that agents pass to each other as untrusted input and evaluate it before it reaches a tool. Make permissions explicit in the infrastructure and not only in prompts. A document that carries hidden instructions should not be able to steer the next agent. Mask sensitive fields before any call to an external model provider.
Governance and the EU AI Act
For high-risk systems Regulation (EU) 2024/1689 requires automatic event logging in Article 12 and human oversight in Article 14. Annex III lists areas such as risk assessment and pricing in life and health insurance. An orchestration layer that touches those decisions must trace every agent and provide a way for a person to intervene.
Design these controls once at the orchestration level. Check the current application dates with your legal team because the calendar has been amended since the regulation was adopted. Under the EU AI Act a trace is a compliance asset and not only a debugging tool.
Cost control: token budgets and caps
Anthropic’s finding that token use explains most of the performance variance cuts both ways. More tokens can improve results and they also drive the bill. Set a token budget per agent and per run. Cap the number of model calls and stop runaway executions automatically.
Route routine steps to smaller models and reserve the stronger model for hard decisions. Cloud infrastructure bills grow with scale and usage, so every service you add must justify its cost. Measure cost per completed task and not per call. A system that retries silently can look inexpensive per call and still be expensive per outcome.
From pilot to production: an implementation path
Move from pilot to production in four stages:
- Pick one workflow with high value and a clear success measure.
- List the agents and write a responsibility matrix with inputs and outputs.
- Build a minimal prototype that emits a full trace.
- Run the pilot on real traffic and collect metrics before adding agents.
Expert tip:
Add the second agent only when you can name the failure that one agent could not solve. Each added agent should have a written reason and a measured result. Include the requirements for scale in the pilot plan: expected volume and latency targets and the resources needed to run it.
Frameworks and tools: what they cover and where they stop
Open-source frameworks such as LangGraph and CrewAI and the OpenAI Agents SDK give developers building blocks for routing and state and handoff. Cloud services such as Amazon Bedrock and Microsoft’s agent services host the same ideas as managed offerings. Their features cover capabilities such as state storage and tool integrations. Watch for vendor lock in when your agent workflows depend on one service. An orchestration framework answers how agents coordinate.
It does not supply your business data model or your approval rules or your audit policy. Your team writes those on top and plans the code and resources to maintain them. To compare layers beyond orchestration such as model gateways and pricing models read our guide to an AI agent platform. Choose a framework when your engineers want full control and accept the maintenance that comes with it. Test the behavior of the system under load and generate realistic traffic before launch. Good practice is to keep an explicit exit plan so that infrastructure choices do not become lock in.
Where Kleio fits: orchestrating a high-value purchase journey
One purchase journey, four agents and a human advisor
Kleio applies agent orchestration to complex, high-value sales such as real estate and travel and automotive retail. In those journeys the buyer rarely knows what to ask. A discovery agent qualifies the need. A product agent reasons over the catalog. A quote or financing agent builds the offer. A scheduling agent books the appointment. A final handoff sends the buyer to a human advisor with the enriched context.
At each handoff the diagram shows what travels: the shared state and the access rights of the next agent and the trace. It also shows which standard applies. MCP connects agents to tools and data. A2A connects agents to each other. UCP exposes the catalog to AI assistants.
State and memory shared through three purpose-built stores
The agents above read the same business facts. Kleio’s Knowledge Engine spreads a catalog across three purpose-built stores for product and document and memory content. The memory store keeps the context of each conversation so that the next agent does not start from zero.
A separate layer holds configuration and governance. This design answers the shared-state problem described earlier with a business data model and not only a generic buffer. Each industry has its own ontology because travel is not automotive and real estate is not wholesale. See the Kleio platform for the full stack.
Access control and versioned configurations per operation
Kleio applies per-operation RBAC so that each agent has only the rights its operation needs. Agent configurations are versioned. A change can therefore be traced and rolled back. Sensitive fields are pseudonymized in the flow before any call to an external model provider.
The platform is tenant-isolated and SOC 2 compliant. These controls map to the failure families above. Versioned configurations support recovery and per-operation rights limit the effect of one faulty agent. A risk team can review the same trace that engineers use for debugging.
Orpi: orchestration live across 1,250 agencies
Orpi, a French real estate network, runs the platform on orpi.com and in each of its 1,250 agencies for 8,000 advisors. Its CEO Guillaume Martinaud described the result:
"[...] the platform is live, on Orpi.com and in each of our 1,250 agencies."
The quote comes from the client and not from a Kleio promise. It shows agents running on a complex purchase journey with advisors in the loop. Read the Orpi case study for the full account.
When Kleio is not the right fit
Kleio is not an open-source orchestration framework and not a general language model. It is not a self-service tool and it has no free tier. Deployment is guided by the Kleio team for large enterprises with complex, high-value sales.
If your engineers want to build and own every layer for an internal workflow a framework fits better. If your agents must reason over a large catalog and a risk team must approve them a vertical orchestration layer deserves a place on your shortlist. Bring one real journey to the first demo.







