By the time an enterprise has three AI capabilities in production, it has a problem. Each capability talks to different models, different APIs, different data sources. Authentication is handled three ways. Error handling is inconsistent. Rate limiting is an afterthought. The orchestration layer is the fix, and it's cheaper to build it early than to retrofit it later.
What You Need to Know
- Every enterprise running multiple AI capabilities needs an orchestration layer. It sits between your application code and the AI services, handling routing, authentication, error management, and observability.
- Without orchestration, each AI capability becomes a standalone integration. This means duplicated code, inconsistent error handling, and no central visibility into AI service usage.
- The orchestration layer is not a model router. It's a service management layer that handles the operational concerns of running AI in production: rate limiting, failover, cost tracking, and audit logging.
- Build it by capability two. If you wait until capability four, you're retrofitting three integrations. Start the pattern early and each new capability plugs in cleanly.
The Problem
Hassan and I have both seen this pattern play out. An enterprise builds its first AI capability. The team integrates directly with OpenAI's API. Authentication tokens are in environment variables. Error handling catches timeouts and retries. It works.
Capability two arrives. Different team, or the same team six months later. They integrate with a different model for a different task. New API client. New error handling. New authentication pattern. Also works.
Capability three. Same again. By now, you have three separate integrations with three sets of credentials, three monitoring approaches, and three ways of handling the same failure modes.
This is not a hypothetical. It is the default state of enterprise AI integration unless you actively prevent it.
The symptoms:
- No central view of AI costs. Each capability tracks its own usage (if it tracks at all). The CFO asks "how much are we spending on AI?" and nobody has a single answer.
- Inconsistent reliability. One capability retries on timeout. Another doesn't. One has circuit breaking. Others crash and require manual restart.
- No audit trail. Compliance asks "what data was sent to which AI service on what date?" and the answer is scattered across three separate logging systems.
- Model migration is painful. When you want to switch from GPT-4 to GPT-4 Turbo (or from OpenAI to Anthropic), you're changing code in three places with three risk profiles.
3-5x
more engineering effort to retrofit an orchestration layer after three capabilities are in production, compared to building it alongside the first or second
Source: RIVER, internal engineering estimates, 2024
The Architecture
The orchestration layer sits between application code and AI services. It is a thin but critical piece of infrastructure.
Request Routing
Application code sends a request to the orchestration layer, not directly to an AI service. The request specifies what it needs (a completion, an embedding, a classification), not which service should handle it.
The orchestration layer routes the request to the appropriate service based on configuration: model selection, cost tier, latency requirements, or load balancing rules. The application doesn't know or care which model handles the request.
Application → Orchestrator → AI Service (OpenAI / Anthropic / local model)
↓
Logging, metrics, audit trail
This decoupling means you can change AI providers without changing application code. Model A for production, Model B for testing. Provider A as primary, Provider B as fallback.
Authentication Management
One place where all AI service credentials are managed. Not scattered across environment variables in multiple services. The orchestration layer handles token rotation, key management, and per-service authentication.
For enterprises with security requirements (and all enterprises should have them), this centralisation is significant. A single point for credential management means a single point for credential auditing.
Error Handling and Resilience
Consistent error handling across all AI services:
Retry logic. Transient failures (timeouts, rate limits, temporary outages) are retried with exponential backoff. The retry policy is consistent regardless of which AI service is involved.
Circuit breaking. If an AI service is consistently failing, the circuit breaker opens and requests are routed to a fallback (a different model, a cached response, or a graceful degradation path).
Timeout management. Different AI tasks have different latency profiles. A document extraction can take 30 seconds. A classification should take 2 seconds. The orchestration layer enforces per-task timeouts, not a global default.
Fallback chains. Primary model fails? Fall back to secondary. Secondary fails? Return a cached response or degrade gracefully. The fallback logic lives in the orchestrator, not scattered across application code.
Rate Limiting and Cost Control
AI API costs scale with usage, and usage can spike unexpectedly. The orchestration layer enforces rate limits per capability, per user, or per cost tier.
Per-capability budgets. "Claims processing can use up to $X per day in API calls." If the budget is exhausted, requests queue or degrade.
Per-user limits. Prevents a single user or process from consuming disproportionate resources.
Cost tracking. Every request is logged with its token consumption and cost. Real-time dashboards show spending by capability, by model, by time period.
The orchestration layer is boring infrastructure. Boring infrastructure that works consistently is worth more than clever architecture that fails unexpectedly.
John Li
Chief Technology Officer
Observability
Every request through the orchestration layer is instrumented:
- Request logging. What was asked, what was returned, how long it took, which model handled it. Redact sensitive content, log everything else.
- Performance metrics. Latency percentiles, token usage, error rates, by capability and by model.
- Audit trail. For compliance and governance. Who sent what data to which AI service, when, and what was returned.
- Alerting. Latency spikes, error rate increases, budget threshold breaches, model degradation signals.
Implementation Patterns
Pattern 1: Gateway Service
A dedicated service (API gateway style) that all AI requests pass through. Clean separation of concerns. The gateway handles routing, auth, rate limiting, and logging. Application code makes HTTP calls to the gateway.
Best for: Organisations with multiple teams building AI capabilities independently. The gateway enforces consistency without requiring teams to use the same libraries or frameworks.
Pattern 2: Shared Library
A shared library that wraps AI service interactions. Application code imports the library and calls its functions. The library handles routing, auth, error handling, and logging internally.
Best for: Smaller teams or monolithic applications where a separate service is overhead. Simpler to deploy and maintain, but requires all capabilities to use the same runtime.
Pattern 3: Sidecar
A sidecar process that sits alongside each application instance, proxying AI requests. Each capability deploys with its own sidecar, but the sidecar implementation is shared.
Best for: Microservice architectures where a central gateway is a bottleneck and a shared library isn't feasible across different runtimes.
Our Default
We typically start with Pattern 2 (shared library) for the first two or three capabilities, then evaluate whether the complexity warrants moving to Pattern 1 (gateway) as the number of capabilities grows. Hassan and I agree on this: start simple, add architecture when the pain justifies it.
When to Build It
Before capability two. The orchestration layer should be in place before the second AI capability goes to production. Building it alongside capability one adds modest scope. Retrofitting after three capabilities are live is significantly more expensive.
If you already have multiple capabilities without an orchestration layer, the retrofit is still worth doing. The ongoing operational cost of managing N separate integrations grows with each new capability. The orchestration layer is a one-time investment that reduces the marginal cost of every subsequent integration to near zero.
What Not to Over-Engineer
A few things to resist:
- Don't build a model evaluation framework into the orchestrator. Model selection is a data science concern. The orchestrator routes to configured models. It doesn't decide which model is best.
- Don't build prompt management into the orchestrator. Prompts are application logic. The orchestrator transports requests. It doesn't manage prompt templates.
- Don't build a custom queue system. Use existing message queues if you need async processing. The orchestrator handles synchronous request/response patterns.
Keep it focused. Routing, auth, resilience, rate limiting, observability. That's the job. Everything else belongs in application code or dedicated services.