The Case for Heterogeneous Orchestration

Most agent orchestration systems share an assumption so fundamental it’s rarely examined: that models are interchangeable. You pick a provider, wire up an API key, and route every task through the same model. Maybe you have two tiers — a big model and a small one — but the routing logic is manual, the selection is static, and the decision criteria live in someone’s head rather than in the system itself.

This works well enough when you’re building demos. It stops working when you’re processing thousands of documents a week and someone asks why you’re spending $0.50 to summarize a paragraph.

I’ve been building an orchestration layer called HAOL — Heterogeneous Agent Orchestration Layer — that treats model diversity as a first-class architectural concern. Not as a cost optimization hack or fallback strategy. As the fundamental design principle.

The homogeneity tax#

The default architecture for most agent systems looks something like this: a task comes in, you send it to your best model, you get a response, you log it. If you’re sophisticated, maybe you have a fast model for “simple” tasks and a smart model for “hard” ones, with an ill-defined boundary between them.

This is expensive in ways that aren’t obvious until you’re operating at scale. A T1 task — a simple lookup, a one-sentence summary — costs fractions of a cent when routed correctly. Send it to a frontier model and you’re overpaying by two orders of magnitude. Multiply that across thousands of daily tasks in an enterprise pipeline and you’re burning budget that could fund actual capability improvements. Chen et al. demonstrated this empirically with FrugalGPT, showing that LLM cascading — querying progressively more expensive models until a confidence threshold is met — can reduce costs by up to 98% while matching GPT-4 quality [1].

But cost isn’t even the interesting problem. The interesting problem is that different models are actually different. They have different strengths, different failure modes, different latency profiles. A model that’s exceptional at code generation might be mediocre at structured data extraction. A model that’s fast and cheap might handle 80% of your volume perfectly well. Treating them as interchangeable ignores information that the system should be using. The X-MAS benchmark confirmed this at scale: across 27 LLMs and 1.7 million evaluations, no single model excelled universally, and heterogeneous multi-agent systems that assigned specialized models to specific roles consistently outperformed homogeneous configurations — by up to 47% on certain benchmarks [2].

How HAOL thinks about routing#

HAOL classifies every incoming task along three dimensions: what capabilities does it require, how complex is it, and what’s it allowed to cost. The complexity question maps to a four-tier system — T1 (simple) through T4 (expert) — where each tier defines a cost ceiling and a pool of eligible agents.

The routing decision itself happens through a cascade router: a three-layer classification system that trades off speed against accuracy.

Layer 0 is pure pattern matching. Regex, prefix checks, keyword detection — the kind of thing that resolves in microseconds. If your prompt starts with “summarize” or contains “json,” the router already knows what tier you’re in. No API calls, no latency, no cost. This handles the majority of well-structured requests, and it handles them fast.

Layer 1 kicks in when patterns aren’t enough. It embeds the incoming prompt and compares it against a bank of 32 reference utterances — eight per tier — using cosine similarity. The intuition here is straightforward: “What is the tallest mountain?” and “What is the capital of France?” use completely different words but live in the same complexity neighborhood. Embeddings capture that neighborhood. The top five nearest neighbors vote on the tier, weighted by similarity score. If the vote is decisive (above a 0.72 confidence threshold), we’re done.

The voting mechanism is simple enough to read in one pass:

export function weightedTierVote(matches: SimilarityMatch[]): { tier: TierId; confidence: number } {
  if (matches.length === 0) {
    return { tier: 3 as TierId, confidence: 0 };
  }

  const tierWeights = new Map<number, number>();
  let totalWeight = 0;

  for (const match of matches) {
    const weight = match.score;
    tierWeights.set(match.tier_id, (tierWeights.get(match.tier_id) ?? 0) + weight);
    totalWeight += weight;
  }

  let bestTier = 3 as TierId;
  let bestWeight = 0;
  for (const [tier, weight] of tierWeights) {
    if (weight > bestWeight) {
      bestWeight = weight;
      bestTier = tier as TierId;
    }
  }

  const confidence = totalWeight > 0 ? bestWeight / totalWeight : 0;
  return { tier: bestTier, confidence };
}

Each match votes for its tier, weighted by how similar it is to the incoming prompt. If four of five nearest neighbors are T1 utterances and only one is T2, the weighted vote heavily favors T1. The confidence is the winning tier’s share of total weight — a clean signal of how unanimous the neighborhood is.

Layer 2 is LLM escalation for the genuinely ambiguous cases. “Help me with my project” could be T1 or T4 depending on context that keywords and embeddings can’t resolve. A cheap, fast model — Haiku — makes the judgment call. It costs about a tenth of a cent per classification and responds in under a second.

If all three layers are inconclusive, the system defaults to T3. Conservative. Might overspend, but won’t underdeliver.

The thing I like about this design is that it’s proportional. Simple tasks get simple classification. Ambiguous tasks get progressively more intelligence thrown at the problem. And because each layer has clear failure modes and fallbacks, the system degrades gracefully — if the embedding API is down, you skip Layer 1 and go straight to the LLM. If the LLM key isn’t configured, you fall back to pattern matching plus a conservative default. The system never hard-fails on classification.

This cascade pattern — try cheap first, escalate only when confidence is low — has independent validation. Aggarwal et al. formalized it as a POMDP-based routing problem with AutoMix, showing that self-verification routing can cut compute costs by over 50% without degrading output quality [3]. RouteLLM, from Berkeley’s LMSYS group, demonstrated that routers trained on preference data can achieve over 95% of GPT-4 performance while routing only 26% of queries to the expensive model [4].

This philosophy extends into the implementation. The LLM escalation provider, for example, wraps its entire API call in a catch that returns a conservative default rather than propagating the error:

// From escalation.ts — if the API call or Zod parse fails for any reason:
} catch {
  // Conservative fallback: T3, no capabilities, moderate confidence
  return { tier: 3 as TierId, capabilities: [], confidence: 0.5 };
}

The Zod schema validates the LLM’s JSON response against the expected shape — tier must be 1–4, confidence between 0 and 1. If the model hallucinates, returns malformed JSON, or the API times out entirely, the system doesn’t crash. It falls back to T3 with moderate confidence and keeps going. The routing log records what happened, so you can find and fix the gap later.

Why Dolt#

Every routing decision, agent configuration change, and policy adjustment in HAOL is stored in Dolt — a database that’s MySQL-compatible on the wire but Git-like under the hood. Every mutation is a commit. You can diff two points in time, branch to test a policy change before merging it, and blame any configuration to trace who changed it and when.

This might seem like overkill for a routing layer, but it solves a problem I kept running into: when something goes wrong in an agent system, the first question is always “what changed?“ Did someone adjust the scoring weights? Did an agent get disabled? Did a new routing rule get added that’s catching prompts it shouldn’t?

With a traditional database, answering these questions means building audit logging from scratch — and hoping you instrumented the right things. With Dolt, the audit trail is the database. The routing log table records every classification decision: which layer handled it, what confidence it had, how long it took. But beyond that, the schema itself is versioned. The routing rules, the agent registry, the policy weights — all of it is diffable, branchable, reversible.

DoltHub has been making a version of this argument explicitly since late 2025: that code agents succeeded because code is under Git, and that agentic systems operating on data need the same safety net — branch, review diffs, merge or discard [5]. The canonical pattern is an agent writing to an isolated branch while a human or automated process inspects the changes before they touch production state.

In regulated environments, this matters. In any environment where you need to explain why the system made a decision, it matters.

Agent selection as a scoring problem#

Once the cascade router assigns a tier and a set of required capabilities, agent selection becomes a constrained optimization. Candidates are filtered first — they must be active, their tier ceiling must be high enough, they must have every required capability, and their estimated cost must be within the tier’s budget. What survives gets scored:

score = capability_match × 0.5 + cost_efficiency × 0.3 + latency × 0.2

Those weights — 0.5, 0.3, 0.2 — are defaults stored as policy in Dolt, not hardcoded constants. You can branch, adjust the weights, compare results, and merge the change without a redeploy. In code, each dimension is normalized so that the best candidate in the pool scores 1.0 and the worst scores 0.0:

// From agent-selection.ts — scoring surviving candidates
const capabilityScore =
  requiredCapabilities.length === 0 ? (maxBonus === 0 ? 1.0 : bonusScore) : 0.6 + 0.4 * bonusScore;

const costScore = costRange === 0 ? 1.0 : 1 - (costs[i] - minCost) / costRange;

const latencyScore =
  latencyRange === 0 ? 1.0 : 1 - (agent.avg_latency_ms - minLatency) / latencyRange;

const totalScore =
  capabilityScore * policy.weight_capability +
  costScore * policy.weight_cost +
  latencyScore * policy.weight_latency;

The capability score is worth calling out. Every candidate in the pool already has the required capabilities — that’s enforced by the filter. So the capability score differentiates on bonus capabilities: an agent that can do code generation, reasoning, and structured output will outscore one that only does code generation, even if both meet the task’s requirements. The idea is that agents with broader capability sets are more likely to handle edge cases within the task.

If execution fails, the fallback strategy kicks in. NEXT_BEST tries the runner-up. TIER_UP relaxes the constraints and re-selects. ABORT gives up. The strategy is itself a policy decision, stored alongside the weights.

What this is and what it isn’t#

HAOL is an MVP. It’s a TypeScript/Node application backed by Dolt, running on Hono, with adapters for Anthropic, OpenAI, and local models. It has a CLI, an HTTP API, and a test suite. It’s functional. It routes tasks, selects agents, executes them, and records the results.

What it isn’t is production-hardened infrastructure. The scoring weights are informed guesses. The reference utterances need tuning against real workload distributions. The cost ceilings are reasonable defaults, not empirical findings.

But the thesis is what matters here. The thesis is that heterogeneity is the natural state of an agent ecosystem, and the orchestration layer should embrace that rather than abstract it away. That routing decisions should be auditable, diffable, and reversible. That classification should be proportional — spending intelligence on ambiguity, not on certainty. And that the right model for the job depends on the job, not on the contract you signed with a provider.

Empirical analysis of over 100 trillion tokens through OpenRouter confirms that the AI inference market is definitively multi-model, not winner-take-all [6]. Steve Yegge arrived at a compatible framing with his “cognitive pyramid” — arguing that all knowledge work decomposes into a hierarchy of cognitive tasks and that probably 50–80% of agent traffic could be routed to cheaper models without quality loss [7]. The question isn’t whether to route heterogeneously. It’s whether the routing intelligence lives in the system or in someone’s head.

The code is on GitHub. I’d welcome feedback from anyone thinking about these problems.

References#

[1] L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176, May 2023. [Online]. Available: https://arxiv.org/abs/2305.05176

[2] R. Ye, X. Liu, Q. Wu, X. Pang, Z. Yin, et al., “X-MAS: Towards building multi-agent systems with heterogeneous LLMs,” arXiv preprint arXiv:2505.16997, May 2025. [Online]. Available: https://arxiv.org/abs/2505.16997

[3] P. Aggarwal et al., “AutoMix: Automatically mixing language models,” in Proc. NeurIPS, 2024. [Online]. Available: https://arxiv.org/abs/2310.12963

[4] I. Ong et al., “RouteLLM: Learning to route LLMs with preference data,” in Proc. ICLR, 2025. [Online]. Available: https://arxiv.org/abs/2406.18665

[5] E. Richardson, “Agentic systems need version control: An example,” DoltHub Blog, Oct. 2025. [Online]. Available: https://www.dolthub.com/blog/2025-10-31-agentic-systems-need-version-control/

[6] M. Aubakirova, A. Atallah, C. Clark, J. Summerville, and A. Midha, “State of AI: An empirical 100 trillion token study with OpenRouter,” Andreessen Horowitz, Jan. 2025. [Online]. Available: https://a16z.com/state-of-ai/

[7] S. Yegge, “Zero framework cognition: A way to build resilient AI applications,” Medium, Oct. 2025. [Online]. Available: https://steve-yegge.medium.com/zero-framework-cognition-a-way-to-build-resilient-ai-applications-56b090ed3e69