Inference Is Cheap Right Now. That’s a Problem.
Everyone building AI systems today is making the same bet: that inference costs will keep falling. That the trajectory from GPT-3 pricing to Claude 3.5 pricing to whatever comes next is a line that only goes down.
I think that bet is dangerously incomplete. And the gap between what per-token pricing suggests and what AI work actually costs is widening, not narrowing.
The argument for ever-cheaper inference is straightforward. Competition drives prices down. Hardware improves. Quantization and distillation make models more efficient. Input token prices have fallen roughly 1,000× in three years — from GPT-3 Davinci at $60 per million tokens to comparable open-source models at $0.06 [1]. That’s among the fastest cost curves in technology history, outpacing even Moore’s Law [2][3].
But this narrative treats inference as a commodity — interchangeable compute that gets cheaper the way storage did. It ignores what’s actually happening to the work we’re asking models to do. And it obscures a growing divergence: the cost of a token is plummeting, but the cost of a decision is not.
Three Forces Creating a Cost Floor#
Reasoning is not inference. The shift toward chain-of-thought, extended thinking, and multi-step reasoning models has fundamentally changed what a single “request” looks like. A reasoning-heavy call to a frontier model doesn’t consume the same compute as a simple completion. When your agent needs to think — plan a task decomposition, evaluate tradeoffs, resolve ambiguity — you’re buying something qualitatively different from what the per-token price suggests.
How different? Artificial Analysis benchmarked reasoning models against standard models on identical tasks and found that OpenAI’s o1 cost $2,767 versus $109 for GPT-4o — a 25× gap [4]. Claude 3.7 Sonnet with extended thinking cost 18× more than Claude 3.5 Sonnet without it. The multiplier comes from two compounding effects: reasoning models charge roughly 6× more per token and generate 8–20× more tokens per request, much of it invisible chain-of-thought that still hits your bill. As these capabilities become table stakes, the floor on what a meaningful inference call costs rises even as the per-token rate falls.
Demand is scaling faster than supply. Enterprise adoption is accelerating. Every company building AI agents, copilots, and automation pipelines is multiplying their inference volume. GPU capacity is expanding, but not at the rate of demand.
The numbers are stark. Google’s AI infrastructure VP told employees the company must “double its serving capacity every 6 months” [5]. Microsoft’s Azure demand has outstripped supply so severely that CEO Satya Nadella admitted, “You may actually have a bunch of chips sitting in inventory that I can’t plug in. In fact, that is my problem today” [6]. The Futurum Group summarized it bluntly: “All the hyperscalers report that their markets are supply-constrained, rather than demand-constrained” [7]. NVIDIA’s Blackwell GPUs are sold out [8]. When demand outstrips supply, prices don’t fall — they stabilize or climb.
Compliance multiplies compute. In regulated industries, a single decision often requires multiple passes: generation, review, validation, audit logging. You’re not making one inference call per task — you’re making three to ten across different models with different trust levels. CloudZero’s analysis of production deployments found that “the true cost of a resolved AI task is often 10 to 50 times higher than the posted ’per call’ price” [9]. The unit of work isn’t a token; it’s a decision, and decisions are getting more expensive even as tokens get cheaper.
None of this means per-token prices will reverse course. But it means the cost landscape is bifurcating. Simple inference for commodity tasks will keep getting cheaper. The frontier — reasoning-heavy, compliance-wrapped, agent-orchestrated workloads — operates under fundamentally different economics where the effective cost per decision can be orders of magnitude higher than the per-token headline price.
And there’s a compounding effect that makes this worse. When Satya Nadella invoked the Jevons paradox after DeepSeek’s launch — “As AI gets more efficient and accessible, we will see its use skyrocket” [10] — he was describing something already happening. Enterprise generative AI spending tripled from $11.5 billion to $37 billion during the same period that per-token costs dropped 1,000×. Tokens per query are growing from roughly 200 in 2020 to 22,000 in 2025, with projections reaching 150,000–1.5 million as agentic workloads expand. Cheaper tokens don’t reduce your bill when every task consumes 100× more of them.
Prompt Caching: The Right Idea at the Wrong Layer#
Anthropic recently shipped automatic prompt caching for the Claude API — a genuinely useful optimization. The mechanics are elegant: cache hits on static prompt prefixes cost 10% of standard input pricing. For long-running agent sessions with large system prompts and tool definitions, the savings are substantial.
A coding agent running 40 turns with a 15,000-token static context goes from 600,000 billed input tokens down to roughly 73,500. That’s real money at scale.
But prompt caching operates at a single layer of the efficiency stack. It optimizes within a model — reducing the cost of repeatedly processing the same context across turns of the same conversation. It doesn’t touch the question of whether you should be using that model in the first place for every task in the pipeline.
That’s where most agent architectures have a deeper problem.
From Monolith to Team#
The default architecture for AI agents today is simple: pick a frontier model, send everything to it, and let the model figure it out. System prompt, tool definitions, context, conversation history, task — all of it goes to the same model every time.
This is the equivalent of hiring a senior architect to also answer the phones, file the paperwork, and sweep the floors. Not because those tasks require that level of capability, but because it’s easier than building a system that routes work to the right person.
When inference is cheap, this works. When it gets more expensive — or when you’re operating at scale and those margins matter — it becomes the architectural decision that kills your economics. The alternative is to stop treating your model as a monolith and start treating it as a team.
A heterogeneous agent architecture decomposes work across multiple models — routing tasks based on complexity, risk, and capability requirements. Instead of sending every request to the most capable (and expensive) model in your stack, you build a system that matches the right model to the right task.
The core concept is a tiered agent taxonomy:
Orchestration agents handle planning, supervision, and routing. These require strong reasoning capabilities and typically run on frontier models — but they process relatively few tokens compared to execution agents. Their job is to decompose objectives, monitor progress, and decide where work goes.
Execution agents do the actual work: implementing changes, running analysis, producing outputs. Many of these tasks are well-scoped and don’t require frontier-level reasoning. A smaller, faster model handles them at a fraction of the cost — and often with lower latency.
Specialist agents handle domain-specific concerns like security review, compliance validation, or quality checks. These run on models fine-tuned or prompted for narrow expertise, and they activate only when their specific capability is needed.
The key insight is that model selection becomes a routing decision, not a default. A planning agent running on Opus decomposes a complex task into subtasks. A router evaluates each subtask’s complexity and risk profile and assigns it to the appropriate model tier. A simple extraction task goes to Haiku. A nuanced analysis goes to Sonnet. A high-stakes decision with compliance implications goes to Opus with human-in-the-loop approval. This isn’t just cost optimization — it’s architectural honesty about what different tasks actually require.
Why This Compounds#
The efficiency gains from heterogeneous routing compound with optimizations like prompt caching rather than competing with them.
Prompt caching reduces the cost of context within a model session. Heterogeneous routing reduces the cost of choosing the wrong model for the task. Layer them together and you get a system where each agent session is both cache-efficient and running on the cheapest model that meets the task’s requirements.
Consider a document processing pipeline that handles incoming files through a sequence of steps: classification, extraction, validation, exception handling.
In a monolithic architecture, every step runs on the same frontier model. Prompt caching helps — the system prompt and tool definitions are cached across turns — but you’re still paying frontier pricing for classification tasks that a small model handles perfectly.
In a heterogeneous architecture, a lightweight model classifies the document. A mid-tier model extracts structured data. A frontier model handles only the exceptions — the ambiguous cases, the edge conditions, the decisions that actually require deep reasoning. Each of these agents maintains its own cached context, optimized for its specific role.
The frontier model might process 10% of the total token volume instead of 100%. That’s not a 10% cost reduction — it’s a fundamental restructuring of where compute goes.
There’s a practical consideration that makes this work. One of the lessons from the Claude Code team is that switching models mid-conversation destroys your cache. If you’re 100,000 tokens deep with Opus and send a simple question to Haiku, the cache doesn’t transfer — you rebuild from scratch. The solution is the subagent pattern: the primary agent stays on its model with its cache intact and delegates to a different model through a separate, focused context. The orchestrator prepares a minimal handoff message with only the relevant context, the subagent processes it, and the result flows back into the primary session without touching the cache. The architecture naturally preserves cache coherency because it was designed around the principle that different models handle different work — not that one model handles everything.
Beyond the Cost Argument#
The case for heterogeneous architecture doesn’t depend on the cost thesis above. Even if per-token prices fall forever, the architectural benefits stand on their own.
It forces you to decompose your system into well-defined agents with clear responsibilities, explicit capabilities, and traceable decision chains. That’s just good engineering, regardless of what models cost.
It makes you model-agnostic by design. When your architecture routes tasks to capability tiers rather than specific models, swapping providers or upgrading models becomes a configuration change rather than a rewrite.
It improves latency. Smaller models respond faster. When 80% of your tasks can run on a model that returns in 200ms instead of 2 seconds, your users notice.
And it gives you a natural framework for governance and auditability. When every task has a defined complexity level, a model assignment rationale, and a complete execution trace, you can answer questions about why the system made a particular decision — which matters increasingly as AI systems move into consequential domains.
Build for the World That’s Already Here#
The relevant question isn’t whether inference costs will reverse — it’s whether the industry’s migration toward reasoning and agentic workloads has already shifted the center of gravity from cheap commodity inference to expensive decision-grade compute. The evidence suggests it has.
The best time to build efficiency into your architecture is when you don’t need it yet. When inference is cheap and margins are comfortable, you have the luxury of getting the design right without the pressure of a cost crisis forcing shortcuts.
Prompt caching, model routing, capability-based task decomposition — these aren’t just cost optimizations. They’re architectural patterns that make AI systems more robust, more auditable, and more adaptable to a cost landscape that is bifurcating rather than uniformly declining.
The teams that build monolithic agent architectures today because per-token pricing is falling will discover that their costs are dominated by reasoning chains, compliance passes, and agentic loops — none of which follow the commodity curve. The teams that build heterogeneous architectures will just adjust their routing thresholds. I know which position I’d rather be in.
References
[1] G. Appenzeller, “Welcome to LLMflation — LLM inference cost is going down fast,” Andreessen Horowitz, Nov. 12, 2024. https://a16z.com/llmflation-llm-inference-cost/
[2] B. Cottier, B. Snodin, D. Owen, and T. Adamczewski, “LLM inference prices have fallen rapidly but unequally across tasks,” Epoch AI, Mar. 12, 2025. https://epoch.ai/data-insights/llm-inference-price-trends
[3] N. Maslej et al., “Artificial Intelligence Index Report 2025,” Stanford Institute for Human-Centered AI, Apr. 2025. https://hai.stanford.edu/ai-index/2025-ai-index-report
[4] K. Wiggers, “The rise of AI 'reasoning' models is making benchmarking more expensive,” TechCrunch, Apr. 10, 2025. https://techcrunch.com/2025/04/10/the-rise-of-ai-reasoning-models-is-making-benchmarking-more-expensive/ (data attributed to Artificial Analysis)
[5] “Google must double AI serving capacity every 6 months to meet demand, AI infrastructure boss tells employees,” CNBC, Nov. 21, 2025. https://www.cnbc.com/2025/11/21/google-must-double-ai-serving-capacity-every-6-months-to-meet-demand.html
[6] S. Nadella and A. Hood, “Microsoft Fiscal Year 2026 First Quarter Earnings Conference Call,” Microsoft Corp., Oct. 29, 2025. https://www.microsoft.com/en-us/investor/events/fy-2026/earnings-fy-2026-q1
[7] N. Patience, “AI Capex 2026: The $690B Infrastructure Sprint,” Futurum Group, Feb. 12, 2026. https://futurumgroup.com/insights/ai-capex-2026-the-690b-infrastructure-sprint/
[8] NVIDIA Corp., “NVIDIA Announces Financial Results for Third Quarter Fiscal 2026,” Nov. 19, 2025. https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-third-quarter-fiscal-2026
[9] K. MacKenzie, “Your Guide to Inference Cost (And Turning It Into Margin Advantage),” CloudZero, Dec. 5, 2025. https://www.cloudzero.com/blog/inference-cost/
[10] S. Nadella, post on X (formerly Twitter), Jan. 27, 2025. https://x.com/satyanadella/status/1883753899255046301