When the subsidies that make flat-rate AI coding cheap eventually shrink, does routing work to local models actually save money — or is that just something everyone assumes?
The Problem: Why This Test Exists
Today, most people using Claude Code are on a flat-rate subscription — heavily subsidized by frontier model providers relative to true compute cost. That subsidy isn't guaranteed to last: frontier model pricing is already trending upward, and it's a widely shared concern (the "AI bubble" conversation) that the current level of subsidy will shrink or disappear as the market matures.
This project set out to answer the question that matters before that happens: when the subsidies shrink, can an organization shift some or all of its agentic coding workload to on-prem models and still come out ahead financially — without a risky, all-at-once migration to full on-prem infrastructure?
This wasn't the original ask. The AI Lab's starting brief was broader — find local models that are particularly good (small, fast, specialized) at specific coding tasks. But a claim like "this model is great at C#" is just words without a way to prove it. So the project started one step earlier: confirm that on-prem delegation saves real money and is practical to set up at all. If it doesn't save money, or it's too hard to implement, there's no point chasing model specialization on top of it.
How It Works: The Test Setup
The harness
Claude Code CLI — publicly available to anyone, partnership or not. It was chosen here mainly out of familiarity given nvisia's Anthropic partnership, but the approach itself is harness-agnostic: routing sub-agent work through a gateway to swap in local models would work with any agentic coding tool that exposes a configurable endpoint. This test specifically ran against an API key rather than the subsidized flat-rate plan, to model what a post-subsidy cost environment looks like.
The routing layer
Claude Code doesn't natively support on-prem models, but it exposes a configurable base URL. That URL was repointed to an open-source tool called agentgateway, which sits between Claude Code and the model providers: every call goes through the gateway first, which routes it to either Anthropic's API (Sonnet, Haiku) or nvisia's own AI Lab hardware running two local, open-weight models (Kimi K2.7 and Gemma 4 31B) — and logs token usage for every call either way.
The two roles
Orchestrator
Plans the requested feature and routes tasks to sub-agents. Run as either Sonnet (API) or Kimi K2.7 (on-prem), swapped per test run.
Sub-agents
Six of them: an editor and a reviewer for each of the three layers in a typical ASP.NET MVC app (C#, JavaScript, Razor). Editors write one file each; reviewers are read-only. Each sub-agent loads its own skill set into context — formatting rules for editors, defect-pattern review guidance for reviewers — and the orchestrator has its own skill defining the project's intended architecture. Sub-agents run as either Haiku (API) or Gemma 4 31B (on-prem).
The test target
A blank ASP.NET Core MVC starter project — no database, no existing backend. Every prompt adds one self-contained feature to that blank slate. The blank-project choice was deliberate: LLM output is inherently non-deterministic, so isolating cost meant removing every other variable a pre-existing, complex codebase would introduce.
Four model pairings, same five prompts across each
Sonnet (orchestrator) + Haiku (sub-agents) — the fully-frontier baseline
Sonnet (orchestrator) + Gemma (sub-agents, on-prem)
Kimi (orchestrator, on-prem) + Gemma (sub-agents, on-prem) — fully on-prem
Kimi (orchestrator, on-prem) + Haiku (sub-agents)
Measuring Cost Accurately
Every comparison is read from Claude Code's built-in /cost command — the actual billed dollar total for a run — not an estimate reconstructed from raw gateway token counts.
- Anthropic bills at several different rates depending on cache behavior (cache reads bill at roughly 10% of a fresh input token's price, for example), and those rates are subject to change — so these results are relative to the version of Claude Code used at the time of testing, and would need re-verification if pricing changes materially.
- Early testing was done by manually calculating cost from Anthropic's published rates, before Justin discovered /cost existed. Those manual estimates were discarded once billing-verified numbers were available — worth noting, though, that the discarded early tests pointed to the same conclusions, which is part of why there's confidence in the results below despite a relatively small formal sample.
- Local model usage is logged at $0 per run in these results. That's a simplification, not a claim that on-prem inference is free — it leaves out hardware amortization and electricity, deliberately, because electricity cost is small relative to everything else and hard to calculate precisely for a single test run.
Guardrails on the Test
Every model pairing, same prompt
All four orchestrator/sub-agent combinations ran identical prompts; never compared across different tasks.
Clean state each run
The project was reset to an identical baseline before every run.
Best-case runs only
Only successful runs were counted for each configuration, to isolate the best-case cost rather than mix in inconsistent results.
Findings: Results
Five prompts of rising complexity (roughly: the more sub-agent work a task required, the higher its complexity score) were run through all four pairings, with cost read directly from /cost:
Savings scale with complexity, not a flat rate. At the simplest task tested, using a local sub-agent was close to a coin flip versus the baseline — task 01 actually shows the local-sub-agent configuration costing marginally more than the baseline. As tasks required more sub-agent work, on-prem delegation pulled consistently ahead — the pattern everyone in the field expects, but this is an attempt to put verified numbers behind it rather than leave it as received wisdom.
The best savings-to-investment ratio wasn't full on-prem — it was hosting just the orchestrator locally. The "Kimi + Haiku" column (local orchestrator, frontier-API sub-agents) came in an order of magnitude below the baseline in every test, without requiring the infrastructure to host and load-balance multiple different sub-agent models. This points to a tiered path for adoption rather than an all-or-nothing decision:
Full on-prem
Both roles hosted locally — maximum savings, effectively $0 marginal API cost in this test, but the GPU investment only makes sense for organizations anticipating very large ongoing API bills.
Host the orchestrator locally, keep sub-agents on frontier models
The best savings captured relative to infrastructure investment in this test. Frontier sub-agents are cheap and easy to parallelize, so cost concentrates in the orchestrator role — hosting just that one role locally captures most of the available savings without needing to build out multi-model sub-agent infrastructure.
Host sub-agents locally only, keep the orchestrator on a frontier model
The lowest-effort entry point into on-prem (GPU capacity behind a simple API), but the most modest, complexity-dependent savings of the three paths — and, per the task 01 result, occasionally a wash.
Where This Falls Short
Best-case only
Every result above excludes timeouts and bad returns. Real-world usage will hit both.
Quality wasn't formally measured
Runs were graded by eyeballing the generated code (does it look right, does it do what was asked), not against a pre-committed, objective test suite.
One stack, one project
Every test ran against a single ASP.NET Core MVC starter. Justin doesn't expect the core savings pattern to change across stacks, but that hasn't been validated yet — of all the limitations here, this is the one he's least concerned about.
What This Means & Next Steps
The foundational question — does on-prem delegation save real money? — now has a measured, billing-verified "yes." That clears the way for the more ambitious question the AI Lab actually wants answered: which specialized local models are worth deploying for which tasks, and does fine-tuning them further on nvisia's own architecture and codebases increase the savings.
Before chasing that, the next round of testing needs to hold up under real conditions instead of best-case ones. Justin is building a small, homemade benchmark modeled on how benchmarks like SWE-bench actually work: give the model a pre-built interface and data contracts to implement against (not a blank page), hide the real test suite while the model works so it can't "cheat" off the tests, then restore the suite afterward to grade the result.
Planned next steps
Capture every run
Timeouts and failures included, not excluded.
Hidden tests
A pre-built spec and a hidden test suite grade quality alongside cost, instead of eyeballing it.
Count iterations
Log how many self-repair turns each model needs to reach a pass, since a sub-agent's failed attempts are what drive up the orchestrator's cost to fix them.
Best-case vs. real
Re-run the comparison under real conditions and see whether the savings margin shrinks, holds, or disappears.
Testing Real Cost Savings in Multi-Agent Coding