Finance will hand you a per-token comparison. Local inference at some fraction of a cent per thousand tokens, the frontier API at some multiple of that, and a delta that looks like a budget line worth defending. The arithmetic is fine. The denominator is wrong.

Tokens are a throughput unit. Nobody buys tokens. You buy answers a human is willing to ship, and the ratio between those two things is not a constant — it moves with model size, task class, and how many attempts it takes to land one. The moment you divide by accepted answers instead of tokens, most local-versus-cloud spreadsheets stop saying what they said.

I priced the numerator side of this last year — datacenter GPUs, power, cooling, and the DevOps tax of running your own inference. That post assumed the denominator. This one goes after it.

Five things the per-token quote silently drops

Retries. A 32B model that gets it right 70% of the time on the first pass costs you 1.43 generations per accepted answer, not 1. Every failed attempt burns the same tokens as a successful one and produces nothing.

Escalation. Some fraction of tasks the local model cannot finish at all. Those tasks pay the local cost and the frontier cost. If you don’t measure that rate, you’re carrying it as an invisible surcharge on the cheap path.

The review pass. This is the big one and it never appears in a token quote. A weaker model shifts work onto a person, and a person costs three to four orders of magnitude more per minute than the accelerator does.

Idle time. Utilization, not throughput, sets unit cost on owned hardware. The box depreciates whether or not anything is in the queue.

Amortization and refresh. Per-token framing treats hardware as free after purchase. It isn’t. It’s a fixed cost divided by however many tokens you actually generate before you replace it.

What my own box costs, and how I got there

I run a dedicated inference host: a 64GB M4 Pro Mac mini serving llama3.3:70b, qwen3.5:27b, qwen2.5-coder:32b, and deepseek-r1:32b through Ollama. Everything below is measured on that machine, and every assumption is on the table so you can redo it with yours.

Assumptions: $2,000 acquisition, 36-month refresh window, roughly 10W idle and 45W under sustained generation at the wall, about 8 output tokens per second on the 70B at 4-bit quantization. Electricity at $0.31/kWh — that’s the rate used in a good watt-by-watt measurement of Apple silicon inference, which also priced per million output tokens and then re-ran the numbers against 30 days of real traffic rather than benchmark loops. Substitute your utility rate.

At my actual duty cycle — about 90 minutes a day of real generation, 45 hours a month:

  • Amortization: $2,000 ÷ 36 = $55.56/month
  • Power: 2.03 kWh generating + 6.75 kWh idling = 8.78 kWh = $2.72/month
  • Output: 45h × 3600s × 8 tok/s = 1.296M tokens
  • ≈ $45 per million output tokens

Now saturate the same box. 24/7 generation, 720 hours, 20.7M tokens, $10.04 of electricity. Same amortization, same tokens per second, same model weights.

≈ $3.17 per million output tokens.

Fourteen-fold swing in unit cost with zero change in hardware or performance. The variable was utilization. This is why an H100 cluster and a Mac mini can both be “cheap per token” on a slide and neither number survives contact with a real request pattern — the cluster amortizes a far larger fixed cost and needs far higher sustained load to get there.

The formula to bring to the meeting

Cost per accepted answer:

CPAA = (A × t_l × c_l) + (e × t_f × c_f) + (m ÷ 60 × W)

A   = attempts per accepted answer (1 ÷ first-pass acceptance rate)
t_l = output tokens per local attempt
c_l = your local cost per token (amortization + power ÷ tokens actually generated)
e   = escalation rate to a frontier model
t_f = output tokens on the escalated call
c_f = frontier rate per token (your rate card)
m   = human review minutes per accepted answer
W   = loaded hourly cost of the reviewer

Worked, with my $45/M and plausible task numbers. A = 1.4, t_l = 900 tokens, e = 0.15, m = 4 minutes, W = $120/hour loaded. For c_f I’ll use a $15/M output placeholder — plug in whatever your contract says.

  • Local tokens: 1.4 × 900 × $0.000045 = $0.057
  • Escalation: 0.15 × 900 × $0.000015 = $0.002
  • Human review: 4 ÷ 60 × $120 = $8.00

$8.06 per accepted answer. The token lines are 0.7% of it.

Run the same task class entirely on the frontier model. Acceptance goes up, so A drops to 1.15, e goes to zero, and review drops to 2.5 minutes: $0.016 in tokens plus $5.00 in review. $5.02.

The local box wins the token comparison by about six cents and loses the decision by three dollars. Ninety seconds of extra review per answer is worth roughly fifty times the entire per-token delta at my volumes.

The honest counterargument: m is not fixed. On narrow, well-bounded work — commit message drafting, log triage, structured extraction against a fixed schema — a 32B coder model produces output a reviewer skims rather than audits, and m is identical on both paths. When that’s true the local box wins outright and the math is not close. My point isn’t that local always loses. It’s that the term deciding the outcome is one nobody measures, and the term everybody argues about rounds to noise.

Measure m. Measure A. Measure e. Then argue.

Keep escalation a config line, not a rewrite

None of this is measurable if the model choice is compiled into your application. You need the router in front, and the routing decision has to be data.

That’s the pattern in inference-router — an OpenAI-compatible layer that sits between the app and both backends, so which model serves which task class is a config change and every escalation is a countable event. Point the app at one endpoint. Move a task class from qwen2.5-coder:32b to a frontier model by editing a file, watch CPAA, move it back if the review time didn’t drop.

Without that, e is unknowable and you’re arguing from vibes. With it, escalation rate is a metric you can put on a dashboard next to acceptance rate, and the local-versus-cloud question becomes a per-task-class answer instead of a religious one.

While you’re in there, cap output length. One team traced a large share of their bill to uncontrolled, conversational output tokens — verbose responses cost the same whether the extra tokens help or not, and on owned hardware they cost you queue time too.

Buy local for the reasons that hold up

There are three defensible reasons to run inference on hardware you own: you need the data to never leave your boundary, you need latency a network round trip can’t give you, or you need to keep serving when a vendor changes a model version out from under you. All three are real, and none of them require the token math to work out in your favor.

“It’s cheaper per token” is not on that list. It’s the claim finance will test first and the one most likely to fall apart, because it’s the only one that depends on a denominator nobody is tracking.

My mini pays for itself on data residency. At $45 per million output tokens and a 4% duty cycle, it does not pay for itself on price, and I’d rather say that out loud than have someone else find it in a spreadsheet.