Cost at scale: getting a 3B decision model to about a cent per million tokens

In How I built Docket I wrote about fine-tuning a small open model to make decisions: route this ticket, approve this tool call, hold this invoice. Docket beat the hosted reference, Jev, on the enterprise workflows it was trained for. That post was about whether a 3B model could make good decisions.

This one is about whether it can make them cheaply. Decision calls aren't occasional. A team that puts a model behind ticket routing or agent approvals can easily send it millions of requests a day, and at that volume the price per token decides whether self-hosting is worth it at all.

Docket answers a decision in one forward pass: read the context, put a probability on each allowed answer, done. There is no text to generate. So the cost of a decision is almost entirely the cost of reading the prompt, and one number matters more than any other: how many input tokens a single GPU can read per second.

Here is where that number ended up.

RTX 6000 is the NVIDIA RTX PRO 6000 Blackwell. "1k" and "500" tokens are the input length of each request, and tokens/s is input tokens per second on one GPU. "$/M full" is the cost per million input tokens with the GPU busy all the time; "$/M 80%" is the amortized cost, explained below. Costs are calculated using the on-demand hourly price of the equivalent AWS SageMaker GPU. JevBench is out of 231, where the full-precision model scores 177. Blank cells weren't measured.

FP8 on the RTX PRO 6000 with 500-token requests is the setup I'd run. FP4 is the cheapest at about a cent per million tokens, but it gives up accuracy on hard decisions.

What this means at scale

Fractions of a cent per million tokens are hard to feel, so here is the same table as a bill. Take a service that reads 10 billion input tokens a day, about 10 million decisions of 1,000 tokens each:

The self-hosted rows use the amortized cost. At this volume, the difference between the L40S and the setup I'd run is about $24,000 a year for the same decisions, and FP4 would save another $28,000 if its accuracy were acceptable for the workload. The rest of this post is how I got from the first row of Docket's numbers to the last, what worked, and what didn't.

Setup

• Model and server. Docket-3B served by docket-serve on vLLM, on one GPU, with the server limited to four CPU cores.
• Requests. Synthetic enterprise decisions across 20 decision types, one question each, trimmed to about 1,000 or about 500 input tokens. A pool of 12,000 unique 1,000-token requests and 16,000 unique 500-token requests. No request is sent twice to the same server, so caching can't make the numbers look better than real traffic would.
• Load. A fixed number of requests in flight, from 8 to 128, with each client sending its next request as soon as the previous one returns. Throughput is measured over 30 seconds after a short warm-up.
• GPUs. An NVIDIA L40S and an NVIDIA RTX PRO 6000 Blackwell, rented by the hour.
• Cost. The GPU's hourly price divided by the millions of input tokens it reads in an hour. Hourly prices are taken from AWS SageMaker's on-demand rates for these GPUs.

Full-load and amortized cost

The full-load cost assumes the GPU is reading tokens every second of the hour. Real traffic doesn't look like that. It has quiet periods, and a GPU you pay for by the hour costs the same whether it's busy or idle.

The amortized cost spreads the hourly bill over the tokens you actually serve, assuming the GPU is busy 80% of the time on average. It's the full-load cost divided by 0.8, and it's the better number for budgeting a real service.

Where I started

When I released Docket, the serving stack was already built for this workload rather than for chat:

• One pass, one token. The server reads the probabilities of the allowed answers at the answer position and never generates text.
• Continuous batching. Before each GPU pass, vLLM gathers every request that's waiting and runs them together, which keeps the GPU's matrix units busy.
• FP8. The L40S has tensor cores for 8-bit floating point that run roughly twice as fast as 16-bit. Turning FP8 on raised throughput by about 31% with matching accuracy in my checks.
• CUDA graphs for whole prompts. A CUDA graph records the GPU work for a pass once and replays it, skipping per-kernel launch overhead. I raised the capture size so it covers a whole decision prompt.
• The fine-tune merged into the weights, so there's no adapter arithmetic on each request.

On one L40S, that reached about 35 requests per second at 1,000 tokens each: $0.0209 per million tokens at full load, $0.0261 amortized. That's about half of Jev's price, which made self-hosting worth it. The question was how much further it could go.

What didn't work

The 8-bit GGUF

Docket also ships as an 8-bit GGUF file for Macs and CPUs. Smaller weights sound cheaper, so I served it on the same L40S with llama.cpp.

It cost 2.1 times as much as FP8: $0.0438 per million tokens at best, even with a server that batches requests. Reading a long prompt is mostly large matrix multiplications, so it's limited by compute, not memory. Smaller weights help when a model generates text one token at a time and waits on memory. What matters here is whether the multiplications run on the GPU's FP8 hardware, and llama.cpp's 8-bit format doesn't use it. GGUF is the right file for a laptop and the wrong one for a server.

Server tuning on the L40S

Next I changed one server setting at a time:

None of it moved the cost by more than about 10%. More requests in flight didn't help either: throughput levelled off at eight and more concurrency only added latency. Halving the request length doubled the requests per second but barely changed tokens per second, because the GPU does the same work per token either way.

The CPU wasn't the bottleneck. The server's four cores sat between 10% and 21% busy while the GPU reported 98%.

Why: the L40S hit its power limit

When nothing moves a number, it's worth measuring the ceiling directly. I timed only the matrix multiplications in Docket's layers, at Docket's exact shapes, with nothing else running. That's the fastest the card could read tokens if everything else took no time at all: about 54,000 tokens per second.

Even that was optimistic. A GPU boosts its clock for short bursts, and the test ran in bursts. Under sustained serving, the L40S throttled its clock to stay within its 350-watt power limit, which brings the realistic ceiling down to about 44,000 tokens per second. The server was already reaching about 82% of that.

That explained the flat table. Batching, graphs, fusion and scheduling remove overhead around the matrix multiplications. When the multiplications themselves are capped by power, there's very little left to remove. On this card, about two cents per million tokens is as low as it goes.

What worked

A faster GPU

If the card is the limit, the useful question is cost per unit of compute on a different card. The RTX PRO 6000 Blackwell costs about 1.6 times as much per hour as the L40S. On the same matrix-multiplication test it reads tokens about 1.7 times faster.

In real serving, FP8 on the RTX PRO 6000 reached 71,287 tokens per second at 1,000 tokens and 74,950 at 500: twice the L40S's throughput at half the latency. That's $0.0164 and $0.0156 per million tokens at full load, $0.0205 and $0.0195 amortized.

FP4

Blackwell also supports FP4, 4-bit floating point, at roughly twice its FP8 rate. The L40S doesn't. I built an FP4 copy of Docket-3B (NVFP4, weights and activations both in 4 bits) with llm-compressor, calibrated on a few hundred synthetic prompts.

It reached 116,315 tokens per second at 1,000 tokens and 123,166 at 500: $0.0100 and $0.0095 per million tokens at full load, $0.0125 and $0.0119 amortized. That's about a cent per million tokens, under half of where I started and roughly a quarter of Jev's price.

Accuracy

A cheaper model that makes worse decisions isn't cheaper. So I scored each precision on JevBench's 231 public items with the same harness as Docket's published results.

FP8 stays within a few questions of full precision. FP4 holds the easy and original tiers but loses 11 questions on the hard tier. Most of the losses are long policy questions and questions that combine several rules, with prompts of up to about 3,800 tokens. My reading is that four bits is enough precision to read a short case, but not quite enough to carry a long chain of conditions.

Which one I'd run

FP8 on the RTX PRO 6000, with requests kept around 500 tokens. That's $0.0156 per million tokens at full load and $0.0195 amortized, with a 116 ms median at about 150 requests per second from one GPU. It costs less than half of Jev's price, it's twice as fast as where I started, and its accuracy stays close to full precision.

FP4 makes sense for short, well-structured decisions at very high volume, once it's been checked on your own labelled cases. I wouldn't use it as a drop-in replacement for hard, long decisions.

What I took away

• Measure the ceiling before tuning against it. One measurement explained a whole table of flat results, and I'd have found it sooner if I'd run it first.
• Check the power draw. A spec-sheet number is a burst number. The clock a GPU holds under sustained load is the one your bill depends on.
• The GPU and the number format beat server settings. On a power-limited card, the big wins came from a different card and then a different precision.
• Shorter requests don't make tokens cheaper. They make each request cheaper. If you pay per token, length is close to neutral.
• Give every cheaper precision its own accuracy check, on hard cases. FP4's losses only showed up on the hard tier.