How I built Docket: fine-tuning a small open model to make decisions

A lot of the language-model calls inside real software aren't conversations. They are decisions. Which team should handle this ticket? Should this agent's tool call be approved? Is this invoice safe to pay? How severe is this incident?

TypeSafe's Jev made me take that seriously. Jev is a hosted model built for exactly this: you send it some context, a question and the allowed answers, and it returns one of those answers with a probability for each. There is no paragraph to parse and no invented option. It answers choice, yes/no and score questions, through an API called System One.

A small open-source community has grown around the same idea. Winnow-12B, AWS's Strands Decider, CLM, Laya, jevify and OpenJEV all try to build a Jev-style decision model you can run yourself. Docket is my attempt at the same thing. Its API follows Jev's System One format, so existing clients can switch by changing one URL. What I wanted to find out was whether a small open model, fine-tuned carefully, could match the hosted reference on the decisions businesses actually make, and run on hardware people already have.

This post is the story of building it: what I measured first, how the training data was made, and which changes moved accuracy.

Step 1: measure what already exists

Before training anything, I benchmarked the open alternatives. The yardstick was JevBench's 111 hardest public decisions, scored with the same harness for every model.

Three things stood out, and they shaped everything after.

Only a 12B model matched Jev. Winnow-12B tied it, but serving a 12B model cost about three times Jev's own price per million tokens in my tests.

Fine-tuning a small model didn't make it a general decision-maker. Both published Gemma 4B fine-tunes scored below the untuned Gemma model they started from. Several small specialists sat near random guessing on this broad benchmark, even though they did well on their own workflows.

Small fine-tunes do win in their own domain. jevify came within a few points of Jev on its own benchmark. Laya beat it on its four workflows.

So I narrowed the goal. I would not try to build a general replacement for Jev at 3B. I would build a small, self-hostable model that matches or beats Jev on a defined set of enterprise decisions, and measure honestly everywhere else.

Step 2: choose a base model

I picked IBM's Granite 4.2 3B. Out of the box it was weaker than Gemma on the hard benchmark, 53 against 64. But three things mattered more for this project:

• The licence. Granite is Apache-2.0 and not gated, so I could fine-tune it and redistribute the result.
• Speed. In my measurements, one pass over a 1,000-token decision took 39 ms on Granite 3B against 68 ms on Gemma E4B, partly because Granite's tokenizer needs fewer tokens for the same English text.
• Instruction following. Granite 4.2 is IBM's instruction-tuned model, and a decision prompt is mostly instructions: read this, apply these rules, pick one option.

The bet was that fine-tuning would close the quality gap, and that the speed and licence advantages would last. Starting from the instruction model rather than the raw pre-trained one was a reasoned default. I didn't test that choice.

Step 3: build the training data

This was the hardest part of the project, and the most interesting. There is no public dataset of labelled enterprise decisions. Companies don't publish their expense approvals or incident triage logs. And labelling thousands of realistic cases by hand, correctly, was out of reach for one person.

The idea that unlocked it: many enterprise decisions follow written rules, so the correct answer can be computed rather than judged. An expense is reimbursable if it was filed before the deadline, has a receipt, and is under the limit. An invoice should be held if the quantity, the price or the bank details don't match the purchase order. If the rules and the facts are known, code can compute the answer exactly.

Each training case starts from a policy and a set of facts. Code computes the answer. A language model then writes a realistic document from those facts: an email thread, an insurance claim note, a support ticket, a call transcript, an agent's run log. It writes in deliberately varied styles, from formal legal language to a casual Slack message to a passive-aggressive forwarded thread. It never decides anything; its only job is to write.

Generated text is unreliable, so every document is checked before it's used. Every amount, date and name must appear exactly. A separate pass must be able to read the decisive facts back out of the text. And the document must not give the answer away by saying "approved" or "denied". Documents that fail are rewritten or dropped.

Then comes the part I'd recommend to anyone building decision data: every case comes in a set of three. There's the original case. There's a flip, where one decisive fact changes and so does the answer. And there's an irrelevant edit, where an incidental detail changes and the answer stays the same. Untuned Granite rarely tracked the one fact that mattered: it got both halves of a flip pair right only 15% of the time, against Jev's 62%. Training on flips teaches the model to look for the fact that decides the outcome, not the general shape of the case.

The first version covered six workflows: expense policy, insurance claims, agent tool-call approval, invoice matching, incident severity and ticket routing. In the agent workflow, some cases contain instructions injected into an email the agent reads, so the model has to learn not to follow them. Later rounds added four more: PII sharing, leave requests, access requests and contract review.

Alongside the enterprise cases, the training mix includes public datasets, all rendered as the same kind of decision question: WANLI (natural-language inference), Banking77 (customer-intent classification), GSM8K (arithmetic word problems) and AQuA-RAT (multiple-choice maths). The dataset page has illustrative examples of each type.

All splits were fixed before any model saw the data. Cases were grouped, so near-duplicates never crossed between training and test. The test sets used phrasings and writing styles the model never saw in training.

Step 4: train, measure, repeat

Docket is a LoRA fine-tune: a small set of extra weights, under 1% of the model's size, trained on top of the frozen Granite model and merged in at the end. The training goal is narrow on purpose. Each example is shown in exactly the format the server will use. The model is trained only to put its probability on the correct allowed answer at the point where the answer goes. It never learns to write explanations, because Docket never asks for one.

Before each round I wrote down what would count as success, so I couldn't move the goalposts after seeing results. Here is how accuracy moved, round by round.

Round 1: enterprise cases only. On the six workflows it trained on, accuracy jumped from 57% to 90.1%, already ahead of Jev's 85.8% on the same questions. But something got worse where I wasn't looking. On ordinary public language tasks, accuracy held steady while the model became overconfident. The share of answers that were wrong despite at least 90% confidence more than doubled, from 6.1% to 14.6%. Training only on rule-following made the model sure of itself everywhere.

Round 2: mix in public data. Adding the public datasets back into training fixed that. Accuracy on general tasks rose 6.8 points, and confident-wrong answers fell to 1.3%, below Jev's 7.9% on the same set. Enterprise accuracy moved by about a point, within noise. A score on general tasks the model was never trained on also rose, which told me the mix was helping more than just the datasets it included.

Round 3: variety over volume. Round 3 added four new workflows. Each new workflow had only a modest number of training cases, and most still reached 97–100% accuracy. Overall, round 3 added 9.2 points over round 2 on the shared ten-workflow set. Adding new kinds of decision helped more than adding more of the same. The one stubborn workflow was leave requests, at about 70%, where Jev also struggled.

At this point I opened the sealed test, once. Across 3,063 questions in ten workflows, Docket scored 89.9% against Jev's 86.4%. The lead had held on data nothing had been tuned on.

Round 4: reasoning. The broad benchmark told a humbler story. JevBench improved from 153 to 170 out of 231, but all of that gain came from rule-style questions. On the hard tier it was flat, and on questions about grading maths or code answers it got slightly worse. Nothing in training had asked for multi-step reasoning. So the next round targeted it directly, with new rule-checked generators:

• long rulebooks with defined terms and dated amendments;
• deadlines across business days, holidays and time zones;
• expected-cost and probability choices;
• lookups across several JSON records;
• checking whether someone's worked calculation is correct.

GSM8K and AQuA-RAT joined the public mix. Reasoning accuracy on held-out tasks rose from 55.0% to 78.2%, and a new logical-verification test went from 72.5% to 100%. JevBench rose to 177, with the hard tier up from 56 to 63. Enterprise accuracy stayed at 90.1%.

Step 5: make the probabilities honest

A decision model is only useful if its confidence means something. When it says 90%, it should be right about 90% of the time. That's what lets you automate the confident cases and send the rest to a person.

After training, I fitted one temperature per question type on held-out development data. A temperature is a single number that softens or sharpens the probabilities without changing which answer wins. All three came out slightly above one, meaning the trained model was a little overconfident. They ship with the model in a small docket_calibration.json file. On the final enterprise test, only 1.18% of Docket-3B's answers were both wrong and at least 90% confident, close to Jev's 1.14%.

Step 6: measure without fooling yourself

Fine-tuned models are easy to flatter. These rules kept me honest:

• Success criteria written down before training.
• Answers computed by code, never by a model, including Jev.
• Splits by case group, fixed before any model saw the data.
• A sealed test opened once, for frozen candidates only.
• Paired confidence intervals against Jev on exactly the same questions.
• JevBench never used for training or for choosing between candidates. It was read once per finished model.

The final numbers for the released Docket-3B:

Docket leads Jev by 3.6 points on enterprise decisions, with a 95% confidence interval of +2.0 to +5.3. Jev is clearly better on the broad public benchmark. Both are true, and the second one is why I describe Docket as an enterprise decision model, not a general replacement for Jev. The benchmarks page has the full conditions.

Step 7: ship it

The release is two models on Hugging Face and one repository on GitHub:

• Docket-3B: the fine-tune merged into the weights, 7.3 GB in BF16, for NVIDIA GPUs.
• Docket-3B-GGUF: an 8-bit copy, 3.9 GB, for Macs and CPUs. On 1,326 test questions it gave the same answer as the full model 98.87% of the time, with workflow-weighted accuracy of 85.26% against 85.28%.
• docket-serve: the server that turns a System One request into the exact prompt format the model learned, reads its answer, and applies the calibration.

The server matters more than you might expect. Docket's weights load in any tool that runs Granite, but a generic chat server will just chat. The decision behaviour needs the exact prompt format and the readout of the allowed answers, and that lives in about 700 lines of Python. Every other open Jev-style project I tested ships its own server too, for the same reason.

What I'd tell someone starting out

• Benchmark the field before you build. It told me what not to attempt.
• If answers can be computed, compute them. It makes labels trustworthy.
• Teach the deciding fact. Flip pairs did more for robustness than more data would have.
• Mix in public data, or watch your confidence drift.
• Write your success criteria before you train, and keep one test sealed.

If you want to try it, three commands start a decision server on a Mac or a Linux machine:

git clone https://github.com/shah-jenil/docket-serve.git && cd docket-serve
./install.sh gguf      # on an NVIDIA GPU: ./install.sh vllm
./serve.sh 3b-gguf     # on an NVIDIA GPU: ./serve.sh 3b

The API reference shows the request format. The Docket page has the benchmarks and both releases.