IBM Granite under the hood
Docket fine-tunes IBM Granite’s 3B language model for this task. Granite supplies the language understanding; Docket adapts it for typed decisions and serves the results through a System One-compatible API.
IBM GRANITE · JEV-STYLE DECISIONS
Language in.
Decisions out.
Docket is a Jev-style decision model built on IBM Granite. Self-host Docket-3B on a GPU, or Docket-3B-GGUF on a Mac or CPU, to classify inputs, route work, score relevance, and choose an agent’s next action—with typed answers and calibrated probabilities.
The server is on GitHub; the models download from Hugging Face on first start.
git clone https://github.com/shah-jenil/docket-serve.git && cd docket-serve
./install.sh gguf # Mac or CPU; on an NVIDIA GPU: ./install.sh vllm
./serve.sh 3b-gguf # GPU: ./serve.sh 3bJev-style models use language understanding to make bounded decisions. You provide context, a question, and the possible answers. The model scores those answers, giving your software a result it can use directly.
Docket fine-tunes IBM Granite’s 3B language model for this task. Granite supplies the language understanding; Docket adapts it for typed decisions and serves the results through a System One-compatible API.
Choice: select a tool, route, or label. Yes/no: estimate whether a condition holds. Score: rate an item on your ordered scale. Each uses probabilities over the valid answers, with per-type calibration.
Use the answers to branch application logic, prioritise items, or control an agent loop. One forward pass per question produces the answer distribution, without generating an explanation first.
Enterprise decision accuracy
Answer agreement with Docket-3B (BF16)
1,326 questions · not task accuracy
3,063 labelled questions
10 workflows · equally weighted
bf16 · development-fitted calibration
Use the same decision interface across application features, data pipelines, evaluation systems, and agent loops. Define the question and the outputs your software can use.
Assign a label, select a queue, or choose which workflow runs next.
Score relevance or quality, and prioritise items using your own scale.
Assess whether an output meets your criteria, or flag an item for review.
Choose the next tool, check an action, or decide whether to keep working.
These are intended workloads. Evaluate against your own labelled decisions before automating actions.
Docket scores your allowed answers in one forward pass per question. Choice, yes/no, and score replace a text-generation-and-parsing loop. You get probabilities for the alternatives, as well as the selected answer.
Provide context, instructions, and answer options for each question. The same API handles categorical choices, yes/no checks, and scores across different workloads.
Host the model and decision endpoint yourself. The model and server are Apache-2.0, and there is no hosted inference service required by Docket.
The request and response shapes follow TypeSafe System One. Existing clients can target a self-hosted Docket endpoint by changing their base URL.
Docket 3B scored higher on the sealed enterprise benchmark. Jev scored higher on JevBench. These results describe the evaluated workloads; test the model on the decisions you plan to run.
Typical generative fine-tuning teaches a model to reproduce a response token by token. Docket trains at the decision point: given the context and question, put more probability on the correct allowed answer.
The training loss compares only valid answer tokens at the same position used by the serving engine. It targets the choice, yes/no answer, or score level directly, rather than the wording of a generated explanation.
LoRA adapts the Granite weights, and the adapters are merged for deployment. After training, separate temperatures for choice, yes/no, and score are fitted on held-out validation data to calibrate the probabilities. Calibration is a separate step from learning the decisions.
BF16 SAFETENSORS
The reference release. Serve it with vLLM on an NVIDIA GPU for the lowest latency and continuous batching.
./serve.sh 3bGGUF Q8_0
A Q8_0 quantisation of the same weights. Serve it with llama.cpp on an Apple Silicon Mac, a CPU, or a small GPU.
./serve.sh 3b-ggufQuality: bf16, development-fitted calibration. On the 1,326-question core comparison, workflow-weighted accuracy was 85.26% for the GGUF and 85.28% for BF16, with the same calibration.
Both releases use an 8,192-token context by default. That budget includes your context, question, instructions, answer options, and chat formatting. The vLLM backend reserves one answer token, leaving up to 8,191 tokens for the complete rendered prompt.
Quality evaluations use the 8K configuration; the enterprise test uses documents up to about 3,000 tokens. A context limit does not establish accuracy at every length. GPU timings below come from a separate serving reference study.
docket-serve · install and serve scripts · Apache-2.0
Accuracy and serving performance answer different questions. Here’s how Docket 3B compares with the hosted Jev reference.
| Benchmark | Docket 3B | Jev reference |
|---|---|---|
| Enterprise decisions3,063 questions · weighted accuracy | 90.1% | 86.4% |
| JevBench231 public items · correct answers | 177 / 231 | 198 / 231 |
Quality evaluations: October 5–6, 2026, bf16 and an 8K limit. Enterprise accuracy weights workflows equally. Adapters and calibration were fixed before scoring; the reported results informed the model choice. JevBench items were not used in training or development-time adapter selection.
All benchmarks, methodology, and limitationsThe model is only part of the latency budget. The serving stack reduces compute, avoids repeated work, and keeps questions moving through the GPU. The optimisation figures below come from a separate reference study; selected-model FP8 accuracy and timings have not been remeasured.
Lower-precision inference increased L40S throughput by roughly 31–32% over bf16 at eight concurrent requests. Accuracy matched in those reference-weight checks; this comparison has not been repeated for the models in the quality tables.
The server captures whole prefills up to 2,048 tokens by default. For a 250-token 3B prompt in the bf16 control, median engine time fell from 13.7 to 8.7 ms with the larger graph capture.
vLLM batches questions across concurrent requests. On the H100, four requests in flight reached 92.9 requests/s for 3B. Higher throughput comes with higher per-request latency.
Several questions about the same state can reuse cached prefix computation. With prefix caching enabled, three questions took 27.3 ms for 3B in the in-process study.
The engine reads probabilities only over valid answer tokens. There is no multi-token explanation to generate, and no generated JSON to repair.
Oversized inputs return HTTP 422 instead of silently dropping evidence. The server exposes health counters and a latency header, with a portable Transformers backend available for checks.
Reference GPU measurements with separate weights. These timings describe the serving study; they have not been remeasured for the models in the quality tables.
| Model | HTTP p50 | HTTP p95 | HTTP p99 | Requests/s |
|---|---|---|---|---|
| 3B reference run | 17.8 ms | 18.5 ms | 20.3 ms | 55.9 |
GPU1 × NVIDIA H100 SXM 80 GB (Hopper)
InputOne choice question; ~1,000-token prompts. Workload mean: 987.4 tokens.
ConfigurationFP8; CUDA graph capture to 2,048 tokens; one request in flight.
BoundaryEnd-to-end HTTP on the same host, including server handling. External network travel is excluded. Unique requests; cached-prefix speedups are excluded from the headline results.
For comparison, the L40S 48 GB study measured 43 ms for 3B at one request in flight, with roughly 1,000-token prompts and FP8. These are measured GPU configurations, not guaranteed latency on another GPU or deployment.
Start with labelled examples from your own workflow. Set thresholds and human-review paths using those results.