Vela-Decision-170M

A calibrated, efficient decision model for typed questions over a state (text, email, ticket, or JSON). It answers choice, noul and ordinal score questions with a probability distribution, never generating text.

170.57M parameters — 0.41x the size of a comparable open decision model. Measured on one RTX 5060 Laptop against convaiinnovations/laya (typed-decisions checkpoint, 421.3M), over the same 2,000 test decisions, scored by the same metric code.

Read this first: where it wins and where it loses

We measured the baseline ourselves rather than quoting it, and the result is mixed. Publishing the losses is the point of this card.

metric	this model	laya-typed-decisions	winner
accuracy	0.5940	0.7630	laya
Brier	0.5259	0.3982	laya
NLL	0.9009	0.7051	laya
macro F1	0.5438	0.7482	laya
acc@50% coverage	0.6960	0.8920	laya
score MAE	0.1374	0.0784	laya
ECE (15 bins, unfitted)	0.0344	0.2060	this model, 6.0x better
latency per case (5 decisions)	83.5 ms	100.5 ms	this model, 1.20x faster
peak VRAM	1.06 GB	2.55 GB	this model, 2.4x smaller
tokens per decision	125	768	this model, 6.1x fewer
parameters	170.6M	421.3M	this model, 2.5x smaller

All quality metrics are for the exact checkpoint shipped here (sched06, epoch 6), on all 2,000 test decisions.

Correction. An earlier version of this card listed Brier 0.5144, NLL 0.8758, macro F1 0.5383, ECE 0.0377, acc@50 0.7110 and "46.8 ms, 2.1x faster". Those quality figures came from a different checkpoint (ab-causal) that is not what is published here; accuracy happened to be identical at 0.5940, which hid the mistake. The latency figure compared a per-decision number against a per-case number. Both are corrected above. Re-measured head to head in one process on identical input (tools/decision/bench_headtohead.py), the latency advantage is 1.20x, not 2.1x.

This model is not more accurate than Laya. It is 0.169 behind on accuracy and behind on Brier, NLL, macro F1, score MAE and selective accuracy. If your requirement is raw classification accuracy on multi-option questions, use Laya.

What this model does offer is calibration and efficiency at 0.41x the size: its expected calibration error is 6.0x better with no temperature fitting at all, and it answers a decision in a sixth of the context. Read the calibration win with the accuracy loss in mind: a model that is less accurate and also confident is not straightforwardly more trustworthy, and ECE alone does not capture whether the probabilities are useful for deciding.

Why the calibration result is credible

The benchmark's gold carries a soft teacher distribution per decision, not just a label. We regress directly onto that distribution with a strictly proper scoring rule (cross-entropy, plus the ranked probability score for ordinal questions). Training on the target rather than searching for it with RL removes the gradient variance of a policy-gradient approach and is why raw ECE lands below the baseline's temperature-fitted 0.081 without any calibration step.

Note that a low ECE alone is not a quality claim on its own — a near-uniform predictor scores well on ECE while being nearly useless. That is why accuracy, Brier, NLL and coverage are all reported above, and why this model is honest about losing them.

Two caveats on the comparison

• Two of the baseline's published numbers do not reproduce with the metric code from its own repository: we measure Brier 0.3982 (published 0.062) and score MAE 0.0784 (published 0.242). Its accuracy (0.763 vs 0.766), ECE (0.206 vs 0.213) and soft accuracy (0.4712 vs 0.471) all reproduce closely, so the harness is sound — those two columns specifically are not comparable. We do not claim a win on either.
• The accuracy gap is measured, not inferred, with a single 90/10 split. Fold-to-fold spread on this task is 0.0965, so any single-run number here carries roughly ±0.05 of uncertainty.

The act / escalate head

Measured on 300 held-out cases, the fitted head reaches AUROC 0.615 against "was this case answered entirely correctly", with monotone selective accuracy: answering only the top 50% most-confident cases gives 0.387 fully-correct against 0.333 overall.

For comparison, the baseline's act_probability emits exactly 1.0 for every one of the 2,000 decisions (mean 1.0000, std 0.0000, one distinct value). It carries no information and cannot be thresholded at all.

A bug worth recording, because it is the same failure mode: our act head initially scored AUROC exactly 0.500 because it pooled from sequence position 0. Under a causal encoder, position 0 attends only to itself, so it carries no information about the state. Pooling from the final option marker instead fixed it (0.500 → 0.615). If you adapt this model to a causal backbone, pool the act head from a late position.

act_head.pt holds the fitted head. Without it, the model's built-in act_out is a zero-initialised constant and carries no signal.

Architecture

SmallLM encoder (156.92M, causal, 12 layers, d=1024, GQA 16/4, RoPE)
  └─ 1 transformer layer head (13.65M)
       ├─ option-marker scorer: each option's [MASK] scored, softmaxed within
       │  that question only
       ├─ learned per-option-slot prior
       ├─ monotone cumulative-link parameterisation for ordinal questions
       └─ act / escalate head (fitted post-hoc)

Two deliberate departures from the baseline:

1. Causal, not bidirectional. The backbone is pretrained with a causal mask. Switching attention to bidirectional puts every layer off-distribution from step 0 and measurably hurt: validation accuracy trailed at every epoch after the first and by 0.158 at epoch 4. The state is encoded first and each option marker sits at the end of its own option text, so causal attention still lets every marker see the whole state.

2. Packed, not one-sequence-per-question. The state is encoded once per case and all of its questions are packed after it, recording each option's marker position. Same outputs, 6.1x fewer tokens per decision. Latency is 16.7 ms per decision against the baseline's 20.1 ms on identical hardware, a 1.20x advantage that comes mostly from the parameter count rather than from the shorter sequence.

Intended use

Typed, closed-set decisions where a calibrated probability matters more than open-ended text: routing, triage, escalation gates, schema-constrained classification over an application-defined option set.

Not intended for: open-ended generation, and not as a replacement for a higher-accuracy classifier where accuracy is the objective.

Usage

from transformers import AutoConfig, AutoModel, PreTrainedTokenizerFast

tok = PreTrainedTokenizerFast(
    tokenizer_file="tokenizer.json",
    mask_token="<|mask|>", pad_token="<|endoftext|>",
    eos_token="<|endoftext|>", unk_token="<|endoftext|>")
model = AutoModel.from_pretrained(".", trust_remote_code=True).eval()

The answer space is defined at request time. Render the state, then each question, then one line per option each ending in the mask token. The hidden state at each mask is scored and softmaxed within that question only:

<mask>                                     <- pooled position for the act head
subject: Duplicate charge on invoice #4411
body: We were billed twice for March.
question (department): Which department should handle this?
- billing: invoices, payments, refunds <mask>
- technical: bugs, outages, system errors <mask>
- sales: pricing, new contracts <mask>
- other: everything else <mask>

Pass marker_pos (the token index of each mask, -1 for padding) and option_slot (each option's 0-based position within its question). demo.py in this repo is a complete runnable example covering all three primitives plus the act head.

The state is encoded once and every question is packed after it, which is where the token saving comes from. For a single question you can pass exactly one marker position.

Reproducing the benchmark

git clone https://huggingface.co/ParallaxOpen/Vela-Decision-170M
cd Centauri   # the training + evaluation harness lives in the main repo
python tools/decision/verify_benchmark.py
python tools/decision/eval_decision.py --ckpt <checkpoint>
python tools/decision/eval_laya_local.py        # measures the baseline locally
python tools/decision/report_all.py

Ethics and limitations

The benchmark encodes annotator judgements; its annotators agree on only 59.4% of labels (mean inter-annotator total variation 0.202), so a perfect model would still not reach 1.0. Predictions should be gated on confidence rather than trusted uniformly — acc@50% coverage is 0.696 against 0.636 at 80% coverage.