Jables/helmsman-qwen3-1.7b-v2

Helmsman — Qwen3-1.7B adaptive-effort tool-caller (v1)

The System-2 escalation tool-caller for the ZeroClaw browser agent, LoRA-fine-tuned from mlx-community/Qwen3-1.7B-bf16 on CALIBER. It reads the "log pose" — deciding how much to think — and steers effort: deliberate on ambiguous / multi-step turns, act directly on unambiguous single-tool turns, and abstain / ask / refuse-destructive correctly.

v2.1 — externally validated (the honest headline)

Unlike v1 (measured only on our own benchmark), v2.1 is measured on the independent Berkeley Function Calling Leaderboard (BFCL v4) using its official ast_checker and the registered qwen3-1.7b prompt-mode config (the published ~55% anchor):

BFCL v4	base Qwen3-1.7B	v1	v2.1
simple (single call)	0.640	0.860	0.900
multiple (tool selection)	0.667	0.633	0.633
irrelevance (don't call)	1.000	0.467	0.767

v2.1 is a genuine, off-our-turf improvement over v1: the fine-tune bought real single-call skill (0.90, above the anchor), and a targeted contrastive fit/no-fit augmentation fixed v1's over-calling on irrelevant tools (0.47 → 0.77) without losing single-call accuracy — the failure mode our own benchmark had structurally hidden. Honest tradeoff vs v1: slightly more under-thinking (underThinkRate 0.00 → 0.21). multi_step dependency chains remain the unsolved frontier (~0.06). See results-bfcl-v2.json.

Intended use & deployment posture

• This is an escalation brain, NOT a low-latency router. The <150 ms per-message hot path in ZeroClaw stays a heuristic + MiniLM classifier (the 0.5B ONNX decoder is 9.7 s WASM-q8 / 14.9 s WebGPU per message; a thinking model is strictly slower). Helmsman runs where deliberation is worth seconds: local Ollama / llama.cpp, opt-in, or as a shadow-gold oracle.
• Adaptive think-only-when-hard is itself a latency control: System-1 turns emit no <think> and short output, so the common case stays fast even on the escalation model.
• Output format is deployed Hermes-XML (<think>…</think> + <tool_call>{…}</tool_call>, tool results as role:tool) — no train/serve skew.

Training

• Stage 1 — trace-SFT (LoRA): rank 16 / alpha 32 / 16 layers on q,k,v,o; max_seq_length 4096 (multi-turn + thinking rows truncate at the shared config's 1024); config-caliber-1.7b.yaml.
• Stage 2 — preference refinement: reject-sampling (ReST) → DPO on {chosen = correct adaptive-effort trajectory, rejected ∈ {over-think-on-System1, under-think-on-System2, wrong-tool, hallucinated-args, failure-to-abstain, broken-dependency}}, with an auxiliary NLL term to preserve Hermes formatting. GRPO is a future stage.
• Distill-then-on-policy: a 1.7B must be distilled from a stronger teacher before on-policy refinement helps at this scale (SOD 2605.07725, D3-Gym 2604.27977).

Evaluation

Measured on the matched 155-case multi-turn benchmark (scripts/finetune/eval/trajectory-scorer.js), Helmsman-v1 (LoRA) vs the Qwen3-1.7B base:

Metric	Base	Helmsman v1
accuracy	0.239	0.419
adaptiveEffortAccuracy	~0.43	0.903
abstentionAccuracy	0.766	0.957
overThinkRate¹	0.697	0.197
underThinkRate	0.405	0.000
escalationAccuracy	0.000	0.154
falseEscalationRate	0.000	0.000
malformedRate	0.000	0.000

¹ fraction of act-directly turns carrying a non-empty <think> block. Qwen3 emits an empty <think></think> in non-thinking mode — that is zero reasoning (System-1) and is not counted.

Passed the confirm-gate: Δaccuracy +18.1pp (≥+5pp), abstention preserved, overThink ≤0.25, falseEscalation ≤0.15, malformed ≤0.10. The base does real substantive over-thinking (0.70); Helmsman deliberates on only ~20% of easy turns and otherwise acts directly — adaptive effort 0.43 → 0.90.

Honest limitations: multi_step dependency chains are still weak (0.04) — exact multi-turn tool-call-set match across ≥2 dependent steps is the hard frontier for a 1.7B on a 383-row v1 corpus. External BFCL-v4 / tau2-bench numbers are not yet run. This is a v1 that wins decisively on adaptive-effort + abstention + format discipline, not yet on long multi-step orchestration.

The production in-browser router pointer (promoted-pointer.json) stays verdict: INITIAL and is untouched by this release — Helmsman ships as the System-2 escalation brain (Ollama/local/ shadow), not the <150 ms hot path.

Limitations

Small-model reasoning ceiling; synthetic-distribution risk (mitigated by external gating + an anti-collapse real-data floor); the WHEN2TOOL scope note (value is effort suppression + plan / grounding / abstain, not "reasoning decides the call"). See the dataset card §8.