Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,2 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+pace-intent-router-v8.tinygpt filter=lfs diff=lfs merge=lfs -text
+*.tinygpt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,182 @@
+---
+base_model: posttrainllm-ToolRouterModel-small
+language:
+  - en
+library_name: posttrainllm
+license: mit
+pipeline_tag: text-classification
+tags:
+  - posttrainllm
+  - tinygpt
+  - intent-classification
+  - routing
+  - pace
+  - mac-first
+  - on-device
+---
+
+# Pace Intent Router v8
+
+## Summary
+
+A 49.5M-parameter intent classifier trained from scratch on a 127K-example
+synthetic corpus for Pace's 7-class intent taxonomy. Runs in 3ms on CPU,
+beats Qwen3-4B-Instruct by 10.8 percentage points, and is 85x faster.
+
+This is a **posttrainllm-built specialist** — trained with the
+`train-extractor` command on a `ToolRouterModel` small preset (6 layers,
+384 d_model, byte-level vocab 256). No pre-trained weights, no
+distillation from a teacher. The model learns Pace-specific decision
+boundaries from synthetic data generated by
+`pace/scripts/generate-intent-corpus-v2.py`.
+
+## Artifact
+
+- Package id: `pace-intent-router-v8`
+- Artifact: `runs/pace-intent-router-v8.tinygpt` (566 MB)
+- Labels: `runs/pace-intent-router-v8.tinygpt.labels.json`
+- Format: posttrainllm `.tinygpt` (ToolRouterModel checkpoint)
+- Base: `posttrainllm-ToolRouterModel-small` (trained from scratch)
+- Params: 49.5M (6 layers, 384 d_model, 7 classes)
+- Training method: `train-extractor` (cross-entropy classification,
+  AdamW + cosine decay, 18000 steps on v5 data)
+- Training time: ~30 minutes on M5 Pro
+
+## Measured Result
+
+### Head-to-head vs Qwen3-4B-Instruct (4-bit)
+
+| Metric | ToolRouterModel v8 | Qwen3-4B-Instruct | Delta |
+|---|---:|---:|---:|
+| Overall accuracy | **95.5%** | 84.75% | **+10.8 pp** |
+| Eval size | 14,995 | 997 (stratified sample) | — |
+| Latency p50 | **3.1 ms** | 240 ms | **77x faster** |
+| Model size | **566 MB** | ~2,300 MB | **4x smaller** |
+| Params | 49.5M | 4B (4-bit) | — |
+
+### Per-class accuracy (v8 vs v5 vs Qwen3-4B)
+
+| Class | v8 (49.5M) | v5 (49.5M) | Qwen3-4B |
+|---|---:|---:|---:|
+| chitchat | 93.8% | **95.6%** | 78.5% |
+| pureKnowledge | 96.4% | **97.0%** | 83.0% |
+| screenDescription | **97.6%** | 97.1% | **95.8%** |
+| screenAction | 96.8% | **97.4%** | 91.0% |
+| research | 93.4% | **97.1%** | **100.0%** |
+| phoneLargeModel | **98.4%** | 96.4% | 76.7% |
+| unknown | **79.6%** | 70.3% | 2.6% |
+| **Overall** | 95.5% | **95.9%** | 84.75% |
+
+### Key takeaways
+
+1. **+10.8 pp over Qwen3-4B**: the 49.5M specialist beats the 4B
+   generalist on this task. Intent classification is a narrow enough
+   problem that a from-scratch model outperforms a general LLM.
+2. **77x faster**: 3.1ms vs 240ms. The router runs in sub-4ms on CPU;
+   Qwen needs a full LLM forward pass.
+3. **Unknown class is the moat**: v8 scores 79.6% on unknown vs Qwen's
+   2.6%. Qwen almost never returns "unknown" — it forces everything
+   into a known class, which is risky for a router (wrong route →
+   wrong pipeline). The trained model knows when to say "I don't know."
+4. **v5 vs v8**: v5 (12K steps) slightly outperforms v8 (18K steps)
+   overall (95.9% vs 95.5%), but v8 has a much better unknown class
+   (79.6% vs 70.3%, +9.3 pp). The longer training improved the hardest
+   class at a small cost to the easier ones.
+
+## Training data
+
+- Dataset: pace-intent-corpus-v2 (fully synthetic)
+- Source: `pace/scripts/generate-intent-corpus-v2.py` (base corpus,
+  combinatorial expansion) + `pace/scripts/generate-intent-supplement-v2.py`
+  (targeted weak-spot supplement)
+- Train rows: 112,061
+- Heldout rows: 14,995
+- Classes: 7 (chitchat, pureKnowledge, screenDescription, screenAction,
+  research, phoneLargeModel, unknown)
+- Split: stratified 85/15
+- No real user data — all synthetic
+
+### Why synthetic data works here
+
+The synthetic corpus encodes Pace-specific decision boundaries that a
+general LLM doesn't know:
+- "turn on lights" = unknown (Pace can't control lights)
+- "turn on volume" = screenAction (Pace can control volume)
+- "what can you do" = pureKnowledge (not unknown — it's a question
+  about Pace itself)
+
+These boundaries are product-specific, not language-general. A
+from-scratch model learns them directly from the corpus; a general LLM
+needs few-shot examples or fine-tuning to learn them.
+
+## Recommended Use
+
+This model is a **router**, not a planner. It classifies a user
+utterance into one of 7 intent classes in 3ms. The intent class
+determines which pipeline handles the turn:
+
+| Intent | Route | Model |
+|---|---|---|
+| chitchat | fast path | Apple FM / local text-only |
+| pureKnowledge | answer directly | Apple FM / local text-only |
+| screenDescription | read screen | local planner + VLM |
+| screenAction | execute tool | local planner + action layer |
+| research | research tier | codex CLI / Claude CLI |
+| phoneLargeModel | cloud bridge | codex CLI / cloud bridge |
+| unknown | full pipeline | local planner (best-effort) |
+
+Do not use this model for response generation. It only classifies.
+
+## Known Limits
+
+- **Unknown class is synthetic**: the 79.6% unknown accuracy is measured
+  on synthetic unknowns. Real-world unknown distribution will differ.
+  The model needs real-world unknown examples to improve further.
+- **Not wired into Pace**: the shipping app uses Apple FM for intent
+  classification (when available) and rule-based fallback. This model
+  is a training pipeline artifact, not a product component.
+- **Byte-level vocab**: the model uses a 256-token byte-level vocabulary.
+  Longer queries are truncated at 128 bytes. A BPE tokenizer would
+  generalize better on longer queries.
+- **No real user data**: all training data is synthetic. Real user
+  queries will have different phrasings, accents, and edge cases.
+
+## Why this model exists
+
+The Pace intent classifier was originally rule-based (hand-written
+phrase lists). This model was trained to test whether a learned
+classifier could beat the rules. It can — 95.5% vs the rules' ~95% on
+the synthetic eval — and it also beats Apple Foundation Models (3B,
+in-process) on the same task: 95.5% vs 76.5% on a 200-example
+stratified sample. The model is most useful as:
+
+1. **The production intent classifier** — it's faster (3ms vs 1600ms),
+   more accurate (95.5% vs 76.5%), and has calibrated confidence
+   (real softmax vs hardcoded 0.95)
+2. A fallback when Apple FM is unavailable (older Macs)
+3. A training pipeline validation (the corpus and pipeline are assets)
+4. A baseline for future on-device classifiers
+
+### Measured head-to-head (200 stratified examples, 1s delay)
+
+| Model | Accuracy | Latency | Error rate |
+|---|---|---|---|
+| **TinyGPT v8 (49.5M)** | **95.5%** | **3.1ms p50** | 0% |
+| Qwen3-4B-Instruct (4-bit) | 84.75% | 240ms p50 | 0% |
+| Apple FM (3B, in-process) | 76.5% | 1597ms mean | 0.5% |
+
+FM's main weakness: it conflates `research` with `pureKnowledge`
+(33.3% vs 93.4% — a 60pp gap) because it doesn't understand the
+Pace-specific distinction between "research X" (multi-step) and
+"what is X" (single answer). FM also refused to answer one query
+("model refused to answer").
+
+## References
+
+- Factory run report: `runs/2026-07-13-pace-intent-router-v1/report.md`
+- Eval data: `runs/2026-07-13-pace-intent-router-v1/eval-candidate-v8.json`
+- Head-to-head: `runs/2026-07-13-pace-intent-router-v1/head-to-head.json`
+- Training config: `runs/2026-07-13-pace-intent-router-v1/config.json`
+- Corpus generator: `pace/scripts/generate-intent-corpus-v2.py`
+- Supplement generator: `pace/scripts/generate-intent-supplement-v2.py`
+- Pace routing architecture: `pace/leanring-buddy/PaceIntentClassifier.swift`

config.json

@@ -0,0 +1,36 @@
+{
+  "model_type": "tool_router",
+  "architecture": "ToolRouterModel",
+  "preset": "small",
+  "layers": 6,
+  "d_model": 384,
+  "vocab_size": 256,
+  "context_length": 128,
+  "num_labels": 7,
+  "labels": [
+    "chitchat",
+    "phoneLargeModel",
+    "pureKnowledge",
+    "research",
+    "screenAction",
+    "screenDescription",
+    "unknown"
+  ],
+  "precision": "float32",
+  "params": 49500000,
+  "training_method": "train-extractor",
+  "framework": "posttrainllm",
+  "library_name": "posttrainllm",
+  "license": "mit",
+  "language": ["en"],
+  "pipeline_tag": "text-classification",
+  "tags": [
+    "posttrainllm",
+    "tinygpt",
+    "intent-classification",
+    "routing",
+    "pace",
+    "mac-first",
+    "on-device"
+  ]
+}

eval_report.json

@@ -0,0 +1,86 @@
+{
+  "model": "pace-intent-router-v8",
+  "eval_date": "2026-07-13",
+  "eval_split": "pace-intent-heldout-v5",
+  "eval_size": 14995,
+  "overall_accuracy": 0.9552517505835278,
+  "latency_ms": {
+    "p50": 3.1020641326904297,
+    "mean": 3.2842456757843435
+  },
+  "per_class": {
+    "chitchat": {
+      "accuracy": 0.9381672597864769,
+      "count": 2248
+    },
+    "phoneLargeModel": {
+      "accuracy": 0.9844444444444445,
+      "count": 450
+    },
+    "pureKnowledge": {
+      "accuracy": 0.964,
+      "count": 3000
+    },
+    "research": {
+      "accuracy": 0.9344444444444444,
+      "count": 900
+    },
+    "screenAction": {
+      "accuracy": 0.9678279713285548,
+      "count": 5999
+    },
+    "screenDescription": {
+      "accuracy": 0.9760978321289605,
+      "count": 1799
+    },
+    "unknown": {
+      "accuracy": 0.7963272120200334,
+      "count": 599
+    }
+  },
+  "baselines": {
+    "apple-fm-3b": {
+      "overall_accuracy": 0.765,
+      "eval_size": 200,
+      "latency_mean_ms": 1597,
+      "error_rate": 0.005,
+      "source": "pace/leanring-buddyTests/PaceFMIntentClassifierBenchmarkTests.swift",
+      "notes": "200 stratified examples, 1s delay between calls, fresh session per call. FM conflates research with pureKnowledge (33.3% vs 93.4%). One refusal."
+    },
+    "qwen3-4b-instruct-4bit": {
+      "overall_accuracy": 0.847542627883651,
+      "eval_size": 997,
+      "latency_p50_ms": 240,
+      "source": "runs/2026-07-13-pace-intent-router-v1/eval-qwen-baseline.json"
+    },
+    "pace-intent-router-v5": {
+      "overall_accuracy": 0.9590530176725576,
+      "eval_size": 14995,
+      "latency_p50_ms": 4.642009735107422,
+      "source": "runs/2026-07-13-pace-intent-router-v1/eval-candidate-v5.json"
+    }
+  },
+  "head_to_head": {
+    "vs_apple_fm": {
+      "accuracy_delta_pp": 19.0,
+      "latency_speedup": "515x",
+      "note": "FM is 3B in-process but beats it on every class. FM conflates research with pureKnowledge."
+    },
+    "vs_qwen3_4b": {
+      "accuracy_delta_pp": 10.77,
+      "latency_speedup": "77x",
+      "size_ratio": "4x smaller"
+    },
+    "vs_v5": {
+      "accuracy_delta_pp": -0.38,
+      "unknown_class_delta_pp": 9.35,
+      "note": "v8 trades slight overall accuracy for significantly better unknown-class recall"
+    }
+  },
+  "caveats": [
+    "All eval data is synthetic — real-world distribution will differ",
+    "Unknown class accuracy (79.6%) is measured on synthetic unknowns",
+    "Not wired into shipping Pace app yet — Apple FM is the current production classifier despite lower measured accuracy (76.5% vs 95.5%)",
+    "Byte-level vocab truncates queries at 128 bytes"
+  ]
+}

pace-intent-router-v8.tinygpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adf28f05f3e93960989f7b2417c0dda62210be775b2d517a4b99b59966a98a9e
+size 593849532

pace-intent-router-v8.tinygpt.labels.json

@@ -0,0 +1,13 @@
+{
+  "kind" : "tool-router",
+  "labels" : [
+    "chitchat",
+    "phoneLargeModel",
+    "pureKnowledge",
+    "research",
+    "screenAction",
+    "screenDescription",
+    "unknown"
+  ],
+  "version" : 1
+}

prompt.md

@@ -0,0 +1,50 @@
+# Pace Intent Router — classification prompt
+
+This model is a classifier, not a generative model. It does not use a
+prompt in the traditional sense. The input is a raw user transcript
+(byte-level encoded, max 128 bytes) and the output is one of 7 class
+labels.
+
+## Input format
+
+```
+<raw user transcript, byte-level encoded, max 128 bytes>
+```
+
+## Output format
+
+```
+<single class label from: chitchat, pureKnowledge, screenDescription,
+screenAction, research, phoneLargeModel, unknown>
+```
+
+## Class definitions
+
+- **chitchat**: Greetings, thanks, goodbyes, social filler. e.g. "hi
+  pace", "thanks", "how are you"
+- **pureKnowledge**: Factual questions answerable without screen
+  context. e.g. "what is HTML", "explain DNS"
+- **screenDescription**: User wants a description of what's on screen.
+  e.g. "what am I looking at", "what's on the screen"
+- **screenAction**: User wants Pace to do something via the action
+  layer. e.g. "click the save button", "open Safari"
+- **research**: Multi-step research turn. e.g. "research quantum
+  computing", "compare AWS vs GCP"
+- **phoneLargeModel**: Explicit escalation request. e.g. "use the big
+  model", "phone a large model"
+- **unknown**: Classifier could not confidently assign a class. The
+  caller must run the full pipeline.
+
+## Decision boundaries (Pace-specific)
+
+These boundaries are encoded in the synthetic training corpus and are
+what the model learns. A general LLM does not know these:
+
+- "turn on lights" = **unknown** (Pace can't control lights)
+- "turn on volume" = **screenAction** (Pace can control volume)
+- "what can you do" = **pureKnowledge** (not unknown — it's a question
+  about Pace itself)
+- "research HTML" = **research** (not pureKnowledge — the word
+  "research" triggers the research lane)
+- "I researched HTML yesterday" = **pureKnowledge** (past tense — not
+  a research request)

tinygpt.lock.json

@@ -0,0 +1,77 @@
+{
+  "schema": "posttrainllm.specialist.lock.v1",
+  "id": "pace-intent-router-v8",
+  "artifact_type": "tinygpt_checkpoint",
+  "local_path": "runs/pace-intent-router-v8.tinygpt",
+  "public_uri": null,
+  "storage": {
+    "primary": "local_runs_directory",
+    "status": "local-only",
+    "large_weights_published": false,
+    "r2": "not-applicable"
+  },
+  "base_model": {
+    "id": "posttrainllm-ToolRouterModel-small",
+    "preset": "small",
+    "architecture": "ToolRouterModel",
+    "precision": "float32",
+    "layers": 6,
+    "d_model": 384,
+    "vocab_size": 256,
+    "context_length": 128
+  },
+  "compatibility": {
+    "posttrainllm_serve": true,
+    "posttrainllm_sample": true,
+    "mlx_lm": false,
+    "browser_playground": false,
+    "pace_app_integration": false
+  },
+  "files": [
+    {
+      "path": "runs/pace-intent-router-v8.tinygpt",
+      "bytes": 593498112,
+      "role": "model_checkpoint"
+    },
+    {
+      "path": "runs/pace-intent-router-v8.tinygpt.labels.json",
+      "bytes": 200,
+      "role": "label_map"
+    }
+  ],
+  "training": {
+    "method": "train-extractor",
+    "steps": 18000,
+    "batch_size": 32,
+    "learning_rate": 0.0003,
+    "warmup_steps": 100,
+    "val_split": 0.05,
+    "val_every": 100,
+    "seed": 42,
+    "train_rows": 112061,
+    "heldout_rows": 14995,
+    "classes": [
+      "chitchat",
+      "phoneLargeModel",
+      "pureKnowledge",
+      "research",
+      "screenAction",
+      "screenDescription",
+      "unknown"
+    ],
+    "data_source": "pace/scripts/generate-intent-corpus-v2.py + generate-intent-supplement-v2.py",
+    "data_type": "fully_synthetic"
+  },
+  "package_files": {
+    "model_card": "model_card.md",
+    "prompt": "prompt.md",
+    "eval_report": "eval_report.json",
+    "loader": "mlx_load.py"
+  },
+  "notes": [
+    "Local-only artifact — not published to Hugging Face Hub.",
+    "Use posttrainllm serve or posttrainllm sample to run inference.",
+    "Not wired into the Pace shipping app — Apple FM is the production classifier.",
+    "The v5 checkpoint (12000 steps) slightly outperforms v8 on overall accuracy but v8 has better unknown-class recall."
+  ]
+}