File size: 3,473 Bytes
9f5fbb5 271bd15 9f5fbb5 271bd15 9f5fbb5 271bd15 9f5fbb5 271bd15 9f5fbb5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | # Results: two files, one tensor apart
Date: 2026-09-04. Two runs of the same benchmarks against two model files
at temperature 0.0. Method: [`benchmarking.md`](./benchmarking.md)
(harness, settings, adaptations, extraction rules — everything needed
to reproduce).
## Setup
- File A: `TheDrummer_Orion-26B-A4B-v1-IQ4_NL.gguf` (Gemma 4 26B MoE,
IQ4), served via llama-server, unmodified.
- File B: `lm_head-TheDrummer_Orion-26B-A4B-v1-IQ4_NL.gguf` — file A with
its `lm_head` tensor replaced via `voice cast`, Q8_0. Nothing else changed.
- Both files: temperature 0.0, seed 1234, identical prompts, identical token
budgets (ARC 1024, MMLU 2048, GSM8K 1024), identical answer extraction.
Unanswered items got one rescue pass at 4096 (MMLU) / 2048 (GSM8K),
applied to both files symmetrically. Rescue tie-break: on split pairs
(two verdicts for one doc) the decided verdict beats `unanswered`.
## Scores
| Task | n | A correct | B correct | A acc | B acc |
|---|---|---|---|---|---|
| ARC-Challenge (generative) | 100 | 94 | 95 | 0.940 | 0.950 |
| MMLU (generative, 2 × 57 subjects) | 114 | 101 | 103 | 0.886 | 0.904 |
| GSM8K | 100 | 76 | 92 | 0.760 | 0.920 |
## Observations
1. **ARC (+1) and MMLU (+2) differences fall within slice noise** (±5
points at n≈100). GSM8K (+16) exceeds it; see point 2.
2. **The GSM8K difference sits in unanswered counts.** File A left 23
items silent (empty content after thought exhausted the budget); file
B left 6. On items both files answered, accuracy is 0.987 (A) vs
0.979 (B). The traces hold whatever else there is to see.
3. **Thinking lengths roughly match.** No pairing script was archived, so no
median is claimed; per-file thinking lengths sit in the same range.
4. **Error overlap is minimal.** One GSM8K item was wrong under both
files; remaining errors are different items per file.
## What this does not show
- Small slices: ~100 items per task. Differences under ~5 points are noise.
- Generative scoring is not comparable to published logprob leaderboards;
compare only file A vs. file B within this table.
- One model pair, one replaced tensor, greedy decoding only. Other pairs
may differ — that is what the harness in `benchmarking/` is for.
- Writing voice is measured separately below — same two files, same
neutrality, no judge model.
## Evidence
Per file, archived under `benchmarking/runs/file-a/` and
`benchmarking/runs/file-b/`: harness `results_*.json` and
`samples_*.jsonl` for every run, rescue files for the second passes, and
the thinking log (every prompt, answer, and reasoning trace). No personal
information in any of it — server file paths are redacted to run labels
at the log source.
## Part II — writing voice: 32 prompts, two files
Method: [`creative-bench.md`](./creative-bench.md). 32 EQ-Bench v3 prompts,
one generation per prompt per file at temperature 0.7, 6000-token budget.
Measured, not judged: shared trigram vocabulary, cliche density, texture
stats. Blind A/B pairs archived for readers.
| Measure | File A | File B |
|---|---|---|
| Shared trigram vocab, mean (max) | 0.013 (0.027) across all 32 | same pair |
| Cliche hits per 100 words | 0.141 | 0.075 |
| Avg words per piece | 1018.0 | 878.6 |
| Lexical diversity (TTR) | 0.4439 | 0.5054 |
| Avg sentence words | 9.3 | 8.8 |
Per-prompt exhibits: `creative-bench/runs/file-a-creative.jsonl` and
`file-b-creative.jsonl` (prompt, full text, full reasoning trace each).
|