Benchmark Summary
Full Checkpoint
| Suite | F1 | Examples/s |
|---|---|---|
| Irish core | 0.9935 | 37.7313 |
| Irish extended | 1.0000 | 22.0184 |
| Demographic holdout v2 | 0.9988 | 58.4569 |
| Gov contact policy v1 | 0.9836 | 22.1918 |
| Gov chatbot red-team v2 | 0.9726 | 30.4434 |
| Gov chatbot gap holdout v2 | 0.9969 | 49.7465 |
| Multilingual PPSN overall | 0.9222 | 57.2856 |
| Multilingual PPSN label-only | 0.9940 | — |
ONNX q8
| Suite | F1 | Examples/s |
|---|---|---|
| Irish core | 0.9935 | 61.4970 |
| Irish extended | 1.0000 | 98.7502 |
| Demographic holdout v2 | 0.9988 | 90.7807 |
| Gov contact policy v1 | 0.9836 | 29.7205 |
| Gov chatbot red-team v2 | 0.9726 | 13.0422 |
| Gov chatbot gap holdout v2 | 0.9969 | 30.7289 |
| Multilingual PPSN overall | 0.9222 | 33.5447 |
| Multilingual PPSN label-only | 0.9940 | — |
Comparison
| Artifact | Core F1 | Demographic Holdout v2 F1 | Gov Contact Policy v1 F1 | Multilingual F1 | Core examples/s |
|---|---|---|---|---|---|
| ContextPII q8 | 0.9935 | 0.9988 | 0.9836 | 0.9222 | 61.4970 |
| GlobalPointer rc4 q8 | 1.0000 | 0.6938 | 0.7843 | 0.9333 | 221.5743 |
| DiffMask rc6 q8 | 0.9733 | — | — | 0.9274 | 130.3415 |