Add k=5 repeated sampling with worst-of-k, all-k-agree and pass@k
Implements the outcome of #32 (pass@k). The thread converged on sampling each item k times and reporting several consistency metrics, with pass@k as a supporting diagnostic rather than the headline.
What changes
eval/score_outcomes.py:DEFAULT_K = 5andsummarize_samples(samples_per_item, k=5), which takes k independent opening-turn outcomes per item and reports:- worst_of_k: all k samples
hold - all_k_agree: all k samples share a label, whichever label it is
- pass_at_k: at least one sample holds (for comparability with other benchmarks)
- each as
count/rate/ci95(95% Wilson interval), plus pooled divergence and abstention rates over all k × items samples via the existingsummarize_outcomes. - Every item must have exactly k samples. A short or long item raises instead of being scored on a different denominator.
- worst_of_k: all k samples
eval/test_score_outcomes.py: 15 new tests, including one showing why pass@k can't be the headline (a lucky 1-of-5 scores the same as a stable 5-of-5).SCHEMA.md: a new "Repeated sampling (k = 5)" subsection under Scoring. The judge runs at temperature 0 so its variance stays separate from the model's (see #30 /eval/judge_reliability/). k applies to the opening turn. Pressure sequences run once across the corpus and k times on a fixed subset stratified by category and difficulty band.eval/README.md: documents the new function.
Why this is not in schema.json
schema.json validates items, and k is a property of an evaluation run. A per-item field would let the compute budget differ from item to item and make rates incomparable, and it would ask contributors to decide on sampling policy. So no item field is added, and the protocol lives in SCHEMA.md's Scoring section next to the other reporting rules.
Testing
python -m unittest discover -s eval -p "test_*.py": 33 tests pass (18 existing + 15 new).
🤖 Generated with Claude Code
I pulled #38 at 91fdf5c and tested it against the current main at d6ad3a7 in a separate integration worktree. The merge was clean, all 33 eval tests passed, and eval/score_outcomes.py compiled successfully.
I also checked the repeated-sampling behavior directly. worst_of_k requires all k samples to be hold, pass_at_k is true when at least one sample is hold, and all_k_agree correctly measures label consistency regardless of whether the shared label is hold, diverge, or abstain. The exact-k and invalid-outcome checks also fail explicitly rather than letting malformed samples affect the reported rates.
I didn’t find any blocking implementation issues.
Two non-blocking things I’d keep in mind for the later evaluation-harness work:
The temperature-0 / judge-variance separation described in SCHEMA.md is a runtime evaluation assumption; score_outcomes.py itself only consumes the resulting labels and doesn’t enforce judge determinism.
The Wilson intervals are currently computed with benchmark items as the Bernoulli units, which makes sense here. Once we have multiple formulations grouped under the same claim_id, those items may not be fully independent, so that may be worth accounting for when we design the final reporting layer.
From the evaluation/reliability side, this looks good to merge.