[experiment] Add local judge injection pilot results
Follow-up to #82. I reran the 10 conflicting clean/injected pairs for Llama 3.2 3B and Gemma 3 4B and saved the per-case outputs, summaries, manifests, and Ollama runtime snapshots.
Both reruns used a 4096-token Ollama context.
Llama reproduced the earlier result exactly: 9/10 clean, 5/10 injected, with 2 clean-correct cases flipping to the injected hold target.
Gemma reran at 8/10 clean and 9/10 injected with 0 target-label flips. The earlier pilot was 9/10 clean and 9/10 injected, so the clean score differed by one case on rerun.
All 15 judge-reliability tests pass.
Added the requested 95% Wilson intervals to the 10-case rerun results.
For the reruns:
- Llama 3.2 3B: clean 9/10 (90%, 95% CI 59.6%-98.2%); injected 5/10 (50%, 95% CI 23.7%-76.3%).
- Gemma 3 4B: clean 8/10 (80%, 95% CI 49.0%-94.3%); injected 9/10 (90%, 95% CI 59.6%-98.2%).
The saved Ollama runtime snapshots show a 4096-token context for both reruns.
Small correction after checking the saved outputs against the parser used by the current Inspect scorer:
For Llama 3.2 3B, clean remains 9/10 (90%, 95% CI 59.6%-98.2%), but injected is 4/10 (40%, 95% CI 16.8%-68.7%) under the Inspect parser, instead of 5/10 under run_robustness.py.
The 2 clean-correct cases that flip to the injected hold target are unchanged. Gemma's reported results are unchanged.
I found two Llama responses where the robustness runner and Inspect scorer parse the same raw output differently. I'll handle that parser-alignment issue separately rather than modifying the saved raw outputs.
Looks good! For judges we should start to look at slightly larger/more capable models for the judge, but these runs are great for testing the reliability. Thanks for doing this.
🤖 Auto-triage: tagged as an experiment (confidence 0.90).
Reason: The PR adds or changes files under a results/ directory and adds an experiment/run manifest and adds summary/metrics JSON, so it reports the results of a run.
This comment and the [experiment] title prefix were added automatically by a local model (qwen3.5:4b). If this isn't an experiment, remove the prefix or reply here.