Spaces:
Running
Running
Deploy comtrade_env: green consistency + scope clarifications + landing page
Browse files- .gitattributes +1 -0
- README.md +20 -24
- benchmark_results.png +2 -2
- grpo_envelope.png +3 -0
- llm_results_qwen7b_zeroshot.json +185 -0
.gitattributes
CHANGED
|
@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
benchmark_results.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
training_curve.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
benchmark_results.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
training_curve.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
grpo_envelope.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -38,10 +38,11 @@ tags:
|
|
| 38 |
|
| 39 |
### An OpenEnv Benchmark for Reliable LLM Tool-Use Under Adversarial API Conditions
|
| 40 |
|
| 41 |
-
What happens when **Kimi, Claude, GPT-5, and
|
| 42 |
|
| 43 |
- Kimi and Claude produce **numerically identical** scores on every single task — not close, *identical* (97.5 avg each).
|
| 44 |
-
-
|
|
|
|
| 45 |
- Llama is *bimodal* on that same task, spanning 18.7 to 97.5 across seeds — the discriminative signal is **reliability**, not capability.
|
| 46 |
|
| 47 |
And when we try to **GRPO-train** a Qwen2.5-3B on the benchmark? It enters the learning window at iter 3, trains cleanly for 14 iterations, then **policy-collapses at iter 15** into a degenerate output region.
|
|
@@ -55,8 +56,8 @@ ComtradeBench surfaces these failure modes because it measures **execution relia
|
|
| 55 |
|
| 56 |
> **For judges — 30-second summary**:
|
| 57 |
> - Ten-task OpenEnv benchmark for LLM agent reliability under adversarial API conditions (429/500, pagination drift, duplicates, totals traps, within-episode fault escalation, constrained budgets).
|
| 58 |
-
> - **
|
| 59 |
-
> - **
|
| 60 |
> - **GRPO operating envelope mapped at three points** (under-capacity / learn-then-collapse / saturation) — an actionable finding, not a "we trained something" claim.
|
| 61 |
> - Results live and reproducible in the HF Docker Space.
|
| 62 |
|
|
@@ -79,36 +80,30 @@ The ten tasks cover pagination, deduplication, 429 / 500 retries, non-determinis
|
|
| 79 |
| Agent | Avg (T1-T10) | T9 | Notes |
|
| 80 |
|---|---:|---:|---|
|
| 81 |
| Rule-based baseline | 96.8 | 96.9 | deterministic, no LLM |
|
| 82 |
-
| **Kimi Moonshot V1-128k** | **97.5** | **97.5** | multi-seed std = 0.0
|
| 83 |
-
| **Claude Sonnet 4.6** | **97.5** | **97.5** |
|
|
|
|
| 84 |
| **GPT-5** | 93.2 | **75.7** | reasoning-oriented: 2 steps in 223 s vs Kimi's 7 steps in 8 s |
|
| 85 |
| Llama 3.3 70B (Groq) | 89.3 | 18.7–97.5† | bimodal across seeds |
|
| 86 |
|
| 87 |
† Llama T9 is **bimodal**: the published seed-42 run hit 18.7, multi-seed re-run produced {97.5, 94.5, …} — the low number and the near-frontier numbers both reproduce. Raw per-seed data in `multiseed_llama_t9_summary.json`.
|
| 88 |
|
| 89 |
-
|
|
|
|
|
|
|
| 90 |
|
| 91 |
1. **T9 separates execution-oriented from reasoning-oriented frontier.** Kimi and Claude execute T9 in ~8 s across 7 tool calls and score 97.5. GPT-5 "thinks" for 223 s across 2 tool calls and scores 75.7 — a **21.8-point gap between frontier models** that a pass/fail benchmark would miss entirely. The breakdown tells the story: GPT-5's Efficiency drops to 6/15 (budget burned in reasoning-time) and Observability to ~4/10 (2 steps leave no audit trail).
|
| 92 |
-
2. **Frontier saturates at the top.** Kimi and Claude produce *numerically identical* per-task scores on all 10 tasks. Same seeded environment, same deterministic judge, same solve-path → same score.
|
| 93 |
-
3. **Sub-frontier is high-variance, not uniformly weak.** Kimi T9 std = 0.0 across 5 seeds. Llama T9 spans 18.7 – 97.5. The discriminative signal is *reliability*, not capability: Llama can sometimes match frontier, just not consistently.
|
|
|
|
| 94 |
|
| 95 |
### GRPO training — operating envelope empirically mapped at three points
|
| 96 |
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
(reward_std ≈ 0 → no gradient)
|
| 101 |
-
/
|
| 102 |
-
🟢 Qwen2.5-3B + LoRA
|
| 103 |
-
learns 14 iter (kl 8e-6 → 5.6e-4)
|
| 104 |
-
then policy-collapses at iter 15
|
| 105 |
-
/
|
| 106 |
-
🟡 Qwen2.5-1.5B (50 iter full-param)
|
| 107 |
-
noise-dominated, no net upward trend
|
| 108 |
-
(reward 0.22–0.94, under-capacity on T9/T10)
|
| 109 |
-
```
|
| 110 |
|
| 111 |
-
**Reading the
|
| 112 |
|
| 113 |
All training data is committed as artifacts: `grpo_gradient_training.jsonl` (1.5B per-iter metrics), `grpo_gradient_training_3b.jsonl` (3B per-iter, 15 entries), `grpo_3b_lora_collapse.json` (3B interpretation), `grpo_7b_lora_5iter_saturation.json` (7B interpretation). The same environment code runs in-process during GRPO rollouts and as the deployed Docker service during eval — zero divergence.
|
| 114 |
|
|
@@ -410,13 +405,14 @@ Kimi-128k matches or slightly exceeds the rule-based baseline on **all 10 tasks*
|
|
| 410 |
gap on T4/T5 Robustness (12/15, not 15/15) is a scoring sub-criterion explored in the ablation
|
| 411 |
below, not a silent-retry failure.
|
| 412 |
|
| 413 |
-
### Cross-model comparison —
|
| 414 |
|
| 415 |
| Model | Avg (T1-T10) | T1-T8 avg | T9 score | T10 score |
|
| 416 |
|---|---:|---:|---:|---:|
|
| 417 |
| Rule-based baseline | 96.8 | 96.5 | 96.9 | 98.0 |
|
| 418 |
| **Kimi Moonshot V1-128k** | **97.5** | **97.4** | **97.5** (std 0.0 across 5 seeds) | **98.7** |
|
| 419 |
| **Claude Sonnet 4.6** | **97.5** | **97.4** | **97.5** | **98.7** |
|
|
|
|
| 420 |
| **GPT-5** | 93.2 | 95.0 | **75.7** | 95.7 |
|
| 421 |
| Llama 3.3 70B (Groq) | 89.3 | 97.4 | 18.7 – 97.5 (bimodal†) | 95.7 |
|
| 422 |
|
|
|
|
| 38 |
|
| 39 |
### An OpenEnv Benchmark for Reliable LLM Tool-Use Under Adversarial API Conditions
|
| 40 |
|
| 41 |
+
What happens when **Kimi, Claude, GPT-5, Llama, and open-source Qwen2.5-7B** all run the same 10 API tasks against the same seeded environment and the same deterministic 6-dimensional judge?
|
| 42 |
|
| 43 |
- Kimi and Claude produce **numerically identical** scores on every single task — not close, *identical* (97.5 avg each).
|
| 44 |
+
- **Open-source Qwen2.5-7B-Instruct, zero-shot (no training)**, scores **97.2 — within 0.3 of closed-source frontier**, above baseline, above GPT-5 and Llama 70B.
|
| 45 |
+
- GPT-5 scores **21.8 points lower** than frontier on one specific task — the one with mid-episode fault escalation. Not because it's less capable, but because it *reasons for 223 seconds across 2 tool calls* instead of *executing 7 tool calls in 8 seconds*.
|
| 46 |
- Llama is *bimodal* on that same task, spanning 18.7 to 97.5 across seeds — the discriminative signal is **reliability**, not capability.
|
| 47 |
|
| 48 |
And when we try to **GRPO-train** a Qwen2.5-3B on the benchmark? It enters the learning window at iter 3, trains cleanly for 14 iterations, then **policy-collapses at iter 15** into a degenerate output region.
|
|
|
|
| 56 |
|
| 57 |
> **For judges — 30-second summary**:
|
| 58 |
> - Ten-task OpenEnv benchmark for LLM agent reliability under adversarial API conditions (429/500, pagination drift, duplicates, totals traps, within-episode fault escalation, constrained budgets).
|
| 59 |
+
> - **Five LLMs evaluated cross-model**: Kimi Moonshot V1-128k, Claude Sonnet 4.6, **open-source Qwen2.5-7B-Instruct (zero-shot)**, GPT-5, Llama 3.3 70B. Plus three Qwen2.5 sizes trained with GRPO (1.5B full-param, 3B + LoRA, 7B + LoRA).
|
| 60 |
+
> - **Four independent findings**: (1) T9 separates execution-oriented from reasoning-oriented frontier (Kimi/Claude 97.5 vs GPT-5 75.7), (2) Kimi = Claude numerically identical → ceiling saturation, (3) Llama T9 bimodal → sub-frontier is about reliability not capability, (4) **⭐ Open-source Qwen2.5-7B zero-shot matches closed frontier (97.2 vs 97.5)** — benchmark solvable by well-instructed open 7B without any training.
|
| 61 |
> - **GRPO operating envelope mapped at three points** (under-capacity / learn-then-collapse / saturation) — an actionable finding, not a "we trained something" claim.
|
| 62 |
> - Results live and reproducible in the HF Docker Space.
|
| 63 |
|
|
|
|
| 80 |
| Agent | Avg (T1-T10) | T9 | Notes |
|
| 81 |
|---|---:|---:|---|
|
| 82 |
| Rule-based baseline | 96.8 | 96.9 | deterministic, no LLM |
|
| 83 |
+
| **Kimi Moonshot V1-128k** | **97.5** | **97.5** | closed-source frontier, multi-seed std = 0.0 |
|
| 84 |
+
| **Claude Sonnet 4.6** | **97.5** | **97.5** | closed-source frontier, identical to Kimi |
|
| 85 |
+
| **Qwen2.5-7B-Instruct** ⭐ | **97.2** | **97.5** | **open-source, zero-shot** (no fine-tuning, no training) |
|
| 86 |
| **GPT-5** | 93.2 | **75.7** | reasoning-oriented: 2 steps in 223 s vs Kimi's 7 steps in 8 s |
|
| 87 |
| Llama 3.3 70B (Groq) | 89.3 | 18.7–97.5† | bimodal across seeds |
|
| 88 |
|
| 89 |
† Llama T9 is **bimodal**: the published seed-42 run hit 18.7, multi-seed re-run produced {97.5, 94.5, …} — the low number and the near-frontier numbers both reproduce. Raw per-seed data in `multiseed_llama_t9_summary.json`.
|
| 90 |
|
| 91 |
+
**⭐ Open-source parity**: Qwen2.5-7B-Instruct, run *zero-shot* (no training, no fine-tuning), via Together AI → **97.2 / 100 avg, 97.5 on T9** — within 0.3 points of closed-source frontier (Kimi, Claude), above baseline, and **above GPT-5 by 4.0 points**. ComtradeBench is solvable by mid-size open-weights models at zero training cost. `llm_results_qwen7b_zeroshot.json`.
|
| 92 |
+
|
| 93 |
+
### Four independent findings from the cross-model evaluation
|
| 94 |
|
| 95 |
1. **T9 separates execution-oriented from reasoning-oriented frontier.** Kimi and Claude execute T9 in ~8 s across 7 tool calls and score 97.5. GPT-5 "thinks" for 223 s across 2 tool calls and scores 75.7 — a **21.8-point gap between frontier models** that a pass/fail benchmark would miss entirely. The breakdown tells the story: GPT-5's Efficiency drops to 6/15 (budget burned in reasoning-time) and Observability to ~4/10 (2 steps leave no audit trail).
|
| 96 |
+
2. **Frontier saturates at the top.** Kimi and Claude produce *numerically identical* per-task scores on all 10 tasks. Same seeded environment, same deterministic judge, same solve-path → same score.
|
| 97 |
+
3. **Sub-frontier is high-variance, not uniformly weak.** Kimi T9 std = 0.0 across 5 seeds. Llama T9 spans 18.7 – 97.5. The discriminative signal is *reliability*, not capability: Llama can sometimes match frontier, just not consistently.
|
| 98 |
+
4. **⭐ Open-source 7B closes the gap to frontier without training.** Qwen2.5-7B-Instruct, zero-shot (no fine-tuning, no GRPO), scores 97.2 — within 0.3 points of Kimi/Claude, above GPT-5 and Llama 3.3 70B. This changes the headline: **the benchmark is not "can a closed frontier LLM solve this"; it's "can an agent execute reliably", and a well-instructed open 7B matches that bar**. It also validates the GRPO saturation finding — 7B genuinely is at ceiling for this benchmark, which is why GRPO fine-tuning provides no gradient signal.
|
| 99 |
|
| 100 |
### GRPO training — operating envelope empirically mapped at three points
|
| 101 |
|
| 102 |
+

|
| 103 |
+
|
| 104 |
+
*Three training configurations, three distinct failure modes. 1.5B full-param: under-capacity, reward oscillates 0.22–0.94 with no trend. 3B + LoRA: learns cleanly for 14 iterations (KL grows monotonically 8e-6 → 5.6e-4), then policy-collapses at iter 15. 7B + LoRA: mean reward 0.987 at iter 1, already above baseline — GRPO advantage signal near zero, no gradient propagates.*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
+
**Reading the envelope**: the useful GRPO training band exists (iters 3-14 of the 3B run are empirical proof — real reward variance, monotonically growing KL), but it is **narrow and fragile**. Stable training on the 3B point requires adaptive KL penalty, tighter trust-region clipping, or early-stop on reward-variance collapse — engineering work we did not perform in this release. This is a more actionable finding than "training converged on some model": it names concrete failure modes a practitioner would hit.
|
| 107 |
|
| 108 |
All training data is committed as artifacts: `grpo_gradient_training.jsonl` (1.5B per-iter metrics), `grpo_gradient_training_3b.jsonl` (3B per-iter, 15 entries), `grpo_3b_lora_collapse.json` (3B interpretation), `grpo_7b_lora_5iter_saturation.json` (7B interpretation). The same environment code runs in-process during GRPO rollouts and as the deployed Docker service during eval — zero divergence.
|
| 109 |
|
|
|
|
| 405 |
gap on T4/T5 Robustness (12/15, not 15/15) is a scoring sub-criterion explored in the ablation
|
| 406 |
below, not a silent-retry failure.
|
| 407 |
|
| 408 |
+
### Cross-model comparison — five LLMs, four independent findings
|
| 409 |
|
| 410 |
| Model | Avg (T1-T10) | T1-T8 avg | T9 score | T10 score |
|
| 411 |
|---|---:|---:|---:|---:|
|
| 412 |
| Rule-based baseline | 96.8 | 96.5 | 96.9 | 98.0 |
|
| 413 |
| **Kimi Moonshot V1-128k** | **97.5** | **97.4** | **97.5** (std 0.0 across 5 seeds) | **98.7** |
|
| 414 |
| **Claude Sonnet 4.6** | **97.5** | **97.4** | **97.5** | **98.7** |
|
| 415 |
+
| **Qwen2.5-7B-Instruct (open, zero-shot)** ⭐ | **97.2** | **97.2** | **97.5** | **98.7** |
|
| 416 |
| **GPT-5** | 93.2 | 95.0 | **75.7** | 95.7 |
|
| 417 |
| Llama 3.3 70B (Groq) | 89.3 | 97.4 | 18.7 – 97.5 (bimodal†) | 95.7 |
|
| 418 |
|
benchmark_results.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
grpo_envelope.png
ADDED
|
Git LFS Details
|
llm_results_qwen7b_zeroshot.json
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "Qwen/Qwen2.5-7B-Instruct-Turbo",
|
| 3 |
+
"api_url": "https://api.together.xyz/v1",
|
| 4 |
+
"provider": "Together AI",
|
| 5 |
+
"note": "Open-source Qwen2.5-7B-Instruct evaluated ZERO-SHOT (no fine-tuning, no GRPO training). Same agent loop and default prompt as Kimi/Claude/GPT-5/Llama runs. seed=42, temperature=0.0. Run 2026-04-20 via Together AI API.",
|
| 6 |
+
"seed": 42,
|
| 7 |
+
"tasks_run": [
|
| 8 |
+
"T1_single_page",
|
| 9 |
+
"T2_multi_page",
|
| 10 |
+
"T3_duplicates",
|
| 11 |
+
"T4_rate_limit_429",
|
| 12 |
+
"T5_server_error_500",
|
| 13 |
+
"T6_page_drift",
|
| 14 |
+
"T7_totals_trap",
|
| 15 |
+
"T8_mixed_faults",
|
| 16 |
+
"T9_adaptive_adversary",
|
| 17 |
+
"T10_constrained_budget"
|
| 18 |
+
],
|
| 19 |
+
"average_score": 97.16,
|
| 20 |
+
"average_reward": 0.9716,
|
| 21 |
+
"results": [
|
| 22 |
+
{
|
| 23 |
+
"task_id": "T1_single_page",
|
| 24 |
+
"reward": 0.987,
|
| 25 |
+
"score": 98.7,
|
| 26 |
+
"breakdown": {
|
| 27 |
+
"correctness": 30.0,
|
| 28 |
+
"completeness": 15.0,
|
| 29 |
+
"robustness": 15.0,
|
| 30 |
+
"efficiency": 15.0,
|
| 31 |
+
"data_quality": 15.0,
|
| 32 |
+
"observability": 8.67
|
| 33 |
+
},
|
| 34 |
+
"error": null,
|
| 35 |
+
"steps": 3,
|
| 36 |
+
"elapsed_s": 2.1
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"task_id": "T2_multi_page",
|
| 40 |
+
"reward": 0.987,
|
| 41 |
+
"score": 98.7,
|
| 42 |
+
"breakdown": {
|
| 43 |
+
"correctness": 30.0,
|
| 44 |
+
"completeness": 15.0,
|
| 45 |
+
"robustness": 15.0,
|
| 46 |
+
"efficiency": 15.0,
|
| 47 |
+
"data_quality": 15.0,
|
| 48 |
+
"observability": 8.67
|
| 49 |
+
},
|
| 50 |
+
"error": null,
|
| 51 |
+
"steps": 7,
|
| 52 |
+
"elapsed_s": 4.0
|
| 53 |
+
},
|
| 54 |
+
{
|
| 55 |
+
"task_id": "T3_duplicates",
|
| 56 |
+
"reward": 0.987,
|
| 57 |
+
"score": 98.7,
|
| 58 |
+
"breakdown": {
|
| 59 |
+
"correctness": 30.0,
|
| 60 |
+
"completeness": 15.0,
|
| 61 |
+
"robustness": 15.0,
|
| 62 |
+
"efficiency": 15.0,
|
| 63 |
+
"data_quality": 15.0,
|
| 64 |
+
"observability": 8.67
|
| 65 |
+
},
|
| 66 |
+
"error": null,
|
| 67 |
+
"steps": 5,
|
| 68 |
+
"elapsed_s": 2.4
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"task_id": "T4_rate_limit_429",
|
| 72 |
+
"reward": 0.957,
|
| 73 |
+
"score": 95.7,
|
| 74 |
+
"breakdown": {
|
| 75 |
+
"correctness": 30.0,
|
| 76 |
+
"completeness": 15.0,
|
| 77 |
+
"robustness": 12.0,
|
| 78 |
+
"efficiency": 15.0,
|
| 79 |
+
"data_quality": 15.0,
|
| 80 |
+
"observability": 8.67
|
| 81 |
+
},
|
| 82 |
+
"error": null,
|
| 83 |
+
"steps": 5,
|
| 84 |
+
"elapsed_s": 4.0
|
| 85 |
+
},
|
| 86 |
+
{
|
| 87 |
+
"task_id": "T5_server_error_500",
|
| 88 |
+
"reward": 0.963,
|
| 89 |
+
"score": 96.3,
|
| 90 |
+
"breakdown": {
|
| 91 |
+
"correctness": 30.0,
|
| 92 |
+
"completeness": 15.0,
|
| 93 |
+
"robustness": 12.0,
|
| 94 |
+
"efficiency": 15.0,
|
| 95 |
+
"data_quality": 15.0,
|
| 96 |
+
"observability": 9.34
|
| 97 |
+
},
|
| 98 |
+
"error": null,
|
| 99 |
+
"steps": 5,
|
| 100 |
+
"elapsed_s": 4.1
|
| 101 |
+
},
|
| 102 |
+
{
|
| 103 |
+
"task_id": "T6_page_drift",
|
| 104 |
+
"reward": 0.947,
|
| 105 |
+
"score": 94.7,
|
| 106 |
+
"breakdown": {
|
| 107 |
+
"correctness": 26.0,
|
| 108 |
+
"completeness": 15.0,
|
| 109 |
+
"robustness": 15.0,
|
| 110 |
+
"efficiency": 15.0,
|
| 111 |
+
"data_quality": 15.0,
|
| 112 |
+
"observability": 8.67
|
| 113 |
+
},
|
| 114 |
+
"error": null,
|
| 115 |
+
"steps": 5,
|
| 116 |
+
"elapsed_s": 2.5
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"task_id": "T7_totals_trap",
|
| 120 |
+
"reward": 0.987,
|
| 121 |
+
"score": 98.7,
|
| 122 |
+
"breakdown": {
|
| 123 |
+
"correctness": 30.0,
|
| 124 |
+
"completeness": 15.0,
|
| 125 |
+
"robustness": 15.0,
|
| 126 |
+
"efficiency": 15.0,
|
| 127 |
+
"data_quality": 15.0,
|
| 128 |
+
"observability": 8.67
|
| 129 |
+
},
|
| 130 |
+
"error": null,
|
| 131 |
+
"steps": 5,
|
| 132 |
+
"elapsed_s": 18.5
|
| 133 |
+
},
|
| 134 |
+
{
|
| 135 |
+
"task_id": "T8_mixed_faults",
|
| 136 |
+
"reward": 0.939,
|
| 137 |
+
"score": 93.9,
|
| 138 |
+
"breakdown": {
|
| 139 |
+
"correctness": 25.2,
|
| 140 |
+
"completeness": 15.0,
|
| 141 |
+
"robustness": 15.0,
|
| 142 |
+
"efficiency": 15.0,
|
| 143 |
+
"data_quality": 15.0,
|
| 144 |
+
"observability": 8.67
|
| 145 |
+
},
|
| 146 |
+
"error": null,
|
| 147 |
+
"steps": 3,
|
| 148 |
+
"elapsed_s": 39.5
|
| 149 |
+
},
|
| 150 |
+
{
|
| 151 |
+
"task_id": "T9_adaptive_adversary",
|
| 152 |
+
"reward": 0.975,
|
| 153 |
+
"score": 97.5,
|
| 154 |
+
"breakdown": {
|
| 155 |
+
"correctness": 28.88,
|
| 156 |
+
"completeness": 15.0,
|
| 157 |
+
"robustness": 15.0,
|
| 158 |
+
"efficiency": 15.0,
|
| 159 |
+
"data_quality": 15.0,
|
| 160 |
+
"observability": 8.67
|
| 161 |
+
},
|
| 162 |
+
"error": null,
|
| 163 |
+
"steps": 7,
|
| 164 |
+
"elapsed_s": 4.4
|
| 165 |
+
},
|
| 166 |
+
{
|
| 167 |
+
"task_id": "T10_constrained_budget",
|
| 168 |
+
"reward": 0.987,
|
| 169 |
+
"score": 98.7,
|
| 170 |
+
"breakdown": {
|
| 171 |
+
"correctness": 30.0,
|
| 172 |
+
"completeness": 15.0,
|
| 173 |
+
"robustness": 15.0,
|
| 174 |
+
"efficiency": 15.0,
|
| 175 |
+
"data_quality": 15.0,
|
| 176 |
+
"observability": 8.67
|
| 177 |
+
},
|
| 178 |
+
"error": null,
|
| 179 |
+
"steps": 10,
|
| 180 |
+
"elapsed_s": 6.0
|
| 181 |
+
}
|
| 182 |
+
],
|
| 183 |
+
"key_finding": "Open-source Qwen2.5-7B-Instruct, zero-shot (no training), scores 97.2 / 100 on ComtradeBench — within 0.3 points of closed-source frontier (Kimi 97.5, Claude 97.5), 4.0 points above GPT-5 (93.2), and 7.9 points above Llama 3.3 70B (89.3). On T9 (the execution-vs-reasoning discriminator) Qwen2.5-7B matches frontier at 97.5. This validates: (a) ComtradeBench is solvable by mid-size open-weights models, (b) the benchmark measures execution reliability at a level where a good open 7B equals a closed frontier model, (c) our GRPO saturation finding at 7B is NOT an artifact — the 7B model genuinely is at ceiling for this benchmark.",
|
| 184 |
+
"generated_at": "2026-04-20T13:14"
|
| 185 |
+
}
|