yonghongzhang commited on
Commit
ab8b145
·
verified ·
1 Parent(s): 923d7dd

Deploy comtrade_env: green consistency + scope clarifications + landing page

Browse files
.gitattributes CHANGED
@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  benchmark_results.png filter=lfs diff=lfs merge=lfs -text
37
  training_curve.png filter=lfs diff=lfs merge=lfs -text
 
 
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  benchmark_results.png filter=lfs diff=lfs merge=lfs -text
37
  training_curve.png filter=lfs diff=lfs merge=lfs -text
38
+ grpo_envelope.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -38,10 +38,11 @@ tags:
38
 
39
  ### An OpenEnv Benchmark for Reliable LLM Tool-Use Under Adversarial API Conditions
40
 
41
- What happens when **Kimi, Claude, GPT-5, and Llama** all run the same 10 API tasks against the same seeded environment and the same deterministic 6-dimensional judge?
42
 
43
  - Kimi and Claude produce **numerically identical** scores on every single task — not close, *identical* (97.5 avg each).
44
- - GPT-5 scores **21.8 points lower** than both on one specific task — the one with mid-episode fault escalation. Not because it's less capable, but because it *reasons for 223 seconds across 2 tool calls* instead of *executing 7 tool calls in 8 seconds*.
 
45
  - Llama is *bimodal* on that same task, spanning 18.7 to 97.5 across seeds — the discriminative signal is **reliability**, not capability.
46
 
47
  And when we try to **GRPO-train** a Qwen2.5-3B on the benchmark? It enters the learning window at iter 3, trains cleanly for 14 iterations, then **policy-collapses at iter 15** into a degenerate output region.
@@ -55,8 +56,8 @@ ComtradeBench surfaces these failure modes because it measures **execution relia
55
 
56
  > **For judges — 30-second summary**:
57
  > - Ten-task OpenEnv benchmark for LLM agent reliability under adversarial API conditions (429/500, pagination drift, duplicates, totals traps, within-episode fault escalation, constrained budgets).
58
- > - **Four frontier LLMs evaluated** (Kimi / Claude / GPT-5 / Llama) + **three Qwen2.5 sizes trained** with GRPO (1.5B full-param, 3B + LoRA, 7B + LoRA).
59
- > - **Three independent discriminative signals**: (1) T9 separates execution-oriented from reasoning-oriented frontier (Kimi/Claude 97.5 vs GPT-5 75.7), (2) Kimi = Claude numerically identical → ceiling saturation, (3) Llama T9 bimodal → sub-frontier is about reliability not capability.
60
  > - **GRPO operating envelope mapped at three points** (under-capacity / learn-then-collapse / saturation) — an actionable finding, not a "we trained something" claim.
61
  > - Results live and reproducible in the HF Docker Space.
62
 
@@ -79,36 +80,30 @@ The ten tasks cover pagination, deduplication, 429 / 500 retries, non-determinis
79
  | Agent | Avg (T1-T10) | T9 | Notes |
80
  |---|---:|---:|---|
81
  | Rule-based baseline | 96.8 | 96.9 | deterministic, no LLM |
82
- | **Kimi Moonshot V1-128k** | **97.5** | **97.5** | multi-seed std = 0.0 across 5 seeds |
83
- | **Claude Sonnet 4.6** | **97.5** | **97.5** | numerically identical to Kimi on every task |
 
84
  | **GPT-5** | 93.2 | **75.7** | reasoning-oriented: 2 steps in 223 s vs Kimi's 7 steps in 8 s |
85
  | Llama 3.3 70B (Groq) | 89.3 | 18.7–97.5† | bimodal across seeds |
86
 
87
  † Llama T9 is **bimodal**: the published seed-42 run hit 18.7, multi-seed re-run produced {97.5, 94.5, …} — the low number and the near-frontier numbers both reproduce. Raw per-seed data in `multiseed_llama_t9_summary.json`.
88
 
89
- ### Three independent discriminative signals
 
 
90
 
91
  1. **T9 separates execution-oriented from reasoning-oriented frontier.** Kimi and Claude execute T9 in ~8 s across 7 tool calls and score 97.5. GPT-5 "thinks" for 223 s across 2 tool calls and scores 75.7 — a **21.8-point gap between frontier models** that a pass/fail benchmark would miss entirely. The breakdown tells the story: GPT-5's Efficiency drops to 6/15 (budget burned in reasoning-time) and Observability to ~4/10 (2 steps leave no audit trail).
92
- 2. **Frontier saturates at the top.** Kimi and Claude produce *numerically identical* per-task scores on all 10 tasks. Same seeded environment, same deterministic judge, same solve-path → same score. ComtradeBench today cannot fine-rank two execution-optimised frontier models against each other.
93
- 3. **Sub-frontier is high-variance, not uniformly weak.** Kimi T9 std = 0.0 across 5 seeds. Llama T9 spans 18.7 – 97.5. The discriminative signal is *reliability*, not capability: Llama can sometimes match frontier, just not consistently. Production agent deployment needs the consistent half.
 
94
 
95
  ### GRPO training — operating envelope empirically mapped at three points
96
 
97
- ```
98
- ⚠️ Qwen2.5-7B + LoRA
99
- saturates at init
100
- (reward_std ≈ 0 → no gradient)
101
- /
102
- 🟢 Qwen2.5-3B + LoRA
103
- learns 14 iter (kl 8e-6 → 5.6e-4)
104
- then policy-collapses at iter 15
105
- /
106
- 🟡 Qwen2.5-1.5B (50 iter full-param)
107
- noise-dominated, no net upward trend
108
- (reward 0.22–0.94, under-capacity on T9/T10)
109
- ```
110
 
111
- **Reading the triangle**: the useful GRPO training band exists (iters 3-14 of the 3B run are empirical proof — real reward variance, monotonically growing KL), but it is **narrow and fragile**. Three configurations, three distinct failure modes. Stable training on the 3B point requires adaptive KL penalty, tighter trust-region clipping, or early-stop on reward-variance collapse — engineering work we did not perform in this release. This is a more actionable finding than "training converged on some model": it names concrete failure modes a practitioner would hit.
112
 
113
  All training data is committed as artifacts: `grpo_gradient_training.jsonl` (1.5B per-iter metrics), `grpo_gradient_training_3b.jsonl` (3B per-iter, 15 entries), `grpo_3b_lora_collapse.json` (3B interpretation), `grpo_7b_lora_5iter_saturation.json` (7B interpretation). The same environment code runs in-process during GRPO rollouts and as the deployed Docker service during eval — zero divergence.
114
 
@@ -410,13 +405,14 @@ Kimi-128k matches or slightly exceeds the rule-based baseline on **all 10 tasks*
410
  gap on T4/T5 Robustness (12/15, not 15/15) is a scoring sub-criterion explored in the ablation
411
  below, not a silent-retry failure.
412
 
413
- ### Cross-model comparison — four LLMs, three independent discriminative signals
414
 
415
  | Model | Avg (T1-T10) | T1-T8 avg | T9 score | T10 score |
416
  |---|---:|---:|---:|---:|
417
  | Rule-based baseline | 96.8 | 96.5 | 96.9 | 98.0 |
418
  | **Kimi Moonshot V1-128k** | **97.5** | **97.4** | **97.5** (std 0.0 across 5 seeds) | **98.7** |
419
  | **Claude Sonnet 4.6** | **97.5** | **97.4** | **97.5** | **98.7** |
 
420
  | **GPT-5** | 93.2 | 95.0 | **75.7** | 95.7 |
421
  | Llama 3.3 70B (Groq) | 89.3 | 97.4 | 18.7 – 97.5 (bimodal†) | 95.7 |
422
 
 
38
 
39
  ### An OpenEnv Benchmark for Reliable LLM Tool-Use Under Adversarial API Conditions
40
 
41
+ What happens when **Kimi, Claude, GPT-5, Llama, and open-source Qwen2.5-7B** all run the same 10 API tasks against the same seeded environment and the same deterministic 6-dimensional judge?
42
 
43
  - Kimi and Claude produce **numerically identical** scores on every single task — not close, *identical* (97.5 avg each).
44
+ - **Open-source Qwen2.5-7B-Instruct, zero-shot (no training)**, scores **97.2 — within 0.3 of closed-source frontier**, above baseline, above GPT-5 and Llama 70B.
45
+ - GPT-5 scores **21.8 points lower** than frontier on one specific task — the one with mid-episode fault escalation. Not because it's less capable, but because it *reasons for 223 seconds across 2 tool calls* instead of *executing 7 tool calls in 8 seconds*.
46
  - Llama is *bimodal* on that same task, spanning 18.7 to 97.5 across seeds — the discriminative signal is **reliability**, not capability.
47
 
48
  And when we try to **GRPO-train** a Qwen2.5-3B on the benchmark? It enters the learning window at iter 3, trains cleanly for 14 iterations, then **policy-collapses at iter 15** into a degenerate output region.
 
56
 
57
  > **For judges — 30-second summary**:
58
  > - Ten-task OpenEnv benchmark for LLM agent reliability under adversarial API conditions (429/500, pagination drift, duplicates, totals traps, within-episode fault escalation, constrained budgets).
59
+ > - **Five LLMs evaluated cross-model**: Kimi Moonshot V1-128k, Claude Sonnet 4.6, **open-source Qwen2.5-7B-Instruct (zero-shot)**, GPT-5, Llama 3.3 70B. Plus three Qwen2.5 sizes trained with GRPO (1.5B full-param, 3B + LoRA, 7B + LoRA).
60
+ > - **Four independent findings**: (1) T9 separates execution-oriented from reasoning-oriented frontier (Kimi/Claude 97.5 vs GPT-5 75.7), (2) Kimi = Claude numerically identical → ceiling saturation, (3) Llama T9 bimodal → sub-frontier is about reliability not capability, (4) **⭐ Open-source Qwen2.5-7B zero-shot matches closed frontier (97.2 vs 97.5)** — benchmark solvable by well-instructed open 7B without any training.
61
  > - **GRPO operating envelope mapped at three points** (under-capacity / learn-then-collapse / saturation) — an actionable finding, not a "we trained something" claim.
62
  > - Results live and reproducible in the HF Docker Space.
63
 
 
80
  | Agent | Avg (T1-T10) | T9 | Notes |
81
  |---|---:|---:|---|
82
  | Rule-based baseline | 96.8 | 96.9 | deterministic, no LLM |
83
+ | **Kimi Moonshot V1-128k** | **97.5** | **97.5** | closed-source frontier, multi-seed std = 0.0 |
84
+ | **Claude Sonnet 4.6** | **97.5** | **97.5** | closed-source frontier, identical to Kimi |
85
+ | **Qwen2.5-7B-Instruct** ⭐ | **97.2** | **97.5** | **open-source, zero-shot** (no fine-tuning, no training) |
86
  | **GPT-5** | 93.2 | **75.7** | reasoning-oriented: 2 steps in 223 s vs Kimi's 7 steps in 8 s |
87
  | Llama 3.3 70B (Groq) | 89.3 | 18.7–97.5† | bimodal across seeds |
88
 
89
  † Llama T9 is **bimodal**: the published seed-42 run hit 18.7, multi-seed re-run produced {97.5, 94.5, …} — the low number and the near-frontier numbers both reproduce. Raw per-seed data in `multiseed_llama_t9_summary.json`.
90
 
91
+ **⭐ Open-source parity**: Qwen2.5-7B-Instruct, run *zero-shot* (no training, no fine-tuning), via Together AI → **97.2 / 100 avg, 97.5 on T9** — within 0.3 points of closed-source frontier (Kimi, Claude), above baseline, and **above GPT-5 by 4.0 points**. ComtradeBench is solvable by mid-size open-weights models at zero training cost. `llm_results_qwen7b_zeroshot.json`.
92
+
93
+ ### Four independent findings from the cross-model evaluation
94
 
95
  1. **T9 separates execution-oriented from reasoning-oriented frontier.** Kimi and Claude execute T9 in ~8 s across 7 tool calls and score 97.5. GPT-5 "thinks" for 223 s across 2 tool calls and scores 75.7 — a **21.8-point gap between frontier models** that a pass/fail benchmark would miss entirely. The breakdown tells the story: GPT-5's Efficiency drops to 6/15 (budget burned in reasoning-time) and Observability to ~4/10 (2 steps leave no audit trail).
96
+ 2. **Frontier saturates at the top.** Kimi and Claude produce *numerically identical* per-task scores on all 10 tasks. Same seeded environment, same deterministic judge, same solve-path → same score.
97
+ 3. **Sub-frontier is high-variance, not uniformly weak.** Kimi T9 std = 0.0 across 5 seeds. Llama T9 spans 18.7 – 97.5. The discriminative signal is *reliability*, not capability: Llama can sometimes match frontier, just not consistently.
98
+ 4. **⭐ Open-source 7B closes the gap to frontier without training.** Qwen2.5-7B-Instruct, zero-shot (no fine-tuning, no GRPO), scores 97.2 — within 0.3 points of Kimi/Claude, above GPT-5 and Llama 3.3 70B. This changes the headline: **the benchmark is not "can a closed frontier LLM solve this"; it's "can an agent execute reliably", and a well-instructed open 7B matches that bar**. It also validates the GRPO saturation finding — 7B genuinely is at ceiling for this benchmark, which is why GRPO fine-tuning provides no gradient signal.
99
 
100
  ### GRPO training — operating envelope empirically mapped at three points
101
 
102
+ ![GRPO Operating Envelope — three failure modes](grpo_envelope.png)
103
+
104
+ *Three training configurations, three distinct failure modes. 1.5B full-param: under-capacity, reward oscillates 0.22–0.94 with no trend. 3B + LoRA: learns cleanly for 14 iterations (KL grows monotonically 8e-6 → 5.6e-4), then policy-collapses at iter 15. 7B + LoRA: mean reward 0.987 at iter 1, already above baseline — GRPO advantage signal near zero, no gradient propagates.*
 
 
 
 
 
 
 
 
 
 
105
 
106
+ **Reading the envelope**: the useful GRPO training band exists (iters 3-14 of the 3B run are empirical proof — real reward variance, monotonically growing KL), but it is **narrow and fragile**. Stable training on the 3B point requires adaptive KL penalty, tighter trust-region clipping, or early-stop on reward-variance collapse — engineering work we did not perform in this release. This is a more actionable finding than "training converged on some model": it names concrete failure modes a practitioner would hit.
107
 
108
  All training data is committed as artifacts: `grpo_gradient_training.jsonl` (1.5B per-iter metrics), `grpo_gradient_training_3b.jsonl` (3B per-iter, 15 entries), `grpo_3b_lora_collapse.json` (3B interpretation), `grpo_7b_lora_5iter_saturation.json` (7B interpretation). The same environment code runs in-process during GRPO rollouts and as the deployed Docker service during eval — zero divergence.
109
 
 
405
  gap on T4/T5 Robustness (12/15, not 15/15) is a scoring sub-criterion explored in the ablation
406
  below, not a silent-retry failure.
407
 
408
+ ### Cross-model comparison — five LLMs, four independent findings
409
 
410
  | Model | Avg (T1-T10) | T1-T8 avg | T9 score | T10 score |
411
  |---|---:|---:|---:|---:|
412
  | Rule-based baseline | 96.8 | 96.5 | 96.9 | 98.0 |
413
  | **Kimi Moonshot V1-128k** | **97.5** | **97.4** | **97.5** (std 0.0 across 5 seeds) | **98.7** |
414
  | **Claude Sonnet 4.6** | **97.5** | **97.4** | **97.5** | **98.7** |
415
+ | **Qwen2.5-7B-Instruct (open, zero-shot)** ⭐ | **97.2** | **97.2** | **97.5** | **98.7** |
416
  | **GPT-5** | 93.2 | 95.0 | **75.7** | 95.7 |
417
  | Llama 3.3 70B (Groq) | 89.3 | 97.4 | 18.7 – 97.5 (bimodal†) | 95.7 |
418
 
benchmark_results.png CHANGED

Git LFS Details

  • SHA256: 2691fdb58076a168516027eb05fd85ac282132d94a083491de422c348bc83d5f
  • Pointer size: 131 Bytes
  • Size of remote file: 155 kB

Git LFS Details

  • SHA256: c3b7c958c69396eb3f22e1c03265b08d7ad4f3db5f7f3bc7c9a772a9304e1aae
  • Pointer size: 131 Bytes
  • Size of remote file: 167 kB
grpo_envelope.png ADDED

Git LFS Details

  • SHA256: 819668de75b7be83e8e0ce9cc226ae60f9a72622913ac6b9f4e42b14b6a057d0
  • Pointer size: 131 Bytes
  • Size of remote file: 222 kB
llm_results_qwen7b_zeroshot.json ADDED
@@ -0,0 +1,185 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "Qwen/Qwen2.5-7B-Instruct-Turbo",
3
+ "api_url": "https://api.together.xyz/v1",
4
+ "provider": "Together AI",
5
+ "note": "Open-source Qwen2.5-7B-Instruct evaluated ZERO-SHOT (no fine-tuning, no GRPO training). Same agent loop and default prompt as Kimi/Claude/GPT-5/Llama runs. seed=42, temperature=0.0. Run 2026-04-20 via Together AI API.",
6
+ "seed": 42,
7
+ "tasks_run": [
8
+ "T1_single_page",
9
+ "T2_multi_page",
10
+ "T3_duplicates",
11
+ "T4_rate_limit_429",
12
+ "T5_server_error_500",
13
+ "T6_page_drift",
14
+ "T7_totals_trap",
15
+ "T8_mixed_faults",
16
+ "T9_adaptive_adversary",
17
+ "T10_constrained_budget"
18
+ ],
19
+ "average_score": 97.16,
20
+ "average_reward": 0.9716,
21
+ "results": [
22
+ {
23
+ "task_id": "T1_single_page",
24
+ "reward": 0.987,
25
+ "score": 98.7,
26
+ "breakdown": {
27
+ "correctness": 30.0,
28
+ "completeness": 15.0,
29
+ "robustness": 15.0,
30
+ "efficiency": 15.0,
31
+ "data_quality": 15.0,
32
+ "observability": 8.67
33
+ },
34
+ "error": null,
35
+ "steps": 3,
36
+ "elapsed_s": 2.1
37
+ },
38
+ {
39
+ "task_id": "T2_multi_page",
40
+ "reward": 0.987,
41
+ "score": 98.7,
42
+ "breakdown": {
43
+ "correctness": 30.0,
44
+ "completeness": 15.0,
45
+ "robustness": 15.0,
46
+ "efficiency": 15.0,
47
+ "data_quality": 15.0,
48
+ "observability": 8.67
49
+ },
50
+ "error": null,
51
+ "steps": 7,
52
+ "elapsed_s": 4.0
53
+ },
54
+ {
55
+ "task_id": "T3_duplicates",
56
+ "reward": 0.987,
57
+ "score": 98.7,
58
+ "breakdown": {
59
+ "correctness": 30.0,
60
+ "completeness": 15.0,
61
+ "robustness": 15.0,
62
+ "efficiency": 15.0,
63
+ "data_quality": 15.0,
64
+ "observability": 8.67
65
+ },
66
+ "error": null,
67
+ "steps": 5,
68
+ "elapsed_s": 2.4
69
+ },
70
+ {
71
+ "task_id": "T4_rate_limit_429",
72
+ "reward": 0.957,
73
+ "score": 95.7,
74
+ "breakdown": {
75
+ "correctness": 30.0,
76
+ "completeness": 15.0,
77
+ "robustness": 12.0,
78
+ "efficiency": 15.0,
79
+ "data_quality": 15.0,
80
+ "observability": 8.67
81
+ },
82
+ "error": null,
83
+ "steps": 5,
84
+ "elapsed_s": 4.0
85
+ },
86
+ {
87
+ "task_id": "T5_server_error_500",
88
+ "reward": 0.963,
89
+ "score": 96.3,
90
+ "breakdown": {
91
+ "correctness": 30.0,
92
+ "completeness": 15.0,
93
+ "robustness": 12.0,
94
+ "efficiency": 15.0,
95
+ "data_quality": 15.0,
96
+ "observability": 9.34
97
+ },
98
+ "error": null,
99
+ "steps": 5,
100
+ "elapsed_s": 4.1
101
+ },
102
+ {
103
+ "task_id": "T6_page_drift",
104
+ "reward": 0.947,
105
+ "score": 94.7,
106
+ "breakdown": {
107
+ "correctness": 26.0,
108
+ "completeness": 15.0,
109
+ "robustness": 15.0,
110
+ "efficiency": 15.0,
111
+ "data_quality": 15.0,
112
+ "observability": 8.67
113
+ },
114
+ "error": null,
115
+ "steps": 5,
116
+ "elapsed_s": 2.5
117
+ },
118
+ {
119
+ "task_id": "T7_totals_trap",
120
+ "reward": 0.987,
121
+ "score": 98.7,
122
+ "breakdown": {
123
+ "correctness": 30.0,
124
+ "completeness": 15.0,
125
+ "robustness": 15.0,
126
+ "efficiency": 15.0,
127
+ "data_quality": 15.0,
128
+ "observability": 8.67
129
+ },
130
+ "error": null,
131
+ "steps": 5,
132
+ "elapsed_s": 18.5
133
+ },
134
+ {
135
+ "task_id": "T8_mixed_faults",
136
+ "reward": 0.939,
137
+ "score": 93.9,
138
+ "breakdown": {
139
+ "correctness": 25.2,
140
+ "completeness": 15.0,
141
+ "robustness": 15.0,
142
+ "efficiency": 15.0,
143
+ "data_quality": 15.0,
144
+ "observability": 8.67
145
+ },
146
+ "error": null,
147
+ "steps": 3,
148
+ "elapsed_s": 39.5
149
+ },
150
+ {
151
+ "task_id": "T9_adaptive_adversary",
152
+ "reward": 0.975,
153
+ "score": 97.5,
154
+ "breakdown": {
155
+ "correctness": 28.88,
156
+ "completeness": 15.0,
157
+ "robustness": 15.0,
158
+ "efficiency": 15.0,
159
+ "data_quality": 15.0,
160
+ "observability": 8.67
161
+ },
162
+ "error": null,
163
+ "steps": 7,
164
+ "elapsed_s": 4.4
165
+ },
166
+ {
167
+ "task_id": "T10_constrained_budget",
168
+ "reward": 0.987,
169
+ "score": 98.7,
170
+ "breakdown": {
171
+ "correctness": 30.0,
172
+ "completeness": 15.0,
173
+ "robustness": 15.0,
174
+ "efficiency": 15.0,
175
+ "data_quality": 15.0,
176
+ "observability": 8.67
177
+ },
178
+ "error": null,
179
+ "steps": 10,
180
+ "elapsed_s": 6.0
181
+ }
182
+ ],
183
+ "key_finding": "Open-source Qwen2.5-7B-Instruct, zero-shot (no training), scores 97.2 / 100 on ComtradeBench — within 0.3 points of closed-source frontier (Kimi 97.5, Claude 97.5), 4.0 points above GPT-5 (93.2), and 7.9 points above Llama 3.3 70B (89.3). On T9 (the execution-vs-reasoning discriminator) Qwen2.5-7B matches frontier at 97.5. This validates: (a) ComtradeBench is solvable by mid-size open-weights models, (b) the benchmark measures execution reliability at a level where a good open 7B equals a closed frontier model, (c) our GRPO saturation finding at 7B is NOT an artifact — the 7B model genuinely is at ceiling for this benchmark.",
184
+ "generated_at": "2026-04-20T13:14"
185
+ }