ArnavKewalram commited on
Commit
4631872
·
verified ·
1 Parent(s): 48dfb73

Add HumanEval pass@1 benchmark: 34.1% (matches Code Llama 7B)

Browse files
Files changed (1) hide show
  1. README.md +24 -2
README.md CHANGED
@@ -41,7 +41,13 @@ model-index:
41
  - task:
42
  type: text-generation
43
  name: Code Generation
 
 
 
44
  metrics:
 
 
 
45
  - type: keyword_score
46
  value: 88.5
47
  name: Keyword Score (8 coding tasks, Q4_K_M)
@@ -80,7 +86,7 @@ widget:
80
 
81
  # gemma-4-E2B-coder-v1
82
 
83
- **The first coding fine-tune of [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it)** — **88.5% keyword score** across 8 languages, runs fully offline on **4 GB RAM**, Apache 2.0.
84
 
85
  | | |
86
  |--|--|
@@ -245,6 +251,22 @@ Trained with [TRL SFTTrainer](https://huggingface.co/docs/trl) + [PEFT LoRA](htt
245
 
246
  ## Evaluation
247
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
248
  Keyword-based evaluation on 8 coding prompts using Q4_K_M GGUF (CPU inference, llama.cpp b9684, temperature 0.2):
249
 
250
  | Prompt | Keywords checked | Score |
@@ -260,7 +282,7 @@ Keyword-based evaluation on 8 coding prompts using Q4_K_M GGUF (CPU inference, l
260
 
261
  **Average keyword score: 88.5%** (8 prompts).
262
 
263
- Keyword scoring checks that expected API/structural elements appear in the output — it is a proxy for code correctness, not a formal benchmark like HumanEval. The Miller-Rabin score (33%) is low because the model wrote a functionally correct implementation using variable names `a` and `x` rather than the keyword-matched names `miller`/`witness`. The retry decorator output was truncated at 512 tokens but is structurally correct.
264
 
265
  ---
266
 
 
41
  - task:
42
  type: text-generation
43
  name: Code Generation
44
+ dataset:
45
+ name: openai_humaneval
46
+ type: openai/openai_humaneval
47
  metrics:
48
+ - type: pass@1
49
+ value: 34.1
50
+ name: HumanEval pass@1 (Q4_K_M, temp=0.2, raw completion)
51
  - type: keyword_score
52
  value: 88.5
53
  name: Keyword Score (8 coding tasks, Q4_K_M)
 
86
 
87
  # gemma-4-E2B-coder-v1
88
 
89
+ **The first coding fine-tune of [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it)** — **34.1% HumanEval pass@1** · matches Code Llama 7B at half the size · runs fully offline on **4 GB RAM** · Apache 2.0.
90
 
91
  | | |
92
  |--|--|
 
251
 
252
  ## Evaluation
253
 
254
+ ### HumanEval (pass@1)
255
+
256
+ Evaluated on the full [OpenAI HumanEval](https://github.com/openai/human-eval) benchmark (164 Python problems) using Q4_K_M GGUF via Ollama, raw completion mode (no chat template), temperature 0.2, 512 max tokens.
257
+
258
+ | Model | Size | HumanEval pass@1 |
259
+ |-------|------|-----------------|
260
+ | **gemma-4-E2B-coder-v1 (Q4_K_M)** | **3.9B** | **34.1%** |
261
+ | Code Llama | 7B | 33.5% |
262
+ | Qwen2.5-Coder | 1.5B | 37.2% |
263
+ | Llama 3.2 | 3B | 25.4% |
264
+ | Gemma 2 | 2B | 18.7% |
265
+
266
+ **34.1% pass@1** — competitive with Code Llama 7B at roughly half the parameter count. Notable given the model was fine-tuned on only 10,000 samples with a 384-token context window; longer-context problems are the primary failure mode.
267
+
268
+ ### Keyword Score
269
+
270
  Keyword-based evaluation on 8 coding prompts using Q4_K_M GGUF (CPU inference, llama.cpp b9684, temperature 0.2):
271
 
272
  | Prompt | Keywords checked | Score |
 
282
 
283
  **Average keyword score: 88.5%** (8 prompts).
284
 
285
+ Keyword scoring checks that expected API/structural elements appear in the output — it is a proxy for code correctness, not a formal benchmark. The Miller-Rabin score (33%) is low because the model wrote a functionally correct implementation using variable names `a` and `x` rather than the keyword-matched names `miller`/`witness`.
286
 
287
  ---
288