ssurface/qwen3-4b-grpo-l2

Qwen3-4B GRPO — Level 2 (Concise)

Qwen3-4B-Instruct fine-tuned with SFT → GRPO for compressed chain-of-thought reasoning at Level 2 (Concise).

Training Pipeline

Qwen/Qwen3-4B-Instruct-2507
  → SFT LoRA (ssurface/qwen3-4b-cot-compress-l2)
  → Merged
  → GRPO with DAPO loss
  → Merged (this model)

Reward Functions

Two reward signals were used during GRPO:

1. Correctness Reward

Checks if the model's final answer (after ####) matches the ground truth. Reward is scaled by problem complexity (1–5 steps):

Complexity	Reward
5 (hardest)	5.0
4	4.0
3	3.0
2	2.0
1 (easiest)	1.0
Wrong answer	0.0

2. Goldilocks Length Penalty (double-hinge)

Penalizes <think> blocks that are too short OR too long for the target level.

Level	Style	Target range (chars)	Too-short penalty	Too-long penalty
1	Verbose	120–240	−0.05/char	−0.01/char
2	Concise	50–110	−0.05/char	−0.01/char
3	Symbolic	40–100	−0.05/char	−0.01/char
4	Shorthand	20–60	−0.05/char	−0.01/char
5	Extreme	10–30	−0.05/char	−0.01/char

This model targets Level 2 (Concise): 50-110 chars of reasoning.

GRPO Hyperparameters

Parameter	Value
Loss type	DAPO
LoRA r	16
LoRA alpha	32
Num generations	8
Max completion length	256
Batch size	16
Gradient accumulation	2
Learning rate	1e-5
Epochs	1
Dataset	GSM8K (train split, level-filtered)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ssurface/qwen3-4b-grpo-l2")
tokenizer = AutoTokenizer.from_pretrained("ssurface/qwen3-4b-grpo-l2")

messages = [{"role": "user", "content": "Solve this using Level 2 (Concise).\nProblem: ..."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))