drlee1/HanForge-47M-SFT

HanForge 47M SFT: Korean Conversational Model

A Korean chat model fine-tuned from drlee1/HanForge-base on 24,693 Korean question-answer pairs prepared by knowledge distillation across five everyday domains.

Updated 2026-10-06. The previous release (2026-05-08) did not start from the base weights and ran with a broken rotary position encoding. This release is retrained from the base model with both issues fixed. See Changelog.

Highlights

• Retrained from the pretrained base: validation perplexity 9.78 on 503 held-out pairs (previous release: 37.83 on the same set)
• Ends its answers: 95% of greedy and 100% of sampled answers finish within 128 tokens on 20 test prompts
• No repetition collapse: the longest run of one repeated token is 2 (previous release: 198 under greedy decoding)
• Pure Korean output: 100% Hangul ratio on the test prompts
• Compact: 47M parameters (35M base plus a separate output layer)

Intended Use

Suitable for:

• Korean chat applications within everyday-conversation domains
• Resource-constrained deployments needing a small Korean model
• Research into small-LM knowledge distillation and instruction tuning

Not suitable for:

• Factual question answering (the model is small and the distilled data is not fact-checked)
• Multi-step reasoning, coding, or technical tasks
• Open-domain conversation outside the five training domains
• Any safety-critical application

How to Use

Tested with transformers 5.5.1 and torch 2.11.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "drlee1/HanForge-47M-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

USER, ASSISTANT = "<|user|>", "<|assistant|>"

def chat(prompt: str, max_new_tokens: int = 200, seed: int = 42, sample: bool = True) -> str:
    torch.manual_seed(seed)
    text = f"{USER}\n{prompt}\n{ASSISTANT}\n"
    inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)

    # Add BOS manually
    bos = inputs["input_ids"].new_full((1, 1), tokenizer.bos_token_id)
    inputs["input_ids"] = torch.cat([bos, inputs["input_ids"]], dim=1)
    inputs["attention_mask"] = torch.cat(
        [inputs["attention_mask"].new_ones((1, 1)), inputs["attention_mask"]], dim=1
    )

    kwargs = dict(do_sample=True, temperature=0.8, top_p=0.9) if sample else dict(do_sample=False)
    out = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        **kwargs,
    )
    return tokenizer.decode(out[0, inputs["input_ids"].size(1):], skip_special_tokens=True).strip()

print(chat("김치찌개 어떻게 만들어요?"))

Decoding tips

• Greedy and sampled decoding both work. Sampling (temperature=0.8, top_p=0.9) gives more varied answers; greedy gives the most stable one.
• Cap output length. 150 to 200 new tokens is enough; most answers end well before that.

Training Data

24,693 Korean question-answer pairs prepared by knowledge distillation. The dataset spans 200 (domain, topic) pairs across five everyday domains, each contributing about 100 user-style questions with concise polite Korean answers.

Domain	Topics covered
Daily greetings & conversation	greetings, thanks, apologies, introductions, mood, comfort, requests
Food & cooking basics	Korean dishes, ingredients, simple recipes, recommendations
Korean culture & geography	cities, mountains, traditional clothing, holidays, traditions
Health & lifestyle habits	exercise, sleep, nutrition, stress, daily routines
Emotions & empathy	sadness, loneliness, anxiety, joy, gratitude, comfort

After filtering for polite endings and language purity (about 8.5% dropped), the training set is 100% Hangul with an average answer length of about 134 characters. A held-out split of 503 pairs is used for validation.

Training Procedure

Full-parameter SFT from drlee1/HanForge-base with response-only loss masking. The output layer starts as a copy of the base model's embedding matrix and is trained separately.

Training samples	24,693
Epochs	5 (7,720 optimizer steps)
Effective batch size	16 (4 × 4 gradient accumulation)
Learning rate	5e-5, cosine, 5% warmup
Sequence length	384
Precision	bf16 mixed
Validation loss / perplexity	2.28 / 9.79 (503 held-out pairs)
Wall-clock time	8.6 minutes on one RTX 5080 16GB

Evaluation

All numbers below were measured on 2026-10-06 with the fixed modeling code. The previous release is shown on the same setup for comparison.

Validation set (503 held-out distillation pairs)

	Base (no SFT)	Previous release	This release
Perplexity	62.26	37.83	9.78

20 test prompts (14 in-distribution, 6 out-of-distribution), max 200 new tokens

Metric	Previous release (greedy / sampled)	This release (greedy / sampled)
Answer ends within 64 tokens	65% / 95%	75% / 95%
Answer ends within 128 tokens	65% / 100%	95% / 100%
Longest run of one repeated token	198 / 60	2 / 2
Distinct first tokens (of 20)	9 / 14	19 / 19
Hangul ratio	100% / 100%	100% / 100%

Sampling uses temperature=0.8, top_p=0.9, seed 42.

Grammar minimal pairs (51 pairs, length-normalized log-likelihood of the correct sentence must be higher)

Base	Previous release	This release
98.0% (50/51)	56.9% (29/51)	90.2% (46/51)

The model answers in-domain questions on topic and in fluent Korean. Facts in its answers are often wrong, and out-of-domain questions get fluent but loosely related answers.

Limitations and Bias

• Small scale (47M): limited factual accuracy and reasoning. Do not rely on it for information.
• Distilled-data origin: answers were produced by a larger model and not fact-checked; its biases are inherited.
• Domain restriction: the five training domains define the reliable scope.
• No alignment or safety tuning: not RLHF'd and no harmful-content filtering.

Changelog

2026-10-06: retrained release

1. Base weights were not loaded during SFT. The training loader only restored weights from pytorch_model.bin, while the base checkpoint was saved as model.safetensors. The previous release was therefore fine-tuned from random initialization (its weights have near-zero cosine similarity with the base). The loader now reads model.safetensors, and this release starts from the base weights.
2. Rotary position encoding was not initialized under transformers 5. from_pretrained left the non-persistent inv_freq buffer uninitialized (zeros or arbitrary values), so position information was missing or random. modeling_hanforge.py now computes the rotary frequencies from the config on every forward pass.
3. Retrained on one RTX 5080 with the same data and hyperparameters as the previous release, and re-evaluated.

2026-05-08: initial release (Mac MPS, 19 minutes).

License

Released under the Apache License 2.0.

Citation

@misc{hanforge_47m_sft_2026,
  author = {DongRyeol Lee},
  title  = {HanForge 47M SFT: A Korean Conversational Model Trained via Knowledge Distillation},
  year   = {2026},
  note   = {Fine-tuned from drlee1/HanForge-base on 24.7k Korean Q\&A pairs across five everyday domains}
}