Qwen3.5-0.8B-TW-IME
Qwen/Qwen3.5-0.8B ไปฅๅฐ็ฃๆญฃ้ซไธญๆ่ชๆๅ LoRA ๆ็บ้ ่จ็ทดๅพๅไฝต็็ๆฌ๏ผ็จ้ๆฏๆฟๆณจ้ณ่ผธๅ ฅๆณ็ๅ้ธๅฅๆๅๆธ๏ผ่จ็ฎ log P(ๅ้ธ๏ฝไธๆ)๏ผ๏ผไธๆฏ่ๅคฉๆจกๅใ
ๅ ่ฎ้ๅนพ้ป
- ๅชๅจใๆณจ้ณ้ธๅญใ้ไธๅไปปๅไธ่ฉๆธฌ้๏ผ่ฆใ่ฉๆธฌใ๏ผใๆฒๆ่ฉๆธฌๅฐ่ฉฑใๆไปค้ตๅพชใๆจ็ใ็จๅผๆ่ฆ่ฆบ่ฝๅ๏ผ่จ็ทดๅช็จ็ดๆๅญ๏ผ้ไบ่ฝๅๅฏ่ฝๆฏๅ็ๅทฎใ
- ่จ็ทด่ณๆๅ จ้จๅฏไปฅ่ช็ฑๅๅฉ็จ๏ผCC0ใๆฟๅบ่ณๆ้ๆพๆๆฌๆขๆฌพ๏ผ่ CC BY 4.0 ็ธๅฎน๏ผใไธๅ่ไฝๆฌไฟ่ญท็ๆณ่ฆ๏ผไปฅๅ็ฑ Apache-2.0 ๆจกๅๅจๆฌๆฉ็ๆ็ๅฅๅญใๆฒๆไฝฟ็จ็ถญๅบ็พ็ง่ๆ่ฒ้จ่พญๅ ธใไพๆบใๆๆฌ่้ๆนๅผ่ฆ
DATA.md่sources.jsonใ- ็ธๅฐๅ็๏ผCommon Voice +1.8๏ฝ1.9ใ็ซๆณ้ข +2.5ใๆฐ่็จฟ +1.2ใๅฐ็ฃ็จๅญ +4.2 ๅ็พๅ้ป๏ผ็ถญๅบ โ0.3๏ผๅฎๆฌก่จ็ทดใ็กไฟก่ณดๅ้๏ผใๆด้ซ็ด็ญๆผๆชๅพฎ่ชฟ็ 2B ๅ็ใ
- ๆฏๅไธๅฐๆก็ Qwen3.5-2B-TW-IME ไฝ็ด 1 ๅ็พๅ้ป๏ผๅจ้กฏ็คบๅกไธๅปถ้ฒๅทฎไธๅค๏ผๅจ CPU ไธ็ดๅฟซไธๅ๏ผๆฏๆฌก้ธๅญ็ด 0.5 ็งๅฐ 0.95 ็ง๏ผใ
- ๆฒๆ้
mtp.*๏ผๅค token ้ ๆธฌ๏ผๅผต้ใ
็จ้
LLaIME๏ผWindows 11 ๆณจ้ณ่ผธๅ ฅๆณ๏ผๆๆๅบ็ๆณจ้ณ็ถ่ฉๅบซ่ฝๆๅ 16 ๅๅ้ธๅฅ๏ผๅ็ฑๆฌๆฉ่ช่จๆจกๅ็ฎๆฏไธๅฅ็ log P๏ผๆๆ้้ ็ใๅ้ธไธๅฎ็ฌฆๅๆๆ็่ฎ้ณ๏ผๆจกๅๅช่ฒ ่ฒฌๆๅบ๏ผไธๆ่ช็ฑ็ๆใ่จ็ทดๆ ผๅผ่่ผธๅ ฅๆณ้็ตฆๆจกๅ็ๅฎๅ จ็ธๅ๏ผ
ไปฅไธๆฏไธๆฎตๅฐ็ฃ็น้ซไธญๆ็ๆๅญ๏ผ\n{ไธๆ}{ๅ้ธๅฅ}
่จ็ทดๆๆๅคฑๅช็ฎๆญฃๆ๏ผๆ็คบๅฅไธ็ฎ๏ผ๏ผๆไปฅๆจกๅๅญธๅฐ็ๆญฃๆฏ่ผธๅ ฅๆณ่ฆ็จ็้ใ
ไฝฟ็จๆนๅผ
่จ็ฎ log P(ๆๅญ๏ฝๆ็คบ๏ผไธๆ)๏ผๅฎไฝ nat๏ผ่ถๅคง่ถ้้ ๏ผ๏ผ
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "alpha34727/Qwen3.5-0.8B-TW-IME"
PROMPT = "ไปฅไธๆฏไธๆฎตๅฐ็ฃ็น้ซไธญๆ็ๆๅญ๏ผ\n"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).cuda().eval()
@torch.no_grad()
def log_prob(context: str, text: str) -> float:
prefix = tok(PROMPT + context, add_special_tokens=False)["input_ids"]
target = tok(text, add_special_tokens=False)["input_ids"]
ids = torch.tensor([prefix + target]).cuda()
logits = model(input_ids=ids).logits[0, len(prefix) - 1 : -1].float()
logp = torch.log_softmax(logits, dim=-1)
return logp[torch.arange(len(target)), torch.tensor(target)].sum().item()
print(log_prob("่ซๅบ็คบๆจ็", "่บซๅ่ญ"), log_prob("่ซๅบ็คบๆจ็", "่บซไปฝ่ญ"))
ๆชๆก
| ๆชๆก | ่ชชๆ |
|---|---|
model.safetensors ่่จญๅฎๆช |
ๅไฝตๅพ็ bf16 ๆฌ้๏ผๆถๆง Qwen3_5ForConditionalGeneration๏ผ่ฆ่ฆบ้จๅ่ๅ็็ธๅ |
Qwen3.5-0.8B-TW-IME-Q4_K_M.gguf |
llama.cpp ็จ็ Q4_K_M ้ๅ๏ผ่ผธๅ ฅๆณๅฏฆ้ไฝฟ็จ็็ๆฌ |
DATA.mdใsources.json |
ๆฏๅ่จ็ทดไพๆบ็ๅบ่ใๆๆฌใไธ่ผๆฅๆ่ SHA-256 |
training/ |
่ณๆไธ่ผใๆธ
็ใ็ๆ่่จ็ทด่
ณๆฌ๏ผ็ๆ็ๅฅๅญ๏ผsynthetic_sentences.txt๏ผ่่จ็ทด่จ้ |
GGUF ไปฅ llama.cpp ็ convert_hf_to_gguf.py --outtype bf16 ่ฝๆช๏ผๅ็จ llama-quantize โฆ Q4_K_M ้ๅใ
่จ็ทด
ๅบๅบ๏ผQwen/Qwen3.5-0.8B๏ผๅพ่จ็ทด็๏ผใ่ชๆใ่็่่จ็ทด่จญๅฎ่ Qwen3.5-2B-TW-IME ๅฎๅ จ็ธๅ๏ผๅชๆๅบๅบๆจกๅใ
่ชๆ๏ผtraining/build_lm_corpus.py๏ผๅ
ฑ 188,098 ๆฎต๏ผ็ด 710 ่ฌๅญ๏ผ๏ผ500 ๆฎต็ไฝ้ฉ่ญ๏ผ187,598 ๆฎต๏ผ4,688,097 token๏ผ็จๆผ่จ็ทด๏ผ
| ไพๆบ | ๅ งๅฎน | ๆๆฌ |
|---|---|---|
| Common Voice zh-TW ๅฅๅญ | 3 ๅๅฅๅญๆช | CC0 |
| ็ซๆณ้ขๅ ฌๅ ฑ | 400 ๅ ด้ขๆ่ๅงๅกๆ็ด้็็ผ่จ๏ผๅปๆ็ผ่จไบบ่ๆฌ่่จป่จ๏ผ๏ผ13.6 ่ฌๅฅ | ๆฟๅบ่ณๆ้ๆพๆๆฌๆขๆฌพ๏ผ็ฌฌ1็ |
| ่กๆฟ้ขๅณๆๆฐ่ใ้ฒๅๅๅ่ฏๅๆๅไธญๅฟ็ฆ้ปๆฐ่ | ๅ 500 ๅ๏ผ1.2 ่ฌๅฅ | ๆฟๅบ่ณๆ้ๆพๆๆฌๆขๆฌพ๏ผ็ฌฌ1็ |
| ๅ จๅๆณ่ฆ่ณๆๅบซ | ็พ่กๆณ่ฆๆขๆ๏ผๆฝๆจฃ 5,000 ๅฅ | ไธๅ่ไฝๆฌไฟ่ญท๏ผ่ไฝๆฌๆณ็ฌฌ 9 ๆข๏ผ |
| ๆฌๆฉ็ๆ็ๆฅๅธธๅฃ่ชๅฅ | Breeze-7B-Instruct-v1_0๏ผApache-2.0๏ผไพใไธป้ก ร ๆ ๅขใ็ๆ 3 ่ฌๅฅ๏ผๆฟพๆ็ฐก้ซๅญใๅคง้ธ็จ่ฉใ้่คๅฅ่่ฉๆธฌๅฅ | ้จๆฌๆจกๅๅ ฌ้ |
่ฉๆธฌๅฅ่ๅ ถๆๅจ็ๆดๅฅๅ จ้จๆ้ค๏ผๆผขๅญๆฏไพไฝๆผ 60% ็ๆๅญไธๆฃ๏ผๆจฃๆฟๅฅๆฏ็จฎๆๅค 2 ๅฅ๏ผๆด็๏ผseed 20261015๏ผใ
ๆนๆณ๏ผtraining/lora_qwen.py๏ผ๏ผ
| ้ ็ฎ | ่จญๅฎ |
|---|---|
| LoRA | r = 16ใalpha = 32ใdropout = 0.05๏ผ่ช่จๆจกๅ็ๆๆ็ทๆงๅฑค๏ผๆณจๆๅใ็ทๆงๆณจๆๅใMLP๏ผ๏ผๅฏ่จ็ทดๅๆธ 10,822,656๏ผ1.25%๏ผใ่ฆ่ฆบๅก่ LM head ไธๅ |
| ่ผธๅ ฅ | ๆ็คบๅฅ๏ผๆญฃๆ๏ผๆๅค 256 token๏ผๆๅคฑๅช็ฎๆญฃๆ |
| ๆไฝณๅ | AdamWใlr 1e-4ใ5% ๆๆฉๅพ้คๅผฆ่กฐๆธใๆขฏๅบฆ่ฃๅช 1.0 |
| ๆนๆฌก | ็ญๆๆนๆฌก 16ใ1 ่ผชๅ ฑ 11,725 ๆญฅ |
| ็กฌ้ซ่ๆ้ | ไธๅผต RTX 5060 Ti 16 GB๏ผ็ด 3 ๅฐๆ๏ผ้กฏๅญๅณฐๅผ 7.6 GB |
| ็จฎๅญ | 20261015๏ผๅช่จ็ทดไธๆฌก |
้ฉ่ญๆๅคฑ๏ผ500 ๆฎต็ๅบ็่ชๆ๏ผๆฏ token ็ nat๏ผ๏ผ5.063 โ 3.869ใ
่ฉๆธฌ
่ผธๅ ฅๆณ็็ๅฏฆๆต็จ๏ผๆณจ้ณ โ ่ฉๅบซ๏ผMcBopomofo๏ผๆฐ้ ท้ณ๏ผๅ 16 ๅๅ้ธๆดๅฅ โ ่ช่จๆจกๅ้ๆใ
ๆททๅๅๆธ ๏ผ LM ๅๆธ ๏ผ 0.75 ร ่ฉๅบซๅๆธ โ 4 ร ้ๅฐ็ฃๆจๆบๅฏซๆณ็ๅญ่ฉๆธ
ๆดๅฅๆญฃ็ขบ็๏ผๅ ๅซ่ฉๅบซๅ 16 ๅๆฒๆๆญฃ่งฃ็ๅฅๅญ๏ผไธๅฎ็ฎ้ฏ๏ผใๅ จ้จ็จ Q4_K_Mใๅไธๅฅ่ฝๆชๆต็จใ
| ่ฉๆธฌ้ | ๅ งๅฎน |
|---|---|
| Common Voice dev๏ผ1,000๏ผ๏ผtest๏ผ2,000๏ผ | Common Voice zh-TW ๅฅๅญๅๆ 4โ24 ๅญ็็ๆฎต๏ผๆททๅๆฌ้ๅจ dev ไธ้ธๅฎ |
| ็ซๆณ้ข๏ผ1,000๏ผ | ๅพ็ซๆณ้ข็ผ่จ็ๅบ็็ๆฎต๏ผไธ้ฒ่จ็ทด |
| ๆฐ่็จฟ๏ผ1,000๏ผ | ๅพ่กๆฟ้ขๆฐ่็จฟ็ๅบ็็ๆฎต๏ผไธ้ฒ่จ็ทด |
| ็ถญๅบ๏ผ2,000๏ผ | ็ถญๅบ็พ็งๅฅๅญ๏ผๅช็จไพๅ่็ๆฏ่ผ๏ผไธ้จๆจกๅๅ ฌ้ |
| ๅฐ็ฃ็จๅญ๏ผ24๏ผ | ๅ้ณไฝๅ ฉๅฒธๅฏซๆณไธๅ็ๅฅๅญ๏ผ็จๅญ่กจๆฏ็้้ฏ่ชคๅพๆๅฏซ็๏ผๅช็ถๅ่ |
| ้ธๅญ | CV dev | CV test | ็ซๆณ้ข | ๆฐ่็จฟ | ็ถญๅบ | ๅฐ็ฃ็จๅญ |
|---|---|---|---|---|---|---|
| ๅช็จ่ฉๅบซ | 71.4% | 71.2% | 69.3% | 56.7% | 52.4% | 83.3% |
| ่ฉๅบซๅ 16 ๅๅซๆญฃ่งฃ๏ผไธ้๏ผ | 93.8% | 92.9% | 92.7% | 82.0% | 77.0% | 95.8% |
| Qwen3.5-0.8B ๅ็๏ผๅๆต็จ่ฝๆชใ้ๅ๏ผ | 85.2% | 83.2% | 85.3% | 76.8% | 72.0% | 87.5% |
| ๆฌๆจกๅ | 87.1% | 85.0% | 87.8% | 78.0% | 71.7% | 91.7% |
| ใๅฐ็ งใQwen3.5-2B ๅ็ | 87.5% | 85.5% | 86.8% | 77.7% | 73.7% | 91.7% |
| ใๅฐ็ งใQwen3.5-2B-TW-IME | 87.7% | 86.1% | 89.3% | 78.5% | 73.0% | 95.8% |
ๅช็่ช่จๆจกๅ๏ผ่ฉๅบซๆฌ้ 0๏ผไปๆๅฐ็ฃ็จๅญๆฃๅ๏ผ๏ผๅ็ โ ๆฌๆจกๅ dev 82.9 โ 86.2%ใtest 82.4 โ 84.0%ใ็ซๆณ้ข 84.4 โ 87.6%ใๆฐ่็จฟ 75.9 โ 78.4%ใ็ถญๅบ 71.8 โ 71.3%ใๅฐ็ฃ็จๅญ 91.7 โ 91.7%ใ
ๅปถ้ฒ๏ผๆฏๆฌก้ธๅญใ16 ๅๅ้ธ็ไธญไฝๆธ๏ผ๏ผRTX 5060 Ti ไธ็ด 48 ms๏ผCommon Voice ็ๆฎต๏ผๅฐ 75 ms๏ผๆฐ่็จฟ๏ผ๏ผi5-12400 CPU ไธ็ด 490 ms๏ผCommon Voice dev ๅ 150 ๅฅ๏ผ2B ็บ 950 ms๏ผใ
้ๅถ
- ๅช่จ็ทดไธๆฌกใๆฒๆ้่ค็จฎๅญ๏ผๆฒๆไฟก่ณดๅ้๏ผ1,000 ๅฅ็้ๅ 1 ้ป็ด็ญๆผ 10 ๅฅใ
- ็ซๆณ้ข่ๆฐ่็จฟๆธฌ่ฉฆ้ๅ่จ็ทด่ชๆๅไพๆบ๏ผไธๅๅฅๅญ๏ผ๏ผ้ๅ ฉ็ต็้ฒๆญฅๅฏ่ฝ้ซไผฐๆจกๅๅจๅ ถไปๆ้ซไธ็ๆๆใ
- ็ถญๅบๆฒๆ้ฒๆญฅ๏ผโ0.3๏ผ๏ผๅ ็บ่ชๆๆฒๆ็ถญๅบ๏ผ็พ็งๅผใๅฐๆๅ่ฉๅค็ๆๅญ่กจ็พ่ผๅทฎใ
- ่ฉๆธฌ็จ็ๆณจ้ณ็ฑ่ฉๅบซ่ชๅ่ฝๅบ๏ผไธๆฏไบบๅทฅๆจๆณจใ
- ็ๆๅฅไพ่ชๅฆไธๅๆจกๅ๏ผๅฏ่ฝๅธถๆๅฎ็็จ่ฉๅๅฅฝ๏ผๅทฒๆฟพๆ็ฐก้ซๅญ่ไธไปฝๅคง้ธ็จ่ฉ่กจ๏ผไฝๆฒๆไบบๅทฅ้ๅฅๆชขๆฅใ
ๆๆฌ
- ๆฌ้ไปฅ Apache-2.0 ้ๅบ๏ผ่ๅบๅบๆจกๅ็ธๅ๏ผๅบๅบ ยฉ Qwen Team๏ผใ
- ่จ็ทด่ณๆ๏ผCommon Voice๏ผCC0๏ผ๏ผ็ซๆณ้ขใ่กๆฟ้ขใๆณๅ้จ็้ๆพ่ณๆไพๆฟๅบ่ณๆ้ๆพๆๆฌๆขๆฌพ๏ผ็ฌฌ1็ไฝฟ็จ๏ผๅบ่ๆจ็คบ่ฆ
DATA.md๏ผBreeze-7B-Instruct๏ผApache-2.0๏ผ็ๆ็ๅฅๅญๆฒๆไฝฟ็จ้ๅถ๏ผ้จๆฌๆจกๅๅ ฌ้ใ - ๅ้ธๅฅ่ๆณจ้ณไพ่ช McBopomofo๏ผMIT๏ผ่ๆฐ้ ท้ณ่ฉๅบซ๏ผLGPL-2.1-or-later๏ผ๏ผๅชๅจ่ฉๆธฌๆไฝฟ็จ๏ผไธๅจ่จ็ทด่ชๆๅ งใ
English summary
A LoRA (r=16, all language-model linear layers) continued-pretraining of Qwen/Qwen3.5-0.8B on the same 4.7M tokens of freely reusable Taiwan Mandarin as Qwen3.5-2B-TW-IME: Common Voice zh-TW (CC0), Legislative Yuan gazette speeches and Executive Yuan news releases (Taiwan Open Government Data License v1.0, CC BY 4.0-compatible), laws (not copyrightable), and 30k everyday sentences generated locally with Breeze-7B-Instruct (Apache-2.0). No Wikipedia or Ministry of Education dictionary data. It scores candidate sentences for a Bopomofo input method; it is not a chat model. Sentence accuracy with the IME's mixing: 87.1 / 85.0% (Common Voice dev / test), 87.8% (Legislative Yuan), 78.0% (news), 71.7% (Wikipedia), vs. 85.2 / 83.2 / 85.3 / 76.8 / 72.0% for the stock 0.8B model, about as good as the stock 2B model and twice as fast on a CPU; single run, no confidence intervals.
- Downloads last month
- 87