NoTokenLM-Gen-4.5

A 20-million-parameter, byte-level, tokenizer-free language model. No subword vocabulary, no BPE — just raw UTF-8 bytes in, raw UTF-8 bytes out.

This is part of the NoTokenLM family: a series of small models built around one guiding question — how much can a genuinely small model do, if the architecture and training are done carefully, without leaning on scale to cover for weak design?

If you're looking for a model that reasons, does math, or holds a long conversation coherently — this isn't that, and this card will tell you exactly why not. If you're curious what a 20M-parameter transformer can actually pull off when it's pointed at simple narrative text — keep reading.

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "omurberaisik/NoTokenLM-Gen-4.5"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model.eval()

ids = tok("Deep in the forest, Jack found a shell", return_tensors="pt")["input_ids"]
out = model.generate(ids, max_new_tokens=35, temperature=0.5, top_k=40)
print(tok.decode(out[0]))

Or with the pipeline API:

from transformers import pipeline

pipe = pipeline("text-generation", model="omurberaisik/NoTokenLM-Gen-4.5", trust_remote_code=True)
print(pipe("The little dog", max_new_tokens=35, temperature=0.5, top_k=40))

Recommended sampling settings are in the How to actually run this thing section below.

---

What is this checkpoint, actually?

Gen-4.5 is a continuation of Gen-3.5-Focused — same architecture family (RoPE, RMSNorm, SwiGLU, byte-level), same lineage — carried forward through more training, not a from-scratch run and not a different model bolted on. Two things changed on top of that continued pretraining:

1. The model got wider. Gen-3.5-Focused ran at d_model=240 (9M parameters). Gen-4.5 grows that to d_model=360 (20M parameters) using a function-preserving width expansion — the existing trained weights are embedded into the larger matrices rather than the model restarting from random init, so the extra capacity is trained on top of what the smaller model had already learned, not instead of it.
2. Training kept going on the same kind of data — short, simple children's-story-style narrative text (simple sentences, one or two characters, a concrete object, a small everyday or lightly fantastical event). No general web text, no encyclopedic corpus, no math, no code was added in this pass. The reason to say this plainly: this checkpoint is not a knowledge model and was never trained to become one. It went from ~9M to ~20M parameters and kept training on the same narrow style of narrative text that shaped Gen-3.5-Focused in the first place — it got more capacity for that same skill, not a broader one.

At the time these results were produced, this checkpoint had gone through 6,900 training steps across 3 sessions post-width-growth, with a best validation loss of 1.0729. Numbers throughout this card come from the EMA (exponential moving average) weights, which is standard practice for evaluation at this scale — it consistently produces smoother, less erratic generations than the raw training weights.

---

How well does it actually write? (1,000-prompt manual evaluation)

We generated 1,000 completions — 1,000 distinct prompts, one generation each, temperature 0.5, top-k 40, 35 new bytes — and every single one was read by hand and sorted into one of three categories. No automated heuristic, no filtering, no cherry-picking; everything below is the real distribution.

Grading criteria (same strict standard as prior NoTokenLM evaluations — "grammatically fine but says nothing coherent" does not count as a win):

Category	Definition	Count	Share
Fully coherent	Correct grammar and the sentence actually makes sense — characters, objects, and pronouns tracked correctly throughout.	749	74.9%
Grammar OK, meaning breaks down	Reads like well-formed English sentence-by-sentence, but drifts into a pronoun/gender mismatch, a non-sequitur detail, or an object described inconsistently with the scene.	246	24.6%
Grammar breaks down	The sentence structure itself collapses — stuck repetition, a garbled or malformed clause.	5	0.5%

Result: 74.9% fully coherent, 99.5% grammatically correct overall, on a test set with no repeated prompts and no simplification.

Real examples, unedited, that landed in "fully coherent":

"Leo was carrying a folded map very carefully, while Sam was making a plan to find a wool scarf."
  -> "He had heard tales of a magical sc[arf]"

"Just before sunset, the young fox was cleaning up at the market after a long day."
  -> "The warm smell of fresh fruit fill[ed the air]"

"Long ago, Rosie found a folded map in the attic."
  -> "It was hidden in a box of old book[s]"

"Every afternoon, Otto spent the whole morning in the flower field."
  -> "She loved to play there every day."

"Without any warning, Sam was looking for a stack of leaves on the hill."
  -> "He had heard tales of a secret soc[iety]"

Real examples that landed in "grammar OK, meaning breaks down" (also unedited):

"On a rainy Tuesday, Anna noticed a wooden box was missing."
  -> "It was her lost city that she had"
  <- grammatically fine, but "her lost city" doesn't follow from a missing wooden box

A pronoun/gender mismatch a sentence or clause after a character is introduced is the single most common failure inside this category.

Real examples that landed in "grammar breaks down":

"That very night, Nora was looking for a paint brush in the toy store."
  -> "She had a big blue blue but never"      <- stuck repetition, then a dangling fragment

"Aria began to dig a hole in the village, while Miles asked a friend for help carrying a glass jar."
  -> "They began to dug and play and sil"      <- wrong verb inflection + trails into word salad

"Early in the spring, Hazel noticed a broken toy was missing."
  -> "It was a toy was found in a box. E"       <- doubled subject, syntax collapses

The full per-example labels for all 1,000 outputs are included in gen45_1000_test_outputs.json under the manual_category field.

---

Real-world benchmark results

Manual coherence grading is our own methodology. To see how the model does on standard, external academic benchmarks — the kind used to evaluate real language models, not just our own test set — we ran it against three established suites. None of these involve code or math; all three test grammar, commonsense, and long-range language understanding, which is what a text-only, 20M-parameter narrative model can realistically be judged on. Everything below is a genuine, reproducible score, run directly against the released weights, sample sizes noted throughout.

BLiMP (Benchmark of Linguistic Minimal Pairs)

BLiMP tests grammatical knowledge across 67 distinct syntactic/morphological phenomena, scoring whether a model assigns higher likelihood to a grammatical sentence over a minimally-different ungrammatical one. We ran all 67 paradigms (30 pairs each, 2,010 total pairs) and compared against the official baselines reported in the BLiMP paper (n-gram, LSTM, Transformer-XL, and GPT-2-large, 774M parameters — note this is not the small 124M GPT-2).

Model	Params	BLiMP avg. (67 paradigms)
5-gram	—	61.2%
NoTokenLM-Gen-4.5	20M	66.8%
LSTM	—	69.8%
Transformer-XL	—	69.6%
GPT-2-large	774M	83.0%

Gen-4.5 clears the n-gram baseline and lands close to the LSTM/Transformer-XL range, despite being dramatically smaller and trained on a narrow slice of narrative text rather than a general corpus. It also outperforms GPT-2-large on 10 of the 67 individual paradigms — mostly short-range morphological agreement (e.g. irregular_past_participle_verbs: 100% vs. 86.1%; sentential_negation_npi_licensor_present: 100% vs. 97.1%). It falls sharply behind on paradigms requiring long-distance syntactic dependencies — island constraints, wh-extraction over long distances — several of which score near or below chance (e.g. wh_vs_that_with_gap_long_distance: 6.7%). This tracks with everything else in this card: strong on local, short-range grammar; weak the moment a dependency has to be tracked across a long stretch of the sentence.

HellaSwag (commonsense sentence completion)

HellaSwag gives a context and four possible continuations; the model has to pick the one a human would find most plausible. We scored 150 examples via length-normalized log-likelihood.

	Accuracy
NoTokenLM-Gen-4.5	20.7%
Random chance (4-way)	25.0%

Below random chance. This is a real and honest result, not an oversight: HellaSwag continuations describe everyday human activities in a register the model has never been trained on, and the model's narrative-generation bias appears to actively mislead it here rather than simply providing no signal.

LAMBADA (long-context final-word prediction)

LAMBADA gives a full paragraph and asks the model to predict the final word — a test of whether a model can actually use long-range context, not just local grammar. We scored 100 examples via greedy decoding, exact string match.

Model	Params	LAMBADA accuracy
NoTokenLM-Gen-4.5	20M	12.0%
GPT-2-small	124M	~46%
GPT-2-large	774M	~59%
GPT-3 (zero-shot)	175B	76.2%

This is the clearest evidence of the model's core limitation: it was never trained to hold onto and use long-range context to predict a specific target, and it shows. This result should be read alongside the BLiMP long-distance-dependency paradigms above — both point at the same gap.

The honest summary

Skill	Result
Byte-level spelling accuracy (own eval, 7,271 words)	99.99% correct
Short-range grammar (BLiMP)	66.8% — competitive with LSTM/Transformer-XL
Long-range syntax (BLiMP subset)	near chance on several paradigms
Commonsense reasoning (HellaSwag)	below chance
Long-context word prediction (LAMBADA)	far below even the smallest GPT-2

Put together, this model has clearly and specifically learned how to produce well-formed, correctly-spelled English at the sentence level — and just as clearly has no world knowledge and very limited ability to track information across a long context. That's not a hedge; it's the actual shape of what 20M parameters trained on narrow narrative text buys you, and we'd rather state it plainly than let a single headline number stand in for the whole picture.

---

Why did the coherence score go down from Gen-3.5-Focused's 86%?

Gen-3.5-Focused's internal test reported roughly 86% coherence. The 1,000-prompt manual review above lands at 74.9% fully coherent. That's a real number, not a typo, and it deserves an honest explanation rather than a footnote.

The two evaluations are not comparable, and the difference in methodology explains essentially all of the gap:

• Gen-3.5-Focused's 86% figure came from a small, fixed set of prompts, reused repeatedly. A handful of simple, easy story openers were run over and over, and the model's job was effectively to keep producing plausible continuations of prompts it had, in a real sense, already been extensively probed on. High coherence on a narrow, repeated, easy prompt set tells you the model is stable on that set — it doesn't tell you much about how it handles anything else.
• This evaluation is 1,000 prompts, and every single one is genuinely different — distinct characters, settings, objects, and sentence structures drawn from large independent word pools, combined into two-clause and multi-character constructions the model has to track simultaneously (e.g. "Aria began to dig a hole in the village, while Miles asked a friend for help carrying a glass jar" — two separate characters, two separate actions, one sentence). Nothing here was cherry-picked, repeated, or simplified to make the model look good. This is a substantially harder and more diverse test than the one that produced the 86% figure.

Put plainly: a 75% fully-coherent rate on 1,000 unique, harder, more varied prompts is a stronger result than an 86% rate on 10 easy prompts repeated many times. The drop in the headline number reflects a much tougher exam, not a worse model. If anything, running Gen-4.5 against the old narrow-and-repeated style of prompt would very likely reproduce numbers at or above the old 86% — we just don't think that test tells you anything useful anymore, so we didn't lean on it here.

For reference, when the same 1,000-prompt, unique-prompt methodology used above was run against Gen-3.5-Focused itself (rather than its old 10-prompt-repeated test), it scored:

Category	Share
Fully coherent	69.2%
Grammar OK, meaning breaks down	30.6%
Grammar breaks down	0.2%

If you want the full 1,000 raw outputs behind that score, they'll be posted to the NoTokenLM-Gen-3.5-Focused repo shortly — check there if you want to inspect them directly.

---

What it's actually good at

• Tracking a single clear subject through one sentence. Given a concrete character and a concrete action, the model reliably keeps the continuation on-topic and grammatically sound.
• Two-clause, two-character prompts, most of the time. Roughly three-quarters of prompts combining two separate characters and actions in one sentence still resolved coherently — this is a harder structure than a single-clause opener, and the model holds up on it more often than not.
• Holding a topic across multiple sentences. In informal testing with longer, multi-sentence generations, the model can stay on-topic and track the same subject for up to 4–5 sentences before drifting — beyond the single-completion setup used in the main 1,000-prompt evaluation above. This hasn't been run through the same scale or rigor as that evaluation, so treat it as a directional observation rather than a benchmarked number.
• Staying inside its genre. Every completion stayed within simple narrative/story register — no drift into unrelated registers, no attempted factual claims, no code, no non-narrative text.

What it's not good at, and why

• No world knowledge, by design. This checkpoint was trained exclusively on short narrative story text, not general web or encyclopedic text. It was never exposed to facts about the real world, so it cannot answer factual questions reliably. This isn't a bug to fix with more training on the same data — it would need a different data mix entirely.
• Pronoun and gender consistency across a sentence boundary. This is the single largest source of the "grammar OK, meaning breaks down" category — a character introduced as "she" becomes "he" a clause later, or a named character's implied gender shifts. At 20M parameters with no dedicated consistency-tracking objective, this is the model's most common failure mode.
• Occasional stuck repetition or malformed inflection. In a small number of cases (0.5% here) the model repeats a word ("blue blue") or produces an incorrect verb form ("began to dug") and the sentence doesn't recover within the generation window.

---

How to actually run this thing

Recommended: temperature 0.5, top-k 40. All the numbers in this card were produced at these settings. Going much higher noticeably increases how often generations drift into the "grammar OK, meaning breaks down" and "grammar breaks down" categories; going much lower produces safer but more repetitive text.

---

Architecture details

Parameters	~20.3M
Layers	13
d_model	360 (grown from 240 via function-preserving width expansion)
Attention heads	6
Feedforward dim	960
Vocabulary	256 (raw bytes, no tokenizer)
Context length	1024 bytes
Position encoding	RoPE
Normalization	RMSNorm
Feedforward	SwiGLU
Output layer	Weight-tied to input embedding
Training	Byte-level next-token prediction, continued pretraining from Gen-3.5-Focused (pretraining only — no instruction tuning)

Training data

Short, simple children's-story-style narrative text, continued from the same data mix used for Gen-3.5-Focused rather than replaced. No general web text, no encyclopedic corpus, no math-formatted data, and no instruction-following format is present in this checkpoint's training.

---

Part of the NoTokenLM family — small models, built and evaluated honestly.

---

License

This project is licensed under the Apache License 2.0. See the LICENSE file for the full license text.