Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

When we teach students about LLMs, explanations inherited from word2vec can leave the impression that tokens are inherently semantic units and that their input embeddings are where meaning resides. This work offers a concrete experiment for questioning that picture.

We replace the trainable input embedding table with fixed 16-dimensional binary vectors, deterministically repeated to model width, with no additional trainable input projection. This is a token-identity encoding, not weight quantization. At 1.7B-class scale, these models acquire substantial language-modeling capability, although the learned-input control performs better on several key benchmarks.

There is nothing linguistically special about sixteen dimensions: it is the minimum fixed-length binary width needed to distinguish the 49,152 tokens in our vocabulary. Wider binary vectors—for example, 32-dimensional ones—or suitably constructed, distinct sine/cosine-based token codes are plausible alternatives, but we do not test them here. Preserving token identity is a coding requirement; whether a network can learn effectively from a particular encoding is an empirical question.

The conceptual distinction is between identifying a token and learning to interpret and use token sequences in context. Tokenization defines the input units; it does not make each unit a self-contained piece of meaning.

Nor does removing the trainable input table establish that meaning simply moves into the first trainable layer. The experiment does not determine where semantic information is represented or how it is distributed across layers. Instead, fixing the input coordinates provides a controlled setting for investigating representation formation—without making the model automatically interpretable.

For students, the takeaway is that token identity, input representation, and meaning should not be conflated. Word-vector analogies can remain useful illustrations without becoming the lens through which all LLM computation is understood.

We report one training run per interface, and the output head remains trainable. Models and research artifacts: https://huggingface.co/collections/Bochkov/do-language-models-need-a-trainable-input-embedding-table