ParallaxOpen/Vela-Lumen-31M-v1.1

Vela-Lumen-31M v1.1

A 47.8M parameter language model trained from scratch on a single laptop GPU.

Errata (corrected)

This card previously stated 35.5M parameters and listed "Weight tying: Yes". Both were wrong, and both have been fixed below. The weight file in this repo contains a separate lm_head.weight tensor of shape [24189, 512] (12,384,768 parameters), so the checkpoint is untied and the true count is 47,846,912. The earlier figure came from counting the model as if the input embedding and output projection were shared.

Measured directly from the model.safetensors header, not estimated: 47,846,912 parameters, 67 tensors. The 35.5M number was the tied count (35,462,144).

The previous headline GSM8K figure of 60.2% was also presented without the caveat that it is a teacher-forced / loss-based measurement, not free-running generation. That is now stated at the top of the benchmarks section instead of in a footnote, and the generation results — which are poor — are shown next to it.

---

Status: superseded, kept for reproducibility

v1.1 is superseded and is not recommended for use. It is retained so the training run is reproducible and so the audit history stays available.

Known problems beyond the corrections above:

• Free-running generation is poor. The card's own unseen-problem table already recorded 0.0% on generation tasks. Treat the loss-based benchmark numbers as a training-health signal only.
• The GGUF that used to be published here has been removed. It was not a faithful conversion: vela-lumen-31m-v1.1-f16.gguf was 143.0 MB where a faithful FP16 conversion of 47,846,912 parameters is 95.7 MB — roughly 1.5x too large, i.e. not a clean F16 export. Beyond the size mismatch, the architecture (dense GELU feed-forward with a GatedDeltaNet layer mix) has no equivalent in llama.cpp, so the file cannot produce correct text. It is deleted rather than left in place, because shipping it under a gguf tag invites people to download something that only produces garbage. The blob is still reachable in git history. Use model.safetensors.

Loading the safetensors

This architecture is not part of mainline transformers. The weights require the centauri modeling code; there is no AutoModel entry point in this repository, and none is claimed.

Files in this repo:
  model.safetensors         float32, 191.4 MB, 47,846,912 parameters
  model-fp16.safetensors    float16,  95.7 MB
  tokenizer.json            24,189-token BPE
  train_v11_release.py      the training script that produced this checkpoint

Architecture

Component	Value
Parameters	47,846,912 (47.8M)
Layers	8 (GatedDeltaNet mix)
Hidden dim	512
Q heads	8
KV heads	4 (GQA)
FFN dim	2,048
Norm	RMSNorm
Position	RoPE (theta = 10,000)
Vocab	24,189 (BPE)
Max seq	128 tokens
Weight tying	No — tok_emb and lm_head are separate tensors

Benchmarks

Read the methodology first. These are loss-based measurements:

Evaluation used next-token-prediction loss for GSM8K and TruthfulQA, and loss-comparison for the multiple-choice tasks. This is not free-running generation. A model can score well here while failing to produce a usable answer, and for this checkpoint it does.

Benchmark	v1.1 (loss-based)	n	Method
GSM8K	60.2%	8,792	teacher-forced, same distribution as training
ARC-Challenge	22.5%	2,590	loss-comparison (MC)
HellaSwag	24.9%	5,000	loss-comparison (MC)
TruthfulQA	29.8%	817	teacher-forced
WinoGrande	51.2%	5,000	loss-comparison (MC)

Generation results (free-running, from the original evaluation):

Task	Accuracy	n
Unseen word problems (generation)	0.0%	5
Unseen math MC	20.0%	15
Unseen logic	20.0%	5
Common sense	20.0%	5

The 60.2% -> 0.0% gap is the single most important fact about this checkpoint. The GSM8K score reflects fitting the question format of the training distribution, not arithmetic ability. Later work in this line confirmed the same effect at larger scale, where a 60% held-out score corresponded to 8% on hand-authored unseen questions.

Training

Trained from scratch on a single RTX 5060 laptop GPU. Corpus, step count and wall-clock are recorded in the training log shipped with this repository. train_v11_release.py is the script that produced the checkpoint.

License

CC BY-NC 4.0. Research and non-commercial use.