syntropic-clx/Clyx_0.3-635.51M-BASE

Clyx 0.3 — 635.51M BASE

Clyx 0.3 BASE is a 635.51M-parameter decoder-only causal language model trained from scratch. It is a research model for text continuation, language-model experiments, and further training.

This release is a BASE checkpoint. It is not an instruction-tuned chatbot: prompts are continuation contexts, and the model is not expected to answer questions, obey formatting requests, or use tools reliably.

No external pretrained weights were used.

Model summary

Property	Value
Parameters	635,512,320
Architecture	Decoder-only Transformer
Hidden size	1,536
Transformer layers	24
Query attention heads	24
Key/value attention heads	4
Attention	Grouped-query attention (GQA)
Intermediate size	4,096
Normalization	RMSNorm
Activation	SwiGLU
Positional encoding	RoPE
RoPE base	10,000
Maximum context	2,048 tokens
Vocabulary	32,768 tokens
Tokenizer	ByteLevel BPE
Embeddings	Tied input/output embeddings
Export format	Safetensors
Export precision	FP32

Training

The model was trained from scratch with a custom PyTorch implementation. The prepared corpus contained approximately 2.12B training tokens and approximately 20.8M validation tokens.

The training run targeted approximately 2B presented training tokens. The public export is based on the best validation checkpoint saved during that run.

Setting	Value
Training hardware	NVIDIA A100 SXM4 80GB
Training precision	BF16 mixed precision
Optimizer	Fused AdamW
Learning-rate schedule	Cosine decay with warmup
Initial learning rate	3e-4
Minimum learning rate	3e-5
Gradient clipping	1.0
Optimization sequence length	1,024 tokens
Model context capacity	2,048 tokens
Target training budget	2,000,011,264 presented tokens

Training curve

The chart shows the recorded evaluation checkpoints from the BASE training run. The best validation loss was 2.7136 at step 116,000; the last recorded checkpoint was step 118,000 with validation loss 2.7154. Accuracy lines connect recorded observations for readability; no intermediate accuracy values were inferred. Accuracy is next-token accuracy, not an instruction-following or task benchmark score.

Training data

The corpus contained a mixture of:

• Russian text;
• English text;
• Python code;
• C and C++ code.

The private prepared corpus is not included in this repository. The model repository contains only the resulting inference weights, configuration, tokenizer, documentation, and export metadata.

The source datasets have their own licenses and terms. The Apache-2.0 license on this repository does not replace the obligations attached to the original training sources. Review source licenses before commercial redistribution or using the model in a product.

Intended use

Suitable uses include:

• multilingual text continuation experiments;
• studying a custom Transformer implementation;
• evaluating small from-scratch language models;
• further pretraining on a compatible corpus;
• instruction tuning or domain adaptation performed by the user.

This checkpoint is not intended to be used as a factual database, autonomous agent, safety-critical system, or production conversational assistant without additional training and evaluation.

Limitations

The model is not instruction-tuned, aligned, or safety-trained. It can produce incomplete, repetitive, grammatically incorrect, factually incorrect, or incoherent text. Performance varies substantially by language, domain, prompt, and sampling settings.

No MMLU, GSM8K, ARC, HumanEval, or other benchmark scores are claimed for this release. A lower language-model loss does not by itself establish instruction-following or reasoning ability.

Inference with ClyxBox

Install the separate inference runtime from PyPI:

python -m pip install clyxbox==0.3.2

Load the model directly from the Hugging Face repository:

from clyxbox import ClyxPipeline

pipe = ClyxPipeline.from_pretrained(
    "syntropic-clx/Clyx_0.3-635.51M-BASE",
    device="cuda",
)

text = pipe(
    "Once upon a time in a dark forest",
    max_new_tokens=128,
    temperature=0,
    top_p=0.95,
    top_k=50,
    repetition_penalty=1.4,
    stream=True,
)

print(text)

When dtype is omitted, ClyxBox uses native BF16 on supported CUDA GPUs and FP32 otherwise (including NVIDIA T4). The default continuation settings are temperature=0.65, top_p=0.90, top_k=50, and repetition_penalty=1.12. Sampling is stochastic; different runs can produce different continuations. These settings are a starting point, not a guarantee of quality.

Because this is a BASE model, a prompt such as 2 + 2 = is treated as text to continue. It is not a reliable request for the answer 4. Instruction-following requires a separate SFT or instruction-tuning stage.

Sharded weights

The 2.54 GB FP32 export is intentionally split into three Safetensors files:

• model-00001-of-00003.safetensors;
• model-00002-of-00003.safetensors;
• model-00003-of-00003.safetensors.

model.safetensors.index.json maps every tensor to its shard. Keep the index and all three shards in the same repository. ClyxBox reads the index automatically and loads the shards one at a time; do not rename or delete individual parts.

Repository files

• model-00001-of-00003.safetensors — first weight shard;
• model-00002-of-00003.safetensors — second weight shard;
• model-00003-of-00003.safetensors — third weight shard;
• model.safetensors.index.json — shard-to-tensor index;
• model_config.json — Clyx architecture configuration;
• config.json — export/runtime metadata;
• tokenizer.json and tokenizer metadata — matching tokenizer;
• export_manifest.json — file sizes, hashes, and export information;
• LICENSE — Apache License 2.0;
• README.md — this model card.

Training checkpoints, optimizer state, private corpus files, logs, and SFT checkpoints are intentionally excluded.

Related resources

• Source code: PavBss/LM
• Inference runtime: clyxbox on PyPI
• Project organization: syntropic-clx

License

Apache License 2.0.