StarpowerTechnology/WVY-Liquid-Recurrent-Depth

WVY Liquid Recurent Depth v1

Tiny-Recursive-LM

LiquidWVY v1 is a tiny experimental model. The architecture combines three ideas:

1. Liquid/LFM-style local mixing: double-gated causal short convolutions cheaply process nearby tokens.
2. Gated DeltaNet memory: recurrent fast-weight memory learns when to forget, update, and retrieve information across longer sequences.
3. Recurrent depth: a shared core is executed several times before output. This is the loop-transformer or recursive-forward-pass idea: the model performs more latent computation while reusing the same core parameters.

Grouped-query attention remains in the architecture for exact global token-to-token access. Every operator is wrapped with RMS normalization, residual connections, and a SwiGLU feed-forward network.

This combination is experimental. The component ideas have research support separately; LiquidWVY v1 has not yet been pretrained or benchmarked as a combined architecture. Training and evaluation are how its value must be established.

Model flow

tokens → embeddings
       → prelude: [LFM short convolution → grouped-query attention]
       → recurrent adapter + shared core: [Gated DeltaNet ↔ attention] × N loops
       → coda: [LFM short convolution → grouped-query attention]
       → RMSNorm → tied language-model head → next-token logits

The prelude grounds the input. The recurrent adapter injects the prelude representation into every loop. The shared core repeatedly updates the hidden state without adding a new copy of its weights for each recurrence. The coda converts the refined state into the representation used for token prediction.

Repository layout

Path	Purpose
src/wvy_experimental/modeling_liquid_wvy.py	Complete LiquidWVY v1 architecture
src/wvy_experimental/configuration_liquid_wvy.py	Hugging Face configuration
src/wvy_experimental/train.py	Fresh pretraining, validation, checkpoints, and export
src/wvy_experimental/scan_data.py	Exact token counter and parameter-target report
src/wvy_experimental/build_config.py	Searches full configurations for a requested parameter count
src/wvy_experimental/prepare_tokenizer.py	Byte-level BPE tokenizer training
configs/	Prepared approximately 10M, 25M, and 50M configurations
assets/tokenizer/	Public LFM2.5 tokenizer assets for compatibility experiments
notebooks/	Google Colab and Kaggle training kits
reference/transformers-lfm2/	Unchanged public LFM2 architecture source used as a reference
huggingface/model-card-template.md	Template for the trained checkpoint repository

Prepared scales

All three configurations use an 8,192-token vocabulary and four recurrent passes by default.

Configuration	Actual parameters	Physical depth	Effective depth
configs/wvy_10m.json	9,852,888	9	24
configs/wvy_25m.json	24,272,848	6	12
configs/wvy_50m.json	50,210,540	7	16

Physical depth counts unique stored layers. Effective depth counts every pass through the shared recurrent layers. Changing recurrent_steps changes computation and effective depth without changing the parameter count.

Supported training data

Pass one file or a directory. Directories are searched recursively. Supported formats are .txt, .md, .json, .jsonl, .csv, and .parquet. Structured files use a text field by default; pass --text-field content for another field.

Local installation

Use Python 3.10 or newer. A CUDA GPU is recommended for real training.

cd WVY-Experimental
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e .

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1.

Prepare a run

Train a compact tokenizer from the corpus:

wvy-tokenizer --data data/ --vocab-size 8192 --output artifacts/tokenizer

The bundled Liquid tokenizer has 65,536 tokens. Its embedding table consumes too much of a 1M–50M budget, so a smaller corpus-specific tokenizer is recommended.

Count the tokens and write the dataset report:

wvy-scan \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --tokens-per-parameter 20 \
  --output artifacts/data_report.json

The ratio is a configurable planning input. No validated scaling law exists for this exact combined architecture. The report therefore presents a suggested starting target rather than a guaranteed optimum.

Generate the nearest model configuration:

wvy-build \
  --report artifacts/data_report.json \
  --tokenizer artifacts/tokenizer \
  --output artifacts/model_config.json

Or prepare all three items with one helper:

scripts/prepare_data_and_model.sh data/ 8192 20

Train from fresh weights

wvy-train \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --config artifacts/model_config.json \
  --output outputs/liquidwvy-v1 \
  --sequence-length 1024 \
  --batch-size 4 \
  --gradient-accumulation 8 \
  --learning-rate 3e-4 \
  --epochs 1 \
  --bf16 \
  --gradient-checkpointing

Use --fp16 on GPUs without BF16 support. Remove both precision flags for FP32. Lower the batch size if memory runs out. Resume an interrupted run with the same arguments plus:

--resume-from-checkpoint outputs/liquidwvy-v1/checkpoint-500

The final directory is outputs/liquidwvy-v1/final/. It contains trained weights, custom architecture files, configuration, and tokenizer assets. Load that checkpoint with:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "outputs/liquidwvy-v1/final",
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "outputs/liquidwvy-v1/final",
    trust_remote_code=True,
)

The model accepts recurrent_steps= during a forward pass. Training defaults to the number in the configuration. Comparing several recurrence counts is part of evaluating whether additional latent computation helps.

Google Colab

Open notebooks/WVY_Colab_Training_Kit.ipynb in Colab, select a GPU runtime, and run the cells in order. It uploads the kit and corpus, installs dependencies, prepares the tokenizer and configuration, trains fresh weights, and downloads the final checkpoint.

For long runs, mount Google Drive and set OUTPUT_DIR to a Drive location so checkpoints survive runtime resets.

Kaggle

Add WVY-Experimental.zip and the corpus as Kaggle notebook inputs, enable a GPU, and open notebooks/WVY_Kaggle_Training_Kit.ipynb. Set KIT_ZIP and DATA_PATH in the first code cell, then run all cells. The trained archive appears in /kaggle/working/.

Hugging Face publishing

Copy huggingface/model-card-template.md to the trained final/ directory as README.md, fill in the corpus, compute, loss, evaluation, and intended-use sections, then upload the complete final/ directory to a new model repository. Loading custom architecture code requires trust_remote_code=True until the architecture is integrated into Transformers.

Implementation status

The Gated DeltaNet path implements the recurrent gated-delta update directly in PyTorch. It is mathematically complete and works on CPU or GPU, but the token recurrence currently executes sequentially. Long-sequence high-throughput training will benefit from a compatible chunkwise CUDA/Triton kernel after the architecture is validated. The standard attention path uses PyTorch scaled-dot-product attention.

No Liquid AI pretrained weights, private corpus, private training pipeline, or teacher logits are included. The reference/ directory contains public source for study; LiquidWVY v1 runtime code is maintained under src/wvy_experimental/.

Research basis and licensing

• LFM2 public architecture: https://github.com/huggingface/transformers/tree/main/src/transformers/models/lfm2
• Gated Delta Networks: https://arxiv.org/abs/2412.06464
• Recurrent-depth latent reasoning: https://arxiv.org/abs/2502.05171

See THIRD_PARTY_NOTICES.md and licenses/. The project name describes an experimental design influenced by public research; it does not claim that Liquid AI created or endorsed LiquidWVY v1.