transcoder replicate bilinear-2layer-seed2-layer1-r7: weights, config, training record

README.md

@@ -0,0 +1,76 @@
+---
+library_name: pytorch
+tags:
+- interpretability
+- transcoder
+- sparse-autoencoder
+- arithmetic
+- replicate
+---
+
+# transcoder-bilinear-2layer-seed2-layer1-r7
+
+Replicate **7** of 10, of a TopK transcoder fitted to **layer
+1** of
+[`itzPotato/arithmetic-bilinear-2layer-seed2`](https://huggingface.co/itzPotato/arithmetic-bilinear-2layer-seed2).
+
+## What varies across the 10 replicates
+
+**The training data subset, and only that.** Each replicate draws a fresh
+500,000-problem training subset from the base model's train
+split. The transcoder's weight initialisation is **identical** across all ten
+(`seed=0`), so the spread across replicates measures variance due
+to the training data with the initialisation held constant.
+
+This is deliberately not the usual "vary the training seed" replicate. It was
+specified this way so that data variance and initialisation variance are not
+confounded.
+
+| | |
+|---|---|
+| varies | training data subset |
+| held fixed | weight init (`seed=0`), validation slice, all hyperparameters |
+| this replicate's subset seed | `1021007` |
+| this replicate's subset digest | `18a0e44c32c346cc` |
+
+**The validation slice does not move.** It is the reserved
+`[500,000, 510,000)`
+problems used by every transcoder in this project, so reconstruction numbers are
+comparable across replicates. Training draws exclude it, so no replicate ever
+trains on its own validation data.
+
+**Training subsets overlap.** Ten disjoint 500,000-problem
+slices plus the 10,000 validation problems would need more of
+the split than exists, so draws are independent rather than a partition, and two
+replicates share roughly 10% of their problems in expectation.
+
+## Architecture and recipe
+
+Identical to [`itzPotato/transcoder-bilinear-2layer-seed2-layer1`](https://huggingface.co/itzPotato/transcoder-bilinear-2layer-seed2-layer1), the
+single non-replicate transcoder for this layer:
+
+| | |
+|---|---|
+| d_model (in and out) | 32 |
+| features | 1024 (32x expansion) |
+| active features per input (k) | 32 |
+| decoder rows | unit norm |
+| loss | MSE only, no L1, no lambda |
+| optimiser | Adam, lr 0.0003, batches of 4096 activation vectors, one pass |
+
+## Reconstruction
+
+| | |
+|---|---:|
+| normalized reconstruction error | **0.0448** |
+| fraction of variance unexplained | 0.0582 |
+| raw MSE | 12.5139 |
+
+`normalized = MSE / mean(target^2)`; predicting a constant zero scores 1.0.
+
+## Loading
+
+```python
+from src.transcoder.source import load_transcoder
+transcoder, provenance = load_transcoder("hub:itzPotato/transcoder-bilinear-2layer-seed2-layer1-r7", require_pinned=True)
+```

config.json

@@ -0,0 +1,15 @@
+{
+  "d_model": 32,
+  "expansion": 32,
+  "k": 32,
+  "lr": 0.0003,
+  "batch_size": 4096,
+  "seed": 0,
+  "unit_norm_decoder": true,
+  "n_features": 1024,
+  "layer": 1,
+  "base_source": "trained:bilinear-2layer-seed2",
+  "replicate": 7,
+  "init_seed": 0,
+  "train_subset_seed": 1021007
+}

transcoder.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22ac8efe94b840bfbcd0b0ceab39174a7c9a8dbe378ad6b096a26d18f3712c0b
+size 266656