---
library_name: pytorch
tags:
- interpretability
- transcoder
- sparse-autoencoder
- arithmetic
- replicate
---

# transcoder-bilinear-2layer-seed2-layer1-r6

Replicate **6** of 10, of a TopK transcoder fitted to **layer
1** of
[`itzPotato/arithmetic-bilinear-2layer-seed2`](https://huggingface.co/itzPotato/arithmetic-bilinear-2layer-seed2).

## What varies across the 10 replicates

**The training data subset, and only that.** Each replicate draws a fresh
500,000-problem training subset from the base model's train
split. The transcoder's weight initialisation is **identical** across all ten
(`seed=0`), so the spread across replicates measures variance due
to the training data with the initialisation held constant.

This is deliberately not the usual "vary the training seed" replicate. It was
specified this way so that data variance and initialisation variance are not
confounded.

| | |
|---|---|
| varies | training data subset |
| held fixed | weight init (`seed=0`), validation slice, all hyperparameters |
| this replicate's subset seed | `1021006` |
| this replicate's subset digest | `b6ce78767750f97e` |

**The validation slice does not move.** It is the reserved
`[500,000, 510,000)`
problems used by every transcoder in this project, so reconstruction numbers are
comparable across replicates. Training draws exclude it, so no replicate ever
trains on its own validation data.

**Training subsets overlap.** Ten disjoint 500,000-problem
slices plus the 10,000 validation problems would need more of
the split than exists, so draws are independent rather than a partition, and two
replicates share roughly 10% of their problems in expectation.

## Architecture and recipe

Identical to [`itzPotato/transcoder-bilinear-2layer-seed2-layer1`](https://huggingface.co/itzPotato/transcoder-bilinear-2layer-seed2-layer1), the
single non-replicate transcoder for this layer:

| | |
|---|---|
| d_model (in and out) | 32 |
| features | 1024 (32x expansion) |
| active features per input (k) | 32 |
| decoder rows | unit norm |
| loss | MSE only, no L1, no lambda |
| optimiser | Adam, lr 0.0003, batches of 4096 activation vectors, one pass |

## Reconstruction

| | |
|---|---:|
| normalized reconstruction error | **0.0450** |
| fraction of variance unexplained | 0.0585 |
| raw MSE | 12.5713 |

`normalized = MSE / mean(target^2)`; predicting a constant zero scores 1.0.

## Loading

```python
from src.transcoder.source import load_transcoder
transcoder, provenance = load_transcoder("hub:itzPotato/transcoder-bilinear-2layer-seed2-layer1-r6", require_pinned=True)
```