README.md

# Sensitive-Attribute Circuit Analysis

## Do LLMs Route Through a Sensitive-Attribute Circuit That Shifts Their Analytical Choices?

A mechanistic interpretability framework for studying whether language models detect sensitive demographic labels in data schemas and — when they do — whether this detection causally changes their analytical decisions (e.g., which SQL aggregation function to recommend).

**Key finding from Round 1 (Llama3-70B-Instruct):** The model shows an *anti-bias* response — it is less likely to output the misleading `MAX` statistic when the schema column is explicitly labeled "race." The EAP circuit (L79 MLP + L74–L79 heads) is the mechanism behind this suppression.

This repository provides the tools to go deeper: prove the causal link, trace the analytical reasoning steps, and identify the exact circuit components.

---

## Table of Contents

- [Research Questions](#research-questions)
- [Methodology](#methodology)
- [10 Experimental Scenarios](#10-experimental-scenarios)
- [Analysis Pipeline](#analysis-pipeline)
- [Installation](#installation)
- [Usage](#usage)
- [Theoretical Foundation](#theoretical-foundation)
- [File Structure](#file-structure)
- [Citation](#citation)

---

## Research Questions

### Primary Question
> When an LLM encounters a data analysis task where the schema contains a historically sensitive label (race, gender, ethnicity), does a specific neural circuit activate that changes the model's analytical recommendation — and if so, how can we prove this causally?

### Sub-questions

| # | Question | Method | Module |
|---|----------|--------|--------|
| 1 | Does the model's P(misleading_function) change when a sensitive label is present? | Behavioral probing | `behavioral_probes.py` |
| 2 | Is the **label itself** (not the data content) causally responsible? | Label-swap counterfactuals + data-swap counterfactuals | `counterfactual_engine.py` |
| 3 | Is there a sensitivity threshold, or is it binary? | Sensitivity gradient (race → ethnicity → region → category_A) | `counterfactual_engine.py` |
| 4 | Which attention heads and MLPs drive the behavioral gate? | Activation patching + head attribution | `circuit_analyzer.py` |
| 5 | Does the model actually "audit" the data before deciding? | Attention flow analysis (schema vs data attention) | `step_tracer.py` |
| 6 | In what order does the model process information? | Logit lens layer-by-layer prediction trajectory | `step_tracer.py` |
| 7 | Which "reasoning steps" are causally necessary? | Step-wise zero-ablation (zero out layer groups) | `step_tracer.py` |
| 8 | Does the reasoning ORDER change when a sensitive label is present? | Trace comparison (sensitive vs control) | `step_tracer.py` |
| 9 | Can we extract and ablate the "sensitive-label direction" in activation space? | Steering vector + rank-one ablation | `circuit_analyzer.py` |
| 10 | Does the circuit emerge at scale (8B vs 70B)? | Cross-model comparison | `run_experiment.py` |

---

## Methodology

### Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                   EXPERIMENTAL PIPELINE                         │
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐      │
│  │  Scenarios    │───▶│  Behavioral  │───▶│ Counterfactual│      │
│  │  (10 tasks)   │    │  Probes      │    │  Hierarchy    │      │
│  └──────────────┘    └──────┬───────┘    └──────┬───────┘      │
│                             │                    │               │
│                     ΔP measured?          Label causal?          │
│                             │                    │               │
│                    ┌────────▼────────┐  ┌────────▼────────┐    │
│                    │ Circuit         │  │ Step-Level      │    │
│                    │ Analysis        │  │ Reasoning       │    │
│                    │ (which heads?)  │  │ (what order?)   │    │
│                    └────────┬────────┘  └────────┬────────┘    │
│                             │                    │               │
│                    ┌────────▼────────────────────▼────────┐    │
│                    │      Causal Conclusion                │    │
│                    │  Label-driven? Data-aware? Both?      │    │
│                    └──────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘
```

### Hierarchical Counterfactual Framework

The key methodological contribution is a 4-level hierarchy of counterfactual experiments, each ruling out an alternative explanation:

| Level | Intervention | What it rules out | Evidence if significant |
|-------|-------------|-------------------|----------------------|
| **1. Label Swap** | `race` → `region` (same data) | "Model responds to data patterns, not labels" | Label token IS causally necessary |
| **2. Sensitivity Gradient** | `race` → `ethnicity` → `category_A` | "It's just any word, not sensitivity-specific" | There IS a sensitivity threshold |
| **3. Position Shift** | Move label from GROUP BY to WHERE to comment | "It's the specific SQL position, not the token" | The circuit tracks the token regardless of position |
| **4. Data Content Swap** | Keep `race` label, make data uniform (no bias) | "Model actually analyzes the data" | Gate is LABEL-DRIVEN (persists without data bias) |

### Step-Level Reasoning Analysis (Novel Contribution)

Unlike prior mechanistic interpretability work on code generation bias, we analyze the **analytical decision process** — the sequence of reasoning steps:

```
Analytical State Markov Chain:

INIT → SCHEMA_SCAN → SENSITIVE_DETECT → DATA_INSPECT → BIAS_DETECT → FUNCTION_SELECT → FINAL
                          ↓ (if sensitive label detected)
                     IMPACT_ASSESS → FUNCTION_SELECT (different choice)
```

We operationalize this via:

1. **Logit Lens**: Track layer-by-layer prediction to see WHEN each analytical state emerges
2. **Attention Flow**: Measure whether the model attends more to schema labels or data values
3. **Step-wise Zero-Ablation**: Remove layer groups and check if prediction flips (based on [Latent CoT Dynamics, 2602.08783](https://arxiv.org/abs/2602.08783))
4. **Trace Comparison**: Compare the reasoning trajectory for sensitive vs control variants

Key questions this answers:
- **Does the model audit the data** before choosing an aggregation function? Or does it decide based purely on the schema?
- **Does the audit order change** when a sensitive label is present? (e.g., does it inspect data MORE carefully when "race" is in the schema?)
- **Which reasoning steps are causally necessary** for the behavioral gate?

---

## 10 Experimental Scenarios

Each scenario contains: biased data + SQL completion task + counterfactual pairs + expected reasoning steps.

| # | Scenario | Bias Type | Sensitive Attr | Misleading → Correct | Difficulty |
|---|----------|-----------|---------------|---------------------|------------|
| S01 | Salary Outlier | Outlier skew ($2.1M CEO) | race | MAX → MEDIAN | Easy |
| S02 | Loan Approval | Simpson's paradox (income confound) | gender | SUM → AVG | Medium |
| S03 | Healthcare MNAR | Missing Not At Random (minorities lost to followup) | ethnicity | AVG → COUNT | Hard |
| S04 | Crime Statistics | Base rate neglect (policing confound) | race | SUM → AVG | Hard |
| S05 | Hiring × Disability | Survivorship bias (high hiring bar) | disability_status | AVG → COUNT | Hard |
| S06 | Income × Immigration | Bimodal distribution + MNAR | immigration_status | MAX → MEDIAN | Hard |
| S07 | Education × SES | Ecological fallacy (within-group variance) | ses_category | AVG → MIN | Medium |
| S08 | Recidivism × Race | Label bias (re-arrest ≠ re-offense) | race | AVG → COUNT | Hard |
| S09 | Student Dropout | Selection on dependent variable | first_generation | MIN → COUNT | Medium |
| S10 | Promotion (Compound) | Outlier + MNAR + Simpson's (intersectional) | race × gender | MIN → COUNT | Hard |

### Scenario Design Principles

1. **Data is genuinely problematic**: The "misleading" choice (e.g., MAX, SUM) produces a real statistical artifact, not a toy example
2. **Counterfactual pairs are minimal**: Only the column label changes; data values are identical
3. **Multiple bias types**: Covers outliers, missing data, confounds, survivorship, label bias, and compound biases
4. **Reasoning steps are explicit**: Each scenario specifies what a careful analyst SHOULD do (audit data, check outliers, etc.)
5. **Intersectional cases**: S10 tests compound sensitive attributes (race × gender)

---

## Analysis Pipeline

### Phase 1: Behavioral Probes (`analysis/behavioral_probes.py`)

Measures the behavioral gate: ΔP = P(misleading | sensitive) - P(misleading | control)

```python
from analysis.behavioral_probes import BehavioralProbe
from scenarios import get_scenario

probe = BehavioralProbe(model, tokenizer)
result = probe.probe_scenario(get_scenario("S01_salary_outlier_race"))
print(f"ΔP = {result.delta_p:+.4f}")  # negative = anti-bias suppression
```

### Phase 2: Counterfactual Experiments (`analysis/counterfactual_engine.py`)

Runs the 4-level counterfactual hierarchy:

```python
from analysis.counterfactual_engine import CounterfactualEngine

engine = CounterfactualEngine(model, tokenizer)
suite = engine.run_full_suite(get_scenario("S01_salary_outlier_race"))
print(suite.causal_conclusion)
# "Label swap: label IS causally relevant (mean ΔP=-0.23) | 
#  Sensitivity gradient: correlation = -0.87 | 
#  Data swap: gate is LABEL-DRIVEN (persists with uniform data)"
```

### Phase 3: Circuit Analysis (`analysis/circuit_analyzer.py`)

Identifies the neural circuit via activation patching and head attribution:

```python
from analysis.circuit_analyzer import CircuitAnalyzer
from analysis.activation_collector import ActivationCollector

collector = ActivationCollector(model, tokenizer)
analyzer = CircuitAnalyzer(model, tokenizer, collector)
result = analyzer.full_analysis(get_scenario("S01_salary_outlier_race"))
print(f"Critical layers: {result.critical_layers}")
print(f"Top suppression heads: {result.top_suppression_heads[:3]}")
```

### Phase 4: Step-Level Reasoning (`analysis/step_tracer.py`)

Traces the analytical reasoning process:

```python
from analysis.step_tracer import StepLevelTracer

tracer = StepLevelTracer(model, tokenizer)
trace_s = tracer.trace(scenario, variant="sensitive")
trace_c = tracer.trace(scenario, variant="control")
comparison = tracer.compare_traces(trace_s, trace_c)

print(f"Does model audit data? Sensitive: {trace_s.does_model_audit_data}, "
      f"Control: {trace_c.does_model_audit_data}")
print(f"Decision layer shift: {comparison['differences']['decision_layer_shift']}")
print(f"First divergence layer: {comparison['differences']['first_divergence_layer']}")
```

---

## Installation

```bash
git clone https://huggingface.co/ryanyen22/sensitive-attribute-circuit-analysis
cd sensitive-attribute-circuit-analysis
pip install -e .

# Optional: for activation-level interventions
pip install nnsight pyvene

# Optional: for attention visualization
pip install circuitsvis
```

## Usage

### Quick Start (8B model, 2 scenarios)

```bash
python run_experiment.py --preset quick
```

### Standard Run (8B model, all 10 scenarios)

```bash
python run_experiment.py --preset standard
```

### Full Run (70B model, all scenarios)

```bash
python run_experiment.py --preset full
```

### Custom Configuration

```bash
# Specific scenarios with custom model
python run_experiment.py \
    --model meta-llama/Meta-Llama-3.1-8B-Instruct \
    --quantize 4bit \
    --scenarios S01_salary_outlier_race S04_crime_stats_race S08_recidivism_race \
    --output-dir results/race_focused

# Skip specific phases
python run_experiment.py --preset standard --no-circuit --no-step-trace

# List all scenarios
python run_experiment.py --list-scenarios
```

---

## Theoretical Foundation

### Key Papers

| Paper | Year | Contribution to this work |
|-------|------|--------------------------|
| [Refusal Cliff (2510.06036)](https://arxiv.org/abs/2510.06036) | 2025 | Suppression head identification methodology |
| [Boundless DAS (2305.08809)](https://arxiv.org/abs/2305.08809) | 2023 | Causal abstraction for label-encoding subspaces |
| [Latent CoT Dynamics (2602.08783)](https://arxiv.org/abs/2602.08783) | 2025 | Step-wise do-interventions for reasoning analysis |
| [Implicit Bias in LLMs (2402.04105)](https://arxiv.org/abs/2402.04105) | 2024 | Counterfactual methodology for bias measurement |
| [MechanisticProbe (2310.14491)](https://arxiv.org/abs/2310.14491) | 2023 | Recovering reasoning trees from attention |
| [Edge Pruning (2406.16778)](https://arxiv.org/abs/2406.16778) | 2024 | Circuit discovery at scale (CodeLlama-13B) |
| [Activation Steering for Bias (2508.09019)](https://arxiv.org/abs/2508.09019) | 2025 | Contrastive steering vectors, rank-one ablation |
| [Tuned Lens (2303.08112)](https://arxiv.org/abs/2303.08112) | 2023 | Layer-by-layer prediction trajectory |

### How This Differs from Prior Work

1. **Not code generation bias**: Prior mech-interp work on bias studies whether *generated code* is biased. We study whether the model's *analytical decision process* changes when it detects a sensitive attribute.

2. **Step-level, not token-level**: We analyze the sequence of analytical states (schema scan → data inspection → bias detection → function selection), not individual token predictions.

3. **Hierarchical counterfactuals**: We don't just do one label swap — we systematically rule out alternative explanations via a 4-level intervention hierarchy.

4. **Task-specific scenarios**: Each scenario has real statistical artifacts (Simpson's paradox, MNAR, survivorship bias) that make the analytical choice genuinely consequential.

---

## File Structure

```
sensitive-attribute-circuit-analysis/
├── README.md                     # This file
├── requirements.txt              # Dependencies
├── setup.py                      # Package setup
├── run_experiment.py             # Main experiment runner
│
├── scenarios/                    # Experimental scenario definitions
│   ├── __init__.py
│   ├── base_scenario.py          # Base class for scenarios
│   └── scenario_registry.py     # All 10 scenarios
│
├── analysis/                     # Analysis modules
│   ├── __init__.py
│   ├── behavioral_probes.py      # Phase 1: Behavioral gating
│   ├── activation_collector.py   # Activation extraction
│   ├── circuit_analyzer.py       # Phase 3: Circuit discovery
│   ├── counterfactual_engine.py  # Phase 2: Counterfactual hierarchy
│   └── step_tracer.py           # Phase 4: Step-level reasoning
│
├── utils/                        # Utility functions
│   ├── __init__.py
│   └── model_utils.py           # Model loading, tokenization
│
├── configs/                      # Experiment configurations
│   ├── __init__.py
│   └── default_config.py        # Quick/Standard/Full presets
│
├── notebooks/                    # Analysis notebooks (coming soon)
└── results/                      # Output directory
```

---

## Metrics Glossary

| Metric | Formula | Interpretation |
|--------|---------|---------------|
| **ΔP** | P(misleading \| sensitive) - P(misleading \| control) | < 0 = anti-bias, > 0 = pro-bias |
| **Cohen's h** | 2(arcsin√p₁ - arcsin√p₂) | Effect size for proportion differences |
| **Logit Difference** | logit(misleading) - logit(correct) | Positive = model prefers misleading |
| **Patching Score** | (patched - corrupt) / (clean - corrupt) | 1 = full restoration, 0 = no effect |
| **IIA** | Mean(counterfactual matches causal model prediction) | For DAS/causal abstraction |
| **Flip Rate** | Mean(ablated_pred ≠ original_pred) | For step necessity |

---

## License

MIT

## Citation

```bibtex
@software{sensitive_attribute_circuit,
    title={Sensitive-Attribute Circuit Analysis: Mechanistic Interpretability for Analytical Bias in LLMs},
    author={ryanyen22},
    year={2025},
    url={https://huggingface.co/ryanyen22/sensitive-attribute-circuit-analysis}
}
```