CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

fev-bench Leaderboard is a Streamlit web application displaying time series forecasting model evaluation results from the fev-bench benchmark. It evaluates 30+ forecasting models using multiple metrics (SQL, MASE, WQL, WAPE) across 100 benchmark tasks.

## Common Commands

```bash
# Run the Streamlit app locally
uv run streamlit run fev-leaderboard-app.py --server.port=8501 --server.address=0.0.0.0

# Regenerate leaderboard tables from autogluon/fev repo (defaults to main branch)
uv run python save_tables.py [commit]  # e.g., uv run python save_tables.py abc123

# Docker build and run
docker build -t fev-leaderboard .
docker run -p 8501:8501 fev-leaderboard
```

Note: Use `uv run` prefix for all Python commands in this project.

No test or lint frameworks are configured.

## Architecture

```
fev-leaderboard-app.py     # Main entry point (Streamlit multi-page router)
save_tables.py             # Generates pre-computed CSV tables from raw summaries
pages/
├── fev_bench.py           # Main leaderboard (100 tasks, loads from tables/)
├── chronos_bench_ii.py    # Alternative leaderboard (27 tasks, fetches from GitHub)
└── about.py               # Help page with links
src/
├── utils.py               # Visualization, formatting, MODEL_CONFIG, color palette
├── strings.py             # UI text, metric descriptions, paper citations
└── task_groups.py         # Task groupings by frequency and domain
tables/                    # Pre-generated CSVs
├── pivot_*.csv            # Full pivot tables (filtered in app by task group)
├── summaries.csv          # Raw evaluation summaries
└── {group}/               # Subdirectories for each task group (full, mini, frequency_*, domain_*)
    ├── leaderboard_*.csv  # Leaderboard tables per metric
    └── pairwise_*.csv     # Pairwise comparison tables per metric
```

**Data flow**: GitHub (autogluon/fev) → `save_tables.py` → pre-computed tables → `fev_bench.py` visualization

## Key Modules

**`src/utils.py`**: Core module containing:
- `MODEL_CONFIG`: Dict mapping model names to (huggingface_url, organization, is_zero_shot, model_type)
- `ALL_METRICS`: Dict with SQL, MASE, WQL, WAPE definitions
- `format_leaderboard()`, `construct_bar_chart()`, `construct_pairwise_chart()`, `construct_pivot_table()`: Styling functions
- `COLORS`: Custom palette (purple, gold, silver, bronze)

**`src/strings.py`**: Documentation strings for metric formulas, win rate/skill score calculations, imputation strategies

## Metrics

| Metric | Type | Description |
|--------|------|-------------|
| SQL | Probabilistic | Scaled Quantile Loss (scale-invariant) |
| MASE | Point | Mean Absolute Scaled Error (scale-invariant) |
| WQL | Probabilistic | Weighted Quantile Loss (scale-dependent) |
| WAPE | Point | Weighted Absolute Percentage Error (scale-dependent) |

## Model Types

Models are categorized as DL (deep learning) or ST (statistical) in `MODEL_CONFIG`. This affects color-coding in visualizations (blue vs. orange).

## Imputation Strategy

- **Failed tasks**: Replaced with Seasonal Naive scores
- **Leaky tasks** (training corpus overlap for zero-shot models): Replaced with Chronos-Bolt scores

## External References

- fev-bench paper: https://arxiv.org/abs/2509.26468
- fev library docs: https://autogluon.github.io/fev/latest/
- GitHub: https://github.com/autogluon/fev