CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

fev-bench Leaderboard is a Streamlit web application displaying time series forecasting model evaluation results from the fev-bench benchmark. It evaluates 30+ forecasting models using multiple metrics (SQL, MASE, WQL, WAPE) across 100 benchmark tasks.

Common Commands

# Run the Streamlit app locally
uv run streamlit run fev-leaderboard-app.py --server.port=8501 --server.address=0.0.0.0

# Regenerate leaderboard tables from autogluon/fev repo (defaults to main branch)
uv run python save_tables.py [commit]  # e.g., uv run python save_tables.py abc123

# Docker build and run
docker build -t fev-leaderboard .
docker run -p 8501:8501 fev-leaderboard

Note: Use uv run prefix for all Python commands in this project.

No test or lint frameworks are configured.

Architecture

fev-leaderboard-app.py     # Main entry point (Streamlit multi-page router)
save_tables.py             # Generates pre-computed CSV tables from raw summaries
pages/
├── fev_bench.py           # Main leaderboard (100 tasks, loads from tables/)
├── chronos_bench_ii.py    # Alternative leaderboard (27 tasks, fetches from GitHub)
└── about.py               # Help page with links
src/
├── utils.py               # Visualization, formatting, MODEL_CONFIG, color palette
├── strings.py             # UI text, metric descriptions, paper citations
└── task_groups.py         # Task groupings by frequency and domain
tables/                    # Pre-generated CSVs
├── pivot_*.csv            # Full pivot tables (filtered in app by task group)
├── summaries.csv          # Raw evaluation summaries
└── {group}/               # Subdirectories for each task group (full, mini, frequency_*, domain_*)
    ├── leaderboard_*.csv  # Leaderboard tables per metric
    └── pairwise_*.csv     # Pairwise comparison tables per metric

Data flow: GitHub (autogluon/fev) → save_tables.py → pre-computed tables → fev_bench.py visualization

Key Modules

src/utils.py: Core module containing:

• MODEL_CONFIG: Dict mapping model names to (huggingface_url, organization, is_zero_shot, model_type)
• ALL_METRICS: Dict with SQL, MASE, WQL, WAPE definitions
• format_leaderboard(), construct_bar_chart(), construct_pairwise_chart(), construct_pivot_table(): Styling functions
• COLORS: Custom palette (purple, gold, silver, bronze)

src/strings.py: Documentation strings for metric formulas, win rate/skill score calculations, imputation strategies

Metrics

Metric	Type	Description
SQL	Probabilistic	Scaled Quantile Loss (scale-invariant)
MASE	Point	Mean Absolute Scaled Error (scale-invariant)
WQL	Probabilistic	Weighted Quantile Loss (scale-dependent)
WAPE	Point	Weighted Absolute Percentage Error (scale-dependent)

Model Types

Models are categorized as DL (deep learning) or ST (statistical) in MODEL_CONFIG. This affects color-coding in visualizations (blue vs. orange).

Imputation Strategy

• Failed tasks: Replaced with Seasonal Naive scores
• Leaky tasks (training corpus overlap for zero-shot models): Replaced with Chronos-Bolt scores

External References

• fev-bench paper: https://arxiv.org/abs/2509.26468
• fev library docs: https://autogluon.github.io/fev/latest/
• GitHub: https://github.com/autogluon/fev