SegTME-UNI2-UperHoVer — Stage 2 (TCGA-UT Scale 0)
Stage 2 of 3 in the SegTME-UNI2 progressive pseudo-label curriculum. Trained from scratch on entropy-filtered pseudo-labels generated by the Stage 1 model (M1, PanNuke) applied to 271,711 TCGA-UT scale-0 patches (0.5 µm/px). This model generates pseudo-labels for Stage 3.
Part of the SegTME-UNI2 framework — Segmentation of Tumour Microenvironment with UNI2, targeting end-to-end cell segmentation, TME feature extraction, and clinical narrative generation from routine H&E histology. Submitted to Medical Image Analysis (Elsevier).
Framework Overview
SegTME-UNI2 addresses the core bottleneck in computational TME analysis: the gap between the scale of available TCGA H&E image data (1,608,061 patches) and the scale of human-annotated pixel-level labels (7,901 PanNuke images, 189,744 annotated nuclei). The solution is a three-stage progressive pseudo-label curriculum that closes this gap without additional manual annotation, staging domain expansion across resolution scales.
Stage 1: PanNuke (7,901 patches, human GT) → M1
↓ M1 infers on TCGA-UT Scale 0 → entropy-filtered pseudo-labels
Stage 2: TCGA-UT Scale 0 (271,711 patches, pseudo-labels) → M2 ← this model
↓ M2 infers on TCGA-UT Scales 0–5 → entropy-filtered pseudo-labels
Stage 3: TCGA-UT Scales 0–5 (1,608,061 patches, pseudo-labels) → M3
Critical design principle: Each stage trains a completely independent model from scratch (UNI2-h pretrained backbone + randomly initialised decoders). No weights are transferred between stages — improvement is driven entirely by increasing pseudo-label quality.
Why Scale 0 first? At 0.5 µm/px, Scale 0 is closest to PanNuke's 0.25 µm/px resolution. The Stage 1 model produces its highest-quality pseudo-labels here, giving Stage 2 the most reliable training signal before expanding to coarser scales.
Pseudo-label generation: M1 runs inference on all 271,711 scale-0 patches. Per-pixel entropy H(p) = −Σ p̂_c·log p̂_c is computed; patches are ranked by mean foreground entropy H̄. Only the top-70% most confident patches (H̄ < τ, τ = 70th percentile) are retained as training data.
Architecture — UNI2-UperHoVer
A novel dual-head segmentation model:
Input (3 × 224 × 224)
↓
UNI2-h ViT-Giant backbone (pretrained on 100M+ histopathology tiles from 100,000 slides)
↓ multi-scale FPN taps at transformer blocks 5, 11, 17, 23
1×1 projection layers → feature pyramid at 4 scales
↓
UperNet Decoder (PPM + FPN fusion)
├─→ Semantic Head (6-class) → semantic segmentation map
└─→ HV Regression Head (2-channel) → horizontal/vertical gradient maps
→ watershed-based nuclear instance separation
UNI2-h backbone specs:
- Architecture: ViT-Giant (patch size 14 px, embedding dim d = 1536, 24 transformer blocks, 24 attention heads, SwiGLU-packed MLP, 8 register tokens)
- Pretrained on: >100 million tiles from 100,000 whole-slide images, 40+ cancer types
- Feature pyramid: blocks {5, 11, 17, 23} → channels {256, 512, 1024, 2048} at strides {s/4, s/8, s/16, s/32}
Dual decoder heads (parameters not shared between heads):
- Semantic head:
num_labels=6,hidden_size=768→ 6-class per-pixel output - HV regression head:
num_labels=2,hidden_size=768→ horizontal (ch 0) and vertical (ch 1) gradient maps
Loss function:
L_total = L_sem + λ · L_hv
L_sem = cross-entropy over valid pixels (ignore_index=255)
L_hv = L_MSE + 2 · L_MSGE (foreground pixels only)
λ = 1.0
Dynamic HV target synthesis: HV maps are generated on-the-fly from semantic labels using connected-component labelling and per-nucleus centroid computation — no instance-level annotations are required at any stage.
Cell Classes (PanNuke Ontology)
| Class | Label | Biological meaning |
|---|---|---|
| 0 | Background | Non-cellular tissue / void |
| 1 | Neoplastic | Tumour nuclei |
| 2 | Inflammatory | Immune cells (lymphocytes, neutrophils, etc.) |
| 3 | Connective | Stromal / connective tissue nuclei |
| 4 | Dead | Necrotic / apoptotic nuclei |
| 5 | Non-neoplastic Epithelial | Normal epithelial nuclei |
Training Configuration — Stage 2
| Hyperparameter | Value |
|---|---|
| Training dataset | TCGA-UT Scale 0 — 271,711 patches, 32 solid cancer types, 0.5 µm/px |
| Labels | Entropy-filtered pseudo-labels from Stage 1 (M1, PanNuke model) |
| Training split | 80% train, 20% pseudo-label validation (PL-val) |
| Input resolution | 256×256 px at 0.5 µm/px → resized to 224×224 for backbone |
| Backbone init | UNI2-h pretrained checkpoint — fresh init, no weights from Stage 1 |
| Decoder init | Randomly initialised (both heads) |
| Optimiser | AdamW (β₁=0.9, β₂=0.999, weight_decay=1×10⁻²) |
| Learning rate | 5×10⁻⁵ — linear decay: LR(t) = 5×10⁻⁵ × (T−t)/T (no warmup) |
| Per-device batch size | 8 |
| Gradient accumulation | 4 steps (effective batch = 32 per GPU) |
| Number of GPUs | 8 × NVIDIA A100 (DDP) |
| Training epochs | 250 |
| Total optimizer steps | 212,500 |
| Mixed precision | bfloat16 |
| Compilation | torch.compile (inductor backend) |
| Eval frequency | every 500 steps |
| Checkpoint metric | Validation mean IoU (↑) |
| Entropy threshold τ | 70th percentile of per-patch mean entropy H̄ |
Augmentation (each with p=0.5): colour jitter (brightness/contrast/saturation ±20%, hue ±5%), HLS-space multiplicative perturbation ∈[0.9, 1.1], horizontal flip, vertical flip.
Results
| Checkpoint | Epoch | Step | Val mean IoU | Eval loss |
|---|---|---|---|---|
| 0.6 | 500 | 0.471 | 0.307 | |
| 25 | 21,500 | 0.658 | 0.145 | |
| 50 | 42,500 | 0.708 | 0.112 | |
| 100 | 84,500 | 0.752 | 0.089 | |
| 150 | 127,500 | 0.782 | 0.077 | |
| 200 | 170,000 | 0.810 | 0.066 | |
| best | 250 | 212,500 | 0.8197 | 0.061 |
Evaluation protocol: mIoU values in this table are the training-time checkpoint-selection mIoU: macro-averaged Jaccard across all 6 classes (Background included) on the TCGA-UT scale-0 20% pseudo-label validation split (PL-val). This measures self-consistency on model-generated labels, not accuracy against human ground truth. The initial mIoU of 0.47 (vs. 0.41 for Stage 1) reflects the higher quality of pseudo-label initialisation compared to the random decoder alone. Best checkpoint is the final checkpoint — the model was still improving at end of training.
Paper Table 7 values (5-class, Background excluded):
- Part A (cross-domain, PanNuke 20% held-out GT): M2 = 0.3292 — lower scores reflect domain shift from TCGA-UT pseudo-label training away from PanNuke, not model degradation.
- Part B (PL-val self-consistency, TCGA-UT s0 held-out): M2 = 0.3666 (compare: M1 on same partition = 0.3631, confirming M2 outperforms M1 on TCGA-UT domain).
Inference
# MPP normalisation before inference
scale = mpp_input / 0.314 # target ~0.35 µm/px
# Tiling
# Tile 224×224 with 50% overlap (stride 112 px); zero-pad to multiple of 14
# Stitch semantic map (foreground-priority) and HV map (last-write)
# Watershed instance separation
# Energy = HV magnitude; EDT seeds; compactness=0.01
The model expects 224×224 RGB patches normalised with ImageNet statistics (μ=(0.485, 0.456, 0.406), σ=(0.229, 0.224, 0.225)).
Repository Contents
| File | Description |
|---|---|
model.safetensors |
Model weights — UNI2-h backbone + both decoder heads (~3.2 GB) |
trainer_state.json |
Full training log: per-step mIoU, loss, learning rate |
training_args.bin |
HuggingFace Trainer configuration |
Related Models in This Series
| Model | Stage | Training data | Best val mIoU | Eval basis |
|---|---|---|---|---|
| PanNuke | 1 | PanNuke — 7,901 patches (GT) | 0.9313 | Human GT |
| This model | 2 | TCGA-UT Scale 0 — 271K patches (PL) | 0.8197 | PL-val† |
| TCGA-UT-012345 | 3 | TCGA-UT Scales 0–5 — 1.6M patches (PL) | 0.7724 | PL-val† |
†PL-val = pseudo-label validation (self-consistency on model-generated labels, not human GT). Cross-domain evaluation on PanNuke GT for M2/M3 is pending (requires dedicated inference run).
Data Availability
- TCGA data: https://portal.gdc.cancer.gov/
- PanNuke dataset: https://warwick.ac.uk/fac/cross_fac/tia/data/pannuke
- UNI2-h backbone: https://huggingface.co/MahmoodLab/uni2-h
- Pseudo-labelled TCGA-UT dataset: https://huggingface.co/datasets/mizjaggy18/tcga-ut-cell-instance-semantic
- Training code: to be released upon paper acceptance
Citation
This model is part of the SegTME-UNI2 framework (manuscript under review). If you use this work, please cite the SegTME-UNI2 paper. Please also cite the UNI2 foundation model, PanNuke dataset, and TCGA-UT dataset:
@article{wanahmad2026segtme,
title={SegTME-UNI2: A Foundation Model-Based Framework for Generalisable Multiclass Cell Segmentation and LLM-Driven Tumour Microenvironment Characterisation in Histopathology},
author={Wan Ahmad, Wan Siti Halimatul Munirah and Samidi, Faris Syahmi and Ahmmed, Mohammad Badal and Thiviyanathan, Vimal Angela and Thavaraj, Selvam James and Abdul Majeed, Anwar P.P.},
journal={arXiv preprint arXiv:2606.17702},
year={2026},
doi={10.48550/arXiv.2606.17702}
}
@article{chen2024uni2,
title={UNI2: Towards a Universal Whole-Slide Foundation Model for Pathology},
author={Chen, Richard J and others},
journal={arXiv preprint arXiv:2406.01647},
year={2024},
doi={10.48550/arXiv.2406.01647}
}
@article{komura2022tcgaut,
title={Universal encoding of pan-cancer histology by deep texture representations},
author={Komura, Daisuke and others},
journal={Cell Reports},
volume={38},
pages={110424},
year={2022}
}
@article{gamper2020pannuke,
title={PanNuke Dataset Extension, Insights and Baselines},
author={Gamper, Jevgenij and others},
year={2020}
}
- Downloads last month
- 4