You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

SegTME-UNI2-UperHoVer — Stage 2 (TCGA-UT Scale 0)

Stage 2 of 3 in the SegTME-UNI2 progressive pseudo-label curriculum. Trained from scratch on entropy-filtered pseudo-labels generated by the Stage 1 model (M1, PanNuke) applied to 271,711 TCGA-UT scale-0 patches (0.5 µm/px). This model generates pseudo-labels for Stage 3.

Part of the SegTME-UNI2 framework — Segmentation of Tumour Microenvironment with UNI2, targeting end-to-end cell segmentation, TME feature extraction, and clinical narrative generation from routine H&E histology. Submitted to Medical Image Analysis (Elsevier).


Framework Overview

SegTME-UNI2 addresses the core bottleneck in computational TME analysis: the gap between the scale of available TCGA H&E image data (1,608,061 patches) and the scale of human-annotated pixel-level labels (7,901 PanNuke images, 189,744 annotated nuclei). The solution is a three-stage progressive pseudo-label curriculum that closes this gap without additional manual annotation, staging domain expansion across resolution scales.

Stage 1: PanNuke (7,901 patches, human GT) → M1
     ↓ M1 infers on TCGA-UT Scale 0 → entropy-filtered pseudo-labels
Stage 2: TCGA-UT Scale 0 (271,711 patches, pseudo-labels) → M2  ← this model
     ↓ M2 infers on TCGA-UT Scales 0–5 → entropy-filtered pseudo-labels
Stage 3: TCGA-UT Scales 0–5 (1,608,061 patches, pseudo-labels) → M3

Critical design principle: Each stage trains a completely independent model from scratch (UNI2-h pretrained backbone + randomly initialised decoders). No weights are transferred between stages — improvement is driven entirely by increasing pseudo-label quality.

Why Scale 0 first? At 0.5 µm/px, Scale 0 is closest to PanNuke's 0.25 µm/px resolution. The Stage 1 model produces its highest-quality pseudo-labels here, giving Stage 2 the most reliable training signal before expanding to coarser scales.

Pseudo-label generation: M1 runs inference on all 271,711 scale-0 patches. Per-pixel entropy H(p) = −Σ p̂_c·log p̂_c is computed; patches are ranked by mean foreground entropy H̄. Only the top-70% most confident patches (H̄ < τ, τ = 70th percentile) are retained as training data.


Architecture — UNI2-UperHoVer

A novel dual-head segmentation model:

Input (3 × 224 × 224)
     ↓
UNI2-h ViT-Giant backbone (pretrained on 100M+ histopathology tiles from 100,000 slides)
     ↓ multi-scale FPN taps at transformer blocks 5, 11, 17, 23
1×1 projection layers → feature pyramid at 4 scales
     ↓
UperNet Decoder (PPM + FPN fusion)
     ├─→ Semantic Head (6-class)        → semantic segmentation map
     └─→ HV Regression Head (2-channel) → horizontal/vertical gradient maps
                                            → watershed-based nuclear instance separation

UNI2-h backbone specs:

  • Architecture: ViT-Giant (patch size 14 px, embedding dim d = 1536, 24 transformer blocks, 24 attention heads, SwiGLU-packed MLP, 8 register tokens)
  • Pretrained on: >100 million tiles from 100,000 whole-slide images, 40+ cancer types
  • Feature pyramid: blocks {5, 11, 17, 23} → channels {256, 512, 1024, 2048} at strides {s/4, s/8, s/16, s/32}

Dual decoder heads (parameters not shared between heads):

  • Semantic head: num_labels=6, hidden_size=768 → 6-class per-pixel output
  • HV regression head: num_labels=2, hidden_size=768 → horizontal (ch 0) and vertical (ch 1) gradient maps

Loss function:

L_total = L_sem + λ · L_hv

L_sem  = cross-entropy over valid pixels (ignore_index=255)
L_hv   = L_MSE + 2 · L_MSGE   (foreground pixels only)
λ      = 1.0

Dynamic HV target synthesis: HV maps are generated on-the-fly from semantic labels using connected-component labelling and per-nucleus centroid computation — no instance-level annotations are required at any stage.


Cell Classes (PanNuke Ontology)

Class Label Biological meaning
0 Background Non-cellular tissue / void
1 Neoplastic Tumour nuclei
2 Inflammatory Immune cells (lymphocytes, neutrophils, etc.)
3 Connective Stromal / connective tissue nuclei
4 Dead Necrotic / apoptotic nuclei
5 Non-neoplastic Epithelial Normal epithelial nuclei

Training Configuration — Stage 2

Hyperparameter Value
Training dataset TCGA-UT Scale 0 — 271,711 patches, 32 solid cancer types, 0.5 µm/px
Labels Entropy-filtered pseudo-labels from Stage 1 (M1, PanNuke model)
Training split 80% train, 20% pseudo-label validation (PL-val)
Input resolution 256×256 px at 0.5 µm/px → resized to 224×224 for backbone
Backbone init UNI2-h pretrained checkpoint — fresh init, no weights from Stage 1
Decoder init Randomly initialised (both heads)
Optimiser AdamW (β₁=0.9, β₂=0.999, weight_decay=1×10⁻²)
Learning rate 5×10⁻⁵ — linear decay: LR(t) = 5×10⁻⁵ × (T−t)/T (no warmup)
Per-device batch size 8
Gradient accumulation 4 steps (effective batch = 32 per GPU)
Number of GPUs 8 × NVIDIA A100 (DDP)
Training epochs 250
Total optimizer steps 212,500
Mixed precision bfloat16
Compilation torch.compile (inductor backend)
Eval frequency every 500 steps
Checkpoint metric Validation mean IoU (↑)
Entropy threshold τ 70th percentile of per-patch mean entropy H̄

Augmentation (each with p=0.5): colour jitter (brightness/contrast/saturation ±20%, hue ±5%), HLS-space multiplicative perturbation ∈[0.9, 1.1], horizontal flip, vertical flip.


Results

Checkpoint Epoch Step Val mean IoU Eval loss
0.6 500 0.471 0.307
25 21,500 0.658 0.145
50 42,500 0.708 0.112
100 84,500 0.752 0.089
150 127,500 0.782 0.077
200 170,000 0.810 0.066
best 250 212,500 0.8197 0.061

Evaluation protocol: mIoU values in this table are the training-time checkpoint-selection mIoU: macro-averaged Jaccard across all 6 classes (Background included) on the TCGA-UT scale-0 20% pseudo-label validation split (PL-val). This measures self-consistency on model-generated labels, not accuracy against human ground truth. The initial mIoU of 0.47 (vs. 0.41 for Stage 1) reflects the higher quality of pseudo-label initialisation compared to the random decoder alone. Best checkpoint is the final checkpoint — the model was still improving at end of training.

Paper Table 7 values (5-class, Background excluded):

  • Part A (cross-domain, PanNuke 20% held-out GT): M2 = 0.3292 — lower scores reflect domain shift from TCGA-UT pseudo-label training away from PanNuke, not model degradation.
  • Part B (PL-val self-consistency, TCGA-UT s0 held-out): M2 = 0.3666 (compare: M1 on same partition = 0.3631, confirming M2 outperforms M1 on TCGA-UT domain).

Inference

# MPP normalisation before inference
scale = mpp_input / 0.314   # target ~0.35 µm/px

# Tiling
# Tile 224×224 with 50% overlap (stride 112 px); zero-pad to multiple of 14
# Stitch semantic map (foreground-priority) and HV map (last-write)

# Watershed instance separation
# Energy = HV magnitude; EDT seeds; compactness=0.01

The model expects 224×224 RGB patches normalised with ImageNet statistics (μ=(0.485, 0.456, 0.406), σ=(0.229, 0.224, 0.225)).


Repository Contents

File Description
model.safetensors Model weights — UNI2-h backbone + both decoder heads (~3.2 GB)
trainer_state.json Full training log: per-step mIoU, loss, learning rate
training_args.bin HuggingFace Trainer configuration

Related Models in This Series

Model Stage Training data Best val mIoU Eval basis
PanNuke 1 PanNuke — 7,901 patches (GT) 0.9313 Human GT
This model 2 TCGA-UT Scale 0 — 271K patches (PL) 0.8197 PL-val†
TCGA-UT-012345 3 TCGA-UT Scales 0–5 — 1.6M patches (PL) 0.7724 PL-val†

†PL-val = pseudo-label validation (self-consistency on model-generated labels, not human GT). Cross-domain evaluation on PanNuke GT for M2/M3 is pending (requires dedicated inference run).


Data Availability


Citation

This model is part of the SegTME-UNI2 framework (manuscript under review). If you use this work, please cite the SegTME-UNI2 paper. Please also cite the UNI2 foundation model, PanNuke dataset, and TCGA-UT dataset:

@article{wanahmad2026segtme,
  title={SegTME-UNI2: A Foundation Model-Based Framework for Generalisable Multiclass Cell Segmentation and LLM-Driven Tumour Microenvironment Characterisation in Histopathology},
  author={Wan Ahmad, Wan Siti Halimatul Munirah and Samidi, Faris Syahmi and Ahmmed, Mohammad Badal and Thiviyanathan, Vimal Angela and Thavaraj, Selvam James and Abdul Majeed, Anwar P.P.},
  journal={arXiv preprint arXiv:2606.17702},
  year={2026},
  doi={10.48550/arXiv.2606.17702}
}
@article{chen2024uni2,
  title={UNI2: Towards a Universal Whole-Slide Foundation Model for Pathology},
  author={Chen, Richard J and others},
  journal={arXiv preprint arXiv:2406.01647},
  year={2024},
  doi={10.48550/arXiv.2406.01647}
}
@article{komura2022tcgaut,
  title={Universal encoding of pan-cancer histology by deep texture representations},
  author={Komura, Daisuke and others},
  journal={Cell Reports},
  volume={38},
  pages={110424},
  year={2022}
}
@article{gamper2020pannuke,
  title={PanNuke Dataset Extension, Insights and Baselines},
  author={Gamper, Jevgenij and others},
  year={2020}
}
Downloads last month
4
Safetensors
Model size
0.8B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train mizjaggy18/SegTME-UNI2-UperHoVer_TCGA-UT-0

Papers for mizjaggy18/SegTME-UNI2-UperHoVer_TCGA-UT-0