CellPrior-Net: Prior-Guided Nuclei Detection and Classification for H&E Whole-Slide Images
CellPrior-Net (CP-Net) introduces an efficient, prior-guided CNN architecture for dense nuclei detection and 3-class classification (tumor, immune, other) on H&E gigapixel whole-slide images. By coupl
Summary
Accurate single-cell quantification in hematoxylin and eosin (H&E) whole-slide images (WSIs) is foundational for deciphering the tumor microenvironment (TME) and calculating quantitative biomarkers such as tumor-infiltrating lymphocyte (TIL) density. However, existing computational pathology pipelines face three major clinical translation bottlenecks:
The Representation Bottleneck: Standard deep learning networks ingest standard 3-channel RGB patches, ignoring the biochemical reality that hematoxylin stains nucleic acids (DNA/RNA) specifically, while eosin stains non-specific cytoplasmic and extracellular proteins.
The Compute & Latency Bottleneck: Modern vision transformer pipelines (e.g., CellViT, CellDETR) and heavy multi-branch networks require massive compute and slow CPU-bound post-processing, taking 10 to 40+ minutes per gigapixel WSI.
The Artifact Dilemma: Clinical WSIs are rife with tissue folds, knife chatter, air bubbles, out-of-focus blur, and pathologist pen ink. Processing uncurated tiles generates massive false-positive cell counts.
Published in the Journal of Pathology Informatics (August 2026, DOI: 10.1016/j.jpi.2026.100716; arXiv:2607.00802), CellPrior-Net (CP-Net) and its parent clinical workflow [[CellQuant-Net]] resolve these challenges:
Prior-Guided 4-Channel Input: Separates the hematoxylin ($H$) stain via Macenko optical density deconvolution, applies a multi-scale Difference of Gaussians (DoG) filter to capture nuclear chromatin density, and feeds a 4-channel tensor ($[R, G, B, \text{DoG}]$) into the network.
UniRepLKNet-N Large-Kernel Encoder: Replaces heavy transformer self-attention with large-kernel depthwise convolutions, delivering a vast receptive field at fractional computational overhead.
GPU-Accelerated Watershed Post-Processing: Combines GPU erosion, Euclidean distance transforms, and multithreaded watershed with magnification-optimized hyperparameters (calibrated for $20\times$ and $40\times$).
End-to-End WSI Pipeline ([[CellQuant-Net]]): Integrates automated deep-learning Quality Assessment (WSI-QA) to eliminate artifacts prior to inference, extracts nuclei types, builds spatial cell-cell connectivity graphs, and exports native QuPath GeoJSON files.
Rigorous Cross-Dataset Benchmark (~10.4M Nuclei across 8 Datasets): Evaluated against 10 state-of-the-art pipelines (CellViT, CellViT++, Cerberus, HoVer-Net, NuLite, PointNu-Net, CellDetr, CellSAM, LKCell, TSFD-Net) across multiple organs, scanners, and magnifications. CP-Net establishes the optimal Pareto efficiency frontier, matching CellViT panoptic quality while cutting inference time by $2\times$ to $3\times$.
Technical Architecture & Mathematical Formulation
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CELLPRIOR-NET (CP-NET) PIPELINE │
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[ Input H&E Tile (256 × 256 px) ]
│
┌─────────────┴────────────────────────┐
│ │
│ (3-channel RGB) ▼
│ ┌─────────────────────────────────┐
│ │ Macenko Stain Deconvolution │
│ │ OD Conversion & Eigenplane │
│ └───────────────┬─────────────────┘
│ │
│ ▼
│ ┌─────────────────────────────────┐
│ │ Isolated Hematoxylin (H) │
│ │ 99th Percentile Normalization │
│ └───────────────┬─────────────────┘
│ │
│ ▼
│ ┌─────────────────────────────────┐
│ │ Multi-Scale DoG Filter │
│ │ σ ∈ {2.5, 3.5, 4.5} (40×) │
│ │ σ ∈ {1.6, 2.4, 3.2} (20×) │
│ └───────────────┬─────────────────┘
│ │
│ ▼
│ ┌─────────────────────────────────┐
│ │ DoG Nuclear Prior Map │
│ └───────────────┬─────────────────┘
│ │
└──────────────────────┬───────────────┘
│ Concatenate [R, G, B, DoG]
▼
┌─────────────────────────────────────────────────────────────────┐
│ 4-CHANNEL INPUT TENSOR [B, 4, H, W] │
└───────────────────────────────┬─────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ UniRepLKNet-N LARGE-KERNEL CNN BACKBONE │
│ • Stage 1: Downsampling & Large-Kernel Depthwise Conv │
│ • Stage 2: Hierarchical Spatial Feature Extraction │
│ • Stage 3: High-Dimensional Morphological Encoding │
│ • Stage 4: Global Context Integration │
└───────────────────────────────┬─────────────────────────────────┘
│ Multi-Scale Skip Connections
▼
┌─────────────────────────────────────────────────────────────────┐
│ U-NET-LIKE PROGRESSIVE FEATURE DECODER │
│ Feature concatenation & progressive spatial upsampling │
└───────────────┬─────────────────────────────────┬───────────────┘
│ │
▼ ▼
┌───────────────────────────────┐ ┌───────────────────────────────┐
│ Binary Detection Head │ │ 3-Class Classification Head │
│ Nuclei vs Background │ │ Tumor vs Immune vs Other │
│ Loss: Dice + FocalTversky │ │ Loss: CE + Dice + FocalTvers │
└───────────────┬───────────────┘ └───────────────┬───────────────┘
│ │
└───────────────┬─────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ MAGNIFICATION-CALIBRATED GPU POST-PROCESSING │
│ 1. Morphological Erosion (re) on binary logits │
│ 2. Euclidean Distance Transform (sd) │
│ 3. Peak Local Maxima Detection (dm, t_min) │
│ 4. GPU/Multithreaded Watershed Boundary Separation │
│ 5. Small Object Noise Filter (A_min) │
│ 6. Instance Segmentation & Phenotypic Class Fusion │
└───────────────────────────────┬─────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ FINAL INSTANCE CELL SEGMENTATION │
│ • GeoJSON boundaries for QuPath │
│ • Phenotype: Tumor / Immune / Other │
│ • Spatial Connectivity Graph │
└─────────────────────────────────────────────────────────────────┘1. Stain Deconvolution & The Hematoxylin Difference of Gaussians (DoG) Prior
Rather than forcing the neural network to infer stain chemistry from standard RGB values, CP-Net extracts an explicit nuclear prior channel through analytical stain separation:
Optical Density (OD) Transformation: where $I_0 = 240$ represents the background transmitted light intensity, and the constant $1$ avoids logarithmic singularity on saturated pixels. Pixels with low optical density ($\text{OD} < \beta = 0.15$) are excluded as transparent background.
Eigenvector Plane & Percentile Angular Projection: The covariance matrix $\Sigma = \text{Cov}(\text{OD}{\text{hat}}^T)$ is decomposed into eigenvalues and eigenvectors. Pixels are projected onto the primary stain plane, and angular coordinates $\phi = \arctan2(T_y, T_x)$ are calculated. The 1st and 99th percentiles of $\phi$ establish the robust boundary vectors for Hematoxylin ($v{\text{min}}$) and Eosin ($v_{\text{max}}$).
Stain Matrix Decomposition & Normalization: Stain concentrations $C$ are derived via least-squares minimization: The first concentration row $C_H$ represents hematoxylin. It is normalized against the 99th percentile: where $C_{H,\text{ref}} = 1.9705$ is the standardized reference maximum hematoxylin concentration.
Multi-Scale Difference of Gaussians (DoG): The grayscale hematoxylin image $I_H$ is convolved with 2D Gaussian kernels across smoothing scales $\sigma$: with scale multiplier $k = \sqrt{2}$. The final nuclear prior map takes the maximum DoG response across a calibrated scale set $\Sigma$:
$40\times$ Magnification ($0.25\ \mu\text{m/px}$): $\Sigma = {2.5, 3.5, 4.5}$
$20\times$ Magnification ($0.50\ \mu\text{m/px}$): $\Sigma = {1.6, 2.4, 3.2}$
The resulting single-channel DoG response is concatenated with the RGB image, forming a 4-channel input tensor $[B, 4, 256, 256]$ that strongly highlights chromatin texture while suppressing eosinophilic stroma and background void.
2. Neural Architecture: UniRepLKNet-N Backbone & Dual Decoder
Existing pipelines either deploy standard CNNs (HoVer-Net) that lack sufficient contextual receptive fields, or vision transformers (CellViT, CellDETR) that incur quadratic memory overheads ($\mathcal{O}(N^2)$) and high FLOP counts:
Encoder: CP-Net uses UniRepLKNet-N, a modern convolutional backbone that features large-kernel depthwise convolutions (e.g., $13\times 13$ to $31\times 31$ kernels reparameterized into small kernels during inference). This provides an ultra-wide effective receptive field (ERF) matching ViTs while retaining convolutional translation equivariance, low memory consumption, and extreme throughput on standard GPUs.
Decoder: A symmetrical U-Net decoder with multi-scale skip connections merges high-level semantic abstractions with low-level spatial contours.
Dual Output Heads:
Binary Nuclei Detection Head: Predicts a single-channel segmentation logit separating nuclei from background stroma.
Nuclei Type Classification Head: Predicts a 4-channel tensor corresponding to background, tumor cells, immune cells, and other stromal/epithelial cells.
3. Multi-Task Objective Function
The network is trained end-to-end using a joint detection and classification loss:
Where the classification and detection objectives are formulated to overcome class imbalance across dense cellular populations:
Focal Tversky loss specifically penalizes false negatives (missed small immune cells) in dense clusters, preventing the network from biasing exclusively toward large, hyperchromatic tumor nuclei.
4. Magnification-Calibrated GPU Watershed Post-Processing
A notorious failure mode in instance segmentation is the merging of abutting nuclei in crowded tumor nests. Rather than relying on computationally sluggish CPU watershed or complex horizontal-vertical distance fields (HoVer-Net), CP-Net optimizes a 6-step GPU-accelerated morphological pipeline:
Morphological Erosion ($r_e$): Separates weakly connected borders and strips single-pixel boundary noise.
Euclidean Distance Transform ($s_d$): Maps distance from nuclear boundaries to centroids.
Peak Local Maxima ($d_m, t_{\text{min}}$): Extracts internal centroid seeds while filtering noise peaks.
Watershed Segmentation: Propagates boundaries from peak seeds along the inverted distance topography.
Small Object Filter ($A_{\text{min}}$): Discards fragments below minimum biological area.
Phenotypic Mask Fusion: Assigns the majority classified cell type within each watershed polygon.
Hyperparameter Optimization via Grid Search (200 Expert-Curated Masks)
The hyperparameters were formally optimized against 200 diverse ground-truth masks (100 at $20\times$, 100 at $40\times$) using an absolute count discrepancy objective:
$r_e$
Erosion radius
1
2
$s_d$
Distance mask kernel
$5 \times 5$
$5 \times 5$
$d_m$
Minimum peak distance
4
8
$t_{\text{min}}$
Peak threshold cutoff
2
2
$A_{\text{min}}$
Minimum object area (px)
10
20
The [[CellQuant-Net]] Clinical WSI Quantification Framework
To bridge the gap between tile-level algorithmic benchmarks and gigapixel clinical whole-slide workflows, the authors wrapped CP-Net into CellQuant-Net, an end-to-end four-stage pipeline:
Multi-Center Evaluation & Benchmark Datasets
The study assembled 8 diverse datasets encompassing ~10.4 million annotated nuclei across different organs, scanners, and resolutions:
PanNuke
Multi-organ (19 tissues)
$40\times$
Multi-scanner
Gamper et al., 2019 (Used for CP-Net training)
CoNSeP
Colorectal
$40\times$
Omnyx VL120
Graham et al., 2019
Lizard
Colorectal
$20\times$
Mixed
Graham et al., 2021 (7,429 tiles)
NuCLS
Breast
$20\times$
Mixed
Amgad et al., 2021
PanopTILs
Breast
$40\times$
Mixed
Saltz et al. / TCGA
SegPath
Multi-organ
$40\times$
Hamamatsu NanoZoomer S60
Ding et al., 2023
TNMI-20×
Non-Small Cell Lung
$20\times$
3DHistech Pannoramic Flash III
Rakaee et al., 2023
TNMI-40×
Non-Small Cell Lung
$40\times$
3DHistech Pannoramic Flash III
Rakaee et al., 2023
ILCD (New)
Hepatocellular Carcinoma
$40\times$
Imperial College London
Jabar et al., 2026 (40 HCC WSIs, pathologist annotated)
Unified 3-Class Taxonomy: Because raw datasets use incompatible annotation labels (ranging from 4 to 14 classes), all labels were standardized into Tumor, Immune, and Other cells, enabling the first equitable, cross-dataset head-to-head evaluation in the literature.
Empirical Results & Head-to-Head Comparisons
1. Cross-Dataset Generalization vs. 10 State-of-the-Art Pipelines
CP-Net was compared against 10 published pipelines: CellViT, CellViT++, Cerberus, HoVer-Net, NuLite, PointNu-Net, CellDetr, CellSAM (detection only), LKCell, and TSFD-Net:
Cross-Dataset Performance Spread: Across external test cohorts, average Panoptic Quality (PQ) ranged widely from 0.05 to 0.34, Detection Quality (DQ) from 0.07 to 0.46, and Segmentation Quality (SQ) from 0.22 to 0.62. This underscores severe domain shifts caused by scanner optical profiles, stain variations, and tissue types.
Top Performers: Across all external datasets, CP-Net, CellViT, and NuLite consistently achieved the highest average PQ and DQ scores.
Impact of Scanner Profiling: Performance was highest on the Omnyx VL120 scanner (CoNSeP) and lowest on heterogeneous, multi-center scanner sets, proving that optical transfer functions remain a major source of out-of-distribution dropoff.
Phenotypic Disparity: All models achieved significantly higher PQ on tumor cells (which are larger, hyperchromatic, and architecturally distinct) than on immune cells (small, hyper-dense, and easily blurred into stromal fibroblasts) or other stromal cells.
Resolution Sensitivity: Models evaluated at $40\times$ achieved systematically superior panoptic quality compared to $20\times$, reflecting finer nuclear membrane and chromatin texture visibility.
2. Runtime & Pareto Frontier Analysis
The study evaluated runtime across gigapixel WSIs on an NVIDIA GPU workstation:
Table 1: Real-World Whole-Slide Execution Time (Minutes)
WSI 1 ($20\times$)
$31,871 \times 25,199\ \text{px}$
5.15 min
2.16 min (2.4× faster)
5.77 min (QA: 2.67m; CP-Net: 2.16m; Post: 0.94m)
WSI 2 ($40\times$)
$44,125 \times 70,507\ \text{px}$
8.30 min
2.77 min (3.0× faster)
16.83 min (QA: 12.58m; CP-Net: 2.77m; Post: 1.48m)
Takeaway: While CellViT achieves marginal PQ improvements in certain isolated settings, CP-Net cuts whole-slide cell inference time by $58%$ to $67%$, making dense whole-slide single-cell analysis clinically practical on standard hardware.
3. Clinical Proof-of-Concept: Overall Survival in Hepatocellular Carcinoma
To demonstrate clinical validity, the authors applied CellQuant-Net to 13 H&E WSIs from the AB-real hepatocellular carcinoma (HCC) cohort with matched clinical outcomes:
Automated WSI-QA excluded pen markings, capsule tears, and blur.
CP-Net quantified tumor-infiltrating lymphocyte (TIL) density across viable tumor beds.
Patients stratified by median TIL density into High vs. Low TIL groups revealed a statistically significant survival benefit for high TIL infiltration ($p = 0.006$, log-rank test).
While preliminary, this confirms that CP-Net generates clinically and biologically meaningful prognostic readouts directly from routine H&E glass slides without expensive multiplex IHC or transcriptomics.
Comparison with Vault Cell Segmentation Engines
Feature / Model
CellPrior-Net (CP-Net)
[[HistoPLUS: Towards Comprehensive Cellular Characterisation of H&E Slides]]
[[NuClick]]
[[HoVer-NeXt]]
[[CytoFormer: A Molecularly Supervised Cell Foundation Model for Histopathology Cell Classification]]
Primary Focus
Fast WSI detection + 3-class classification
Pan-cancer 13-class instance segmentation
Interactive / prompted point-to-mask
Unprompted WSI nuclear instance segmentation
Molecularly supervised single-cell classification
Backbone
UniRepLKNet-N (Large-kernel CNN)
H0-mini (Distilled ViT, 86M)
Lightweight CNN / U-Net
NeXt-Backbone (ConvNeXt / ViT)
ViT-Giant (UNI2-h backbone, 1536-d)
Input Channels
4-channel: RGB + Hematoxylin DoG prior
3-channel RGB
5-channel: RGB + Target Inc + Neighbor Exc
3-channel RGB
$56 \times 56\ \mu\text{m}$ RGB crop ($224 \times 224$ px)
Stain Prior
Explicit Macenko $H$-channel DoG filter
Implicit foundation feature maps
None (Prompt clicks act as spatial priors)
Implicit HV distance fields
Spatial transcriptomics marker supervision
Cell Taxonomy
3 classes (Tumor, Immune, Other)
13 classes (Granular immune & stromal)
Generic nuclear boundary
5–6 generic classes
23 classes (7 pan-tissue + 16 organ-specific)
Post-Processing
Calibrated GPU watershed ($r_e, d_m, A_{\text{min}}$)
CellViT HV regression & post-processing
Gaussian exclusion thresholding
Energy-landscape horizontal-vertical watershed
Per-organ linear head routing
WSI Artifact QA
Yes (Built-in WSI-QA module)
External tiling / QuPath integration
Relies on user click placement
Relies on external tiling filter
Relies on upstream cell segmenter
Primary Role
High-throughput clinical WSI quantification
High-granularity TME phenotyping
Ground-truth dataset curation & active learning
Automated pan-cancer segmentation
High-precision single-cell molecular typing
Practical Implementation & QuPath Workflow
CellQuant-Net is implemented in PyTorch and available at Falah-Jabar-Rahim/CellQuant-Net:
1. Installation & Environment
2. End-to-End Whole-Slide Inference CLI
Place .svs or .ndpi slides in input/, download weights into CP-Net/weights/ and WSI_QA/pretrained_ckpt/, and execute:
Available pretrained checkpoints include:
PanNuke.pth: Multi-organ ($40\times$)SegPath.pth: Multi-organ ($40\times$)CoNSeP.pth: Single-organ Colon ($40\times$)Lizard.pth: Single-organ Colon ($20\times$)ILCD.pth: Single-organ Liver ($40\times$)NuCLS.pth: Single-organ Breast ($20\times$)PanopTILs.pth: Single-organ Breast ($40\times$)TNMI20x.pth/TNMI40x.pth: Single-organ Lung ($20\times$ / $40\times$)
3. QuPath Interactive Review
CellQuant-Net generates zero-friction visualization artifacts directly inside output/CP_Net/<WSI_NAME>/:
Open the analyzed whole-slide image in QuPath.
Drag and drop
qupath_cells.geojsoninto the QuPath window to render all classified cellular polygons (Tumor, Immune, Other).Drag and drop
qupath_cells_connectivity.geojsonto visualize the spatial cell-cell interaction graph.Inspect
WSI_Summary.xlsxandcell_type_stats.csvfor summary infiltration metrics.
Vault Navigation & Related Notes
Dedicated Tool Profile: [[CellQuant-Net]]
Cell Instance Segmentation & Foundation Models: [[HistoPLUS: Towards Comprehensive Cellular Characterisation of H&E Slides]], [[NuClick]], [[CytoFormer: A Molecularly Supervised Cell Foundation Model for Histopathology Cell Classification]], [[HoVer-NeXt]], [[HistoGen: Histopathology Cell Nuclei Image Generation Tool]]
Software Ecosystem & Tools: [[Digital Pathology Software]], [[Digital Pathology]], [[Image Analysis]], [[From Samples to Knowledge 2025: QuPath Training Course]], [[Multiplex Immunofluorescence Image Analysis with QuPath — Part 1: Understanding Digital Images]]
Literature Reviews: [[Articles on computational, digital, and mathematical pathology]], [[Navigating foundation model selection in digital pathology through performance evaluation and tradeoff analysis]], [[A Hybrid MIL Approach Leveraging Convolution and State-Space Model for Whole-Slide Image Cancer Subtyping]]
Last updated