> For the complete documentation index, see [llms.txt](https://www.parapathology.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.parapathology.com/computational-digital-and-mathematical-pathology/croma.md).

# CRoMa

Scores how much of a pathology foundation model's embedding geometry is organised by the *hospital* rather than by the tissue — a small, well-tested, pip-installable metrics library that takes embeddings you already have and never touches a slide. **Worth adopting for exactly one job: deciding which encoder to trust on multi-centre material before committing to it.**

* **Repository:** [clemsgrs/croma](https://github.com/clemsgrs/croma) — Apache-2.0
* **Documentation and live results:** [clemsgrs.github.io/croma](https://clemsgrs.github.io/croma/)
* **Paper:** Grisi C, van der Laak J, Litjens G. *A distributional robustness margin for pathology foundation models.* arXiv:2607.25497 — digested in [A distributional robustness margin for pathology foundation models](/appendix/clippings/a-distributional-robustness-margin-for-pathology-foundation-models.md), which is where the method, the formula and the findings live. This note is about the code.
* **PyPI:** `pip install croma` (v1.0.0)

Written at the Department of Pathology, Radboud UMC, by the authors of the paper.

## What it is

A metrics library, and deliberately nothing more. Its README states the boundary plainly — it *"never loads a model or reads an image — you bring the embeddings"* — and the dependency list backs that up: numpy, pandas, scikit-learn, tqdm. No torch, no image I/O, no model weights in the core install.

It implements five things:

| Symbol  | What it scores                                                                  | Origin                                   |
| ------- | ------------------------------------------------------------------------------- | ---------------------------------------- |
| `RI`    | Robustness Index — counts typed neighbours in a fixed-*k* neighbourhood, pooled | Re-implementation of the PathoROB metric |
| `MaRI`  | Margin-aware RI — same design, distance-weighted                                | This project (as a control)              |
| `CRoMa` | Per-sample signed margin between typed neighbour distances                      | This project (the headline metric)       |
| `apd`   | Average performance drop under a confounder-biased probe sweep                  | Vendored from PathoROB                   |
| `nipd`  | Normalised integrated performance degradation                                   | This project                             |

That it ships **its own competitor's metric** is the useful part. RI is the metric of [Towards robust foundation models for digital pathology](/appendix/clippings/towards-robust-foundation-models-for-digital-pathology.md), the Nature Communications paper this work argues against; having all three in one library on one manifest is what makes the comparison cheap instead of a reimplementation project.

## Using it

The input contract is generic and is the reason this is reusable beyond its own paper: a feature array plus a **manifest** — one row per sample carrying a biological label, a `group_id` (the slide for tiles, the case for slides, used to exclude near-duplicates), and *any column you nominate as the confounder*. Centre, scanner, staining batch, fixation protocol — the code does not care which.

```
RI.compute(features, manifest, confounder_column=..., k_candidates=...)
MaRI.compute(...)
CRoMa.compute(...)
```

Supporting helpers: `croma.expand_features_to_manifest(...)`, `croma.alignment.build_embedding_source_manifest(...)`, and `croma.downstream` for `apd` / `nipd` / `probe_sweep`.

The reproduction pipeline lives under `scripts/` and is **not installed** with the package: `scripts/bench/extract_embeddings.py` to embed a tileset, `scripts/bench/benchmark.py` to score it, `scripts/bench/render.py` for figures, `scripts/repro/reproduce_faithful.py` to recompute the paper end to end. The `repro` extra pulls in torch, timm, transformers and huggingface\_hub for that path.

## Four things that will trip you up

These are the reason this note exists; none of them is in the paper, and two of them break code silently.

1. **There is no CLI any more.** Versions 0.1.0 and 0.2.0 shipped a `croma` console script; it was removed outright in 0.3.0 on the grounds that nothing internal depended on it. As of v1.0.0 the library is Python-API only. Any tutorial or shell snippet invoking `croma ...` predates August 2026.
2. **`napd` became `nipd`, and it is not a rename.** The project's own decision record is explicit that the new name denotes *a different estimand rather than a drop-in rename* — the normalisation base changed from total baseline accuracy to above-chance headroom. An old import fails loudly; an old *number* compared against a new one fails quietly.
3. **The committed results are not the paper's results.** `results/PROVENANCE.json` records a roster of **26** — 25 pathology encoders plus the DINOv2-B natural-image control — against the paper's 20 plus control. Mascaret, Phaet and three RudolfV-2 variants have been added since, and **Mascaret now leads the cross-benchmark Pareto ranking**, a model the manuscript never evaluated. Those CSVs were exported at croma 0.3.0 on 2026-08-13. Reproducing the published tables means checking out an earlier tag, not `main`.
4. **Reuse carries a redistribution obligation.** `NOTICE` records that `src/croma/downstream/_pathorob.py` vendors three functions verbatim from PathoROB under BSD-3-Clause (copyright BIFOLD Pathomics). Apache-2.0 covers the rest, but anyone redistributing that module inherits the BSD notice requirement. This is a licence term, not a citation courtesy.

## The results site is the better front door

[clemsgrs.github.io/croma](https://clemsgrs.github.io/croma/) is a Sphinx site rebuilt from `main` on every push, and its Results section is genuinely interactive rather than a rendered table: a per-sample **distribution explorer** (200-bin histograms per encoder per cohort, with a drag-brush readout and two-encoder overlay), a **Pareto scatter** of median CRoMa against tail severity with the frontier ringed, and an **nIPD evidence browser** for the shortcut-susceptibility results. There are four per-cohort deep-dive pages (Camelyon, TCGA-4×4, Tolkach-ESCA, PCaBiop) and a "request a model evaluation" intake with separate tracks for public tile encoders, public slide encoders and private models.

The README carries only a static top-8 excerpt of the ranking. For the full 26-row panel, the site is the only place it exists.

## What you can and cannot reproduce

| Want to                                        | Feasible?                                                                                                                                            |
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Inspect the published per-model scores         | **Yes, trivially** — `results/*.csv` are committed, with protocol provenance                                                                         |
| Run RI / MaRI / CRoMa on *your own* embeddings | **Yes** — this is the library's actual job, and it is well tested                                                                                    |
| Re-download the three PathoROB tile cohorts    | **Yes** — `scripts/prep/prepare_pathorob.py` pulls them from Hugging Face                                                                            |
| Obtain PCaBiop                                 | Manual — sourced from the Kaggle PANDA competition; the derived dataset is at [waticlems/pcabiop](https://huggingface.co/datasets/waticlems/pcabiop) |
| Re-run the full panel end to end               | **No, by design** — no embeddings are distributed, so you must run every encoder yourself, several of them gated, at real GPU cost                   |

The project states this posture openly in its own decision records rather than implying a one-command reproduction it cannot deliver, which is worth more than a `Makefile` that breaks on first contact.

## Quality signals, read at source

`src/croma/metrics/croma.py` computes `croma = (mean_os - mean_so) / (mean_os + mean_so)` from scikit-learn cosine k-NN distances — the paper's Equation 10 exactly, with explicit per-sample tracking of *why* a value is missing (unresolved neighbour search versus zero-distance degeneracy) rather than a bare NaN. `tests/test_croma.py` builds literal toy embedding arrays and asserts exact metric values against hand-worked expectations: behavioural tests of the mathematics, not smoke tests. CI on `main` was green at the last run.

## Why this is worth having here

This vault's [Hugging Face Digital Pathology](/computational-digital-and-mathematical-pathology/hugging-face-digital-pathology.md) catalogue names the encoders worth reaching for; between them, this library and the two papers behind it supply the axis that catalogue was missing — **not how well an encoder recovers biology, but whether it recovers biology or the laboratory when the two compete.** Both are near-ceiling on the first; they differ enormously on the second.

The practical case for the code rather than the papers: the manifest contract is generic, the dependency footprint is four packages, and nothing leaves the machine. Running it against our own multi-centre cohorts — with `confounder_column` set to scanner or stain batch rather than centre — is a genuinely small piece of work, and it is the only way to learn whether a published robustness ranking survives contact with local material. Everything published so far is measured on public cohorts.

## Current state / open questions

* **Nothing here has been run on our own material.** `[unverified]` — the whole value proposition above is a hypothesis until it is.
* **Six months old, four stars, no forks, single-institution.** The code quality is well above what those numbers suggest, but there is no external-adoption signal yet, and no second implementation to check it against.
* **The metric is contested in the direction that flatters this repo.** CRoMa, RI and MaRI produce broadly the same model ranking (ρ ≈ 0.93–0.95); the argument for CRoMa is the per-sample distribution, not a different leaderboard. Adopting the library does not require accepting that CRoMa displaces RI — you get all three.
* **The moving leaderboard is a maintenance hazard for us, not just for them.** Any figure this vault quotes from the site needs the date attached, because the panel has already grown by five encoders since the manuscript and its current leader is a model the paper never saw.
* **Does the PCaBiop slide-level manifest ship separately from the aggregate scores?** `[unverified]` — not established from the files read.

Derived from: repository read at source level 2026-08-30 to 2026-08-31 — `LICENSE`, `NOTICE`, `pyproject.toml`, `README.md`, `CHANGELOG.md`, `CITATION.cff`, `src/croma/metrics/croma.py`, `src/croma/metrics/ri.py`, `tests/test_croma.py`, `docs/adr/0003-reproducibility-posture.md`, `docs/datasets.rst`, `docs/benchmarking.rst`, `scripts/repro/reproduce_faithful.py`, `scripts/prep/prepare_pathorob.py`, `results/PROVENANCE.json` and `results/*.csv`; the GitHub API for currency, releases and commit count; the PyPI JSON API for the published package; the rendered documentation site and its three widget data endpoints; and arXiv:2607.25497v4 in full.