Skip to content

Home

Ninety seconds: install, bring your embeddings (any encoder, any modality), the dashboard from python app.py to the downloaded PDF, and the MCP call for agents.

PyPI version CI Docs Paper

ShortKit-ML measures the shortcut potential of a machine learning embedding with one calibrated audit and writes the result as an HTML or PDF report.

The audit answers one question: does the embedding carry a sensitive attribute (sex, age group, race, site, scanner) beyond what the task label explains? A conditional probe is the primary test, five secondary detectors are reported as diagnostics, and all six are calibrated by a permutation null that shuffles the attribute within task-label strata, so the probe decides at p < 0.05 and nothing else does.

Start here

  • Walkthrough

    The whole audit on one public data set, from a fresh install to the PDF report, in five terminal recordings and a dashboard video, all replayable on your own data.

    Walkthrough

  • Installation

    Clone and install from source with uv or pip. The audit module is newer than the PyPI release, so source is the way in for now.

    Installation

  • MCP server for agents

    Nine tools over stdio. run_audit runs the audit on files you name in chat, from Claude Code, Claude Desktop or any other MCP client.

    MCP server

  • The paper

    ShortKit-ML: an open toolkit for measuring shortcut potential in medical imaging embeddings. Preprint on medRxiv, with the method, the synthetic calibration and the clinical audits.

    Read the preprint

What the audit needs

Array Shape Meaning
E (n, d) float one embedding per row, from any encoder, any width d
A (n,) the sensitive attribute
Y (n,) the task label the classifier would learn
groups (n,), optional a unit id (patient, identity) when one unit has several rows

Run it

import numpy as np
from shortcut_detect.audit import AuditConfig, conditional_audit
from shortcut_detect.audit.report import audit_report

E = np.load("embeddings.npy")   # (n, d), any encoder
A = np.load("attribute.npy")    # (n,)
Y = np.load("labels.npy")       # (n,)
result = conditional_audit(E, A, Y, config=AuditConfig(seed=42))  # groups=patient_ids when needed
print(result.results["probe"].p_value)
audit_report(result, "audit.pdf", attribute="sex", task="pneumonia")

The dashboard, the 19 exploratory detectors and the mitigation methods are documented under Other package methods.

Cite

@article{cajas2026shortkit,
  title={ShortKit-ML: an open toolkit for measuring shortcut potential in medical imaging embeddings},
  author={Cajas, Sebastian and Marzullo, Aldo and Kapadia, Sahil and Santos, Filipe and Ocampo Osorio, Felipe and Kong, Qingpeng and Quarta, Alessandro and Kuo, Po-Chih and Patel, Milit and Rojas Sillery, Raul Ignacio and Celi, Leo Anthony},
  journal={medRxiv},
  year={2026},
  doi={10.64898/2026.04.29.26352053}
}

MIT licence, see LICENSE.