Methodology

How detectors are scored, and why. Everything is recomputable from committed per-prompt scores.

What's being measured

A prompt-injection detector outputs a risk score in [0, 1] per input; you pick a threshold above which you block. Two questions decide whether it's deployable, and they trade off:

Detection — does it catch attacks it hasn't seen?
False positives — how often does it flag ordinary, benign traffic?

A number on either axis alone is close to meaningless. A 0.95-AUC detector that trips on a quarter of real users is an outage; a detector with a low false-alarm rate that misses attacks is theatre. We report both.

Three families of metric

Metric	Threshold-dependent?	What it tells you
ROC-AUC	No	How well the detector ranks attacks above benign across all thresholds. Can't be gamed by threshold choice.
F1 / FPR at 0.5	Yes	Out-of-the-box behaviour if you don't tune. Realistic (most deployments never tune) but unfair to models whose scores cluster near 0.5.
FPR at fixed catch rate; EER; operating curves	No (normalized)	Tune each detector to the same catch rate (e.g. 95% of attacks), then compare false-alarm cost. Removes the "where's the line" objection.

The leaderboard's headline table shows avg AUC, avg F1@0.5, and FPR@0.5 on real traffic for direct and indirect axes separately. Operating-point data (FPR@95%/99% catch, EER) is in operating_points.json.

Datasets

All evaluation sets are held out — they are not training data for the detectors. Three groups:

Direct attacks (detection axis): four public held-out adversarial benchmarks — rogue, JailbreakBench, xTRam1-test, S-Labs-test.
Real benign traffic (false-positive axis): first user turns sampled deterministically (fixed seed, reservoir sampling) from WildChat-1M and LMSYS-Chat-1M — real messages, not synthetic "benign" prompts.
Indirect / structured injection (separate axis): injection hidden inside data — Z-Edgar, BIPIA, InjecAgent, AgentDojo, HackAPrompt, TensorTrust. Reported separately, not folded into the direct average, because it's a distinct capability.

Some datasets are gated (LMSYS, HackAPrompt) — the harness skips them cleanly if you don't have access.

Scoring protocol — the same for everyone

Pure classifier, one path. Every detector is loaded as a HuggingFace AutoModelForSequenceClassification and scored identically. No model gets a special wrapper or SDK path. The registry (models.yaml) records each model's HF id and the softmax index that means "attack".
No per-model threshold tuning for fixed-threshold metrics: 0.5 for everyone.
Calibration is auto-applied if shipped. If a model's HF repo includes a temperature.json, logits are divided by it before softmax. Models without one get T=1.0.
Averages are unweighted across datasets (JailbreakBench, n=200, counts as much as rogue, n=5,000). Read the per-dataset columns in the raw JSON, not just the average.
Polarity check: a binary detector whose AUC comes out below 0.5 has its label index reversed. An AUC far below 0.5 is a config error, not a bad model.

ROC curves on the leaderboard page

The ROC chart on /leaderboard/ plots TPR vs FPR from det_points.json. Those curves are computed by pooling all four direct-attack datasets by sample count before deriving TPR/FPR at each threshold — so rogue (n=5,000) dominates JailbreakBench (n=200).

The table's Direct avg AUC and Direct avg F1@0.5 columns use unweighted per-dataset averages instead. Both views are valid; they answer slightly different questions. Read per-dataset columns in the raw JSON if you need to reconcile them.

Reproducibility

Raw scores are committed. Each scoring run dumps per-prompt scores (results/scores/, results/scores_indirect/ — scores + labels only, no prompt text). Every table and curve recomputes from those with no GPU.
GPU runs aren't bit-stable; the committed numbers are. PyTorch detectors vary slightly run-to-run on a GPU. Published numbers are pinned by the committed scores, not by trusting any single GPU run.
Each quarterly run is archived at /leaderboard/data/{date}/. The live page reads /leaderboard/data/latest/.

What this benchmark does not claim

Detectors catch the patterns they were trained on. No detector here addresses domain-specific abuse, multi-step/contextual attacks (a single-prompt classifier has no conversation state), or tool misuse. A detector is one layer of defense-in-depth, not the whole wall. See FINDINGS.md.

Run it yourself

git clone https://github.com/bastion-soft/pi-detector-bench.git
cd pi-detector-bench
pip install -e .

python -m scripts.run_leaderboard          --dump-scores results/scores
python -m scripts.measure_false_positives  --dump-scores results/scores
python -m scripts.eval_indirect            --dump-scores results/scores_indirect
python -m scripts.rebuild_results_from_scores
python -m scripts.analyze_operating_points

Full source methodology: METHODOLOGY.md in the harness repo. Propose changes via Submit a detector or a methodology proposal.