Submit a detector

It's a one-file PR. Your model runs through the same generic path as every other entry.

Requirements

• Your detector must be a public HuggingFace AutoModelForSequenceClassification — scored as a plain classifier, no custom code.
• No trust_remote_code. PRs that need it are declined for security reasons.
• Gated models are fine — the harness skips them cleanly if the runner doesn't have HF access approval.

Step 1 — register it

Append an entry to models.yaml:

  - name: "my-detector (220M)"
    hf_id: "my-org/my-prompt-injection-detector"
    attack_label: 1        # softmax index for "attack" (or a list to sum, e.g. [1, 2])
    params: "220M"
    # gated: true          # only if the model needs HF access approval

Step 2 — run the harness

A free Colab T4 works — see the repo's notebooks/benchmark_colab.ipynb.

python -m scripts.run_leaderboard          --dump-scores results/scores
python -m scripts.measure_false_positives  --dump-scores results/scores
python -m scripts.eval_indirect            --dump-scores results/scores_indirect
python -m scripts.rebuild_results_from_scores
python -m scripts.analyze_operating_points
python -m scripts.analyze_operating_points --scores-dir results/scores_indirect --within-set --label indirect

Step 3 — open the PR

Include:

• Your models.yaml entry
• Regenerated results/*.{json,md}
• New per-prompt score files under results/scores*/ (scores + labels only — no prompt text)

Maintainers re-derive your model's tables from the committed scores to confirm they reproduce. Once merged, your detector appears in the next quarterly update on /leaderboard/.

Sanity checks

• If your model's AUC is below 0.5, your attack_label is probably inverted — flip it (0↔1).
• Don't tune a per-model threshold; fixed-threshold metrics use 0.5 for everyone.

Other contributions

• Methodology change — open an issue first, then a PR updating METHODOLOGY.md and regenerating affected results.
• Add a dataset — direct attack set, benign source, or indirect/structured set. See the full guide in CONTRIBUTING.md.