Static leaderboard for completed PromptSecurity experiments.
Live site:
https://datasec-lab.github.io/PromptSecurityLeaderboard/
The public page includes a contribution section with links to:
- PromptSecurity code and evaluation data:
https://github.com/datasec-lab/PromptSecurity - Issue submissions:
https://github.com/datasec-lab/PromptSecurity/issues - Pull requests:
https://github.com/datasec-lab/PromptSecurity/pulls - This leaderboard site:
https://github.com/datasec-lab/PromptSecurityLeaderboard
From repo root:
python leaderboard_site/scripts/build_leaderboard_data.py \
--input-dir experiments/placeholders \
--utility-input-dir experiments/placeholders_utility \
--output leaderboard_site/data/leaderboard.jsonFrom repo root:
python -m http.server 8080Open:
http://localhost:8080/leaderboard_site/
The page includes a live traffic row (Total / This Page / Last 30 Days).
Configure in leaderboard_site/index.html:
<body data-goatcounter-code="your-goatcounter-code">Examples:
data-goatcounter-code="promptsecurityeval"data-goatcounter-code="https://promptsecurityeval.goatcounter.com"
Notes:
- The site auto-loads GoatCounter
count.jsand refreshes counters every 30s. - If counters fail, check GoatCounter site settings for visitor count visibility/API access.
Workflow file:
/.github/workflows/deploy_leaderboard_pages.yml
This workflow builds from the private main repo and publishes the static site to a separate public repo branch.
- Create a public repo (example:
your-org/promptsecurityeval-leaderboard). - In that public repo, enable GitHub Pages:
- Settings -> Pages
- Source:
Deploy from a branch - Branch:
gh-pages(or the branch you choose), folder:/ (root)
In Settings -> Secrets and variables -> Actions:
- Add repository variable:
LEADERBOARD_PUBLIC_REPO=owner/repoof the public target repo.
- Optional repository variable:
LEADERBOARD_PUBLIC_BRANCH= publish branch (defaultgh-pages).
- Add repository secret:
LEADERBOARD_PUBLISH_TOKEN= PAT that can push to the target public repo.
PAT recommendation:
- Fine-grained PAT with
Contents: Read and writeon the target public repo only. - Classic PAT with
reposcope also works, but is broader.
- Push to
main(auto trigger), or - GitHub -> Actions ->
Publish Leaderboard To Public Repo->Run workflow.
The workflow rebuilds leaderboard_site/, bundles only paper-compatible run payloads into leaderboard_site/data/runs/ for the comparison popup, and publishes the static site to the target branch.
- Reads paper-compatible placeholders with primary GPT-bin scores; status is used only to choose the best duplicate for the same paper key.
- Schema
v3includes a paper-compatiblepaperblock for default site tables. - The default generated JSON is aggregate-only:
meta,overview, andpaper. - Public publishing uses
--bundle-runs-dirto include only paper-compatible run-level payloads needed by the comparison popup. - The default leaderboard is restricted to the paper main experiment:
- 11 configured target models.
- 20 configured attack settings.
- 9 configured defenses, with
no_defenseused only as a matched baseline. - HarmBench-style GPT-4.1-mini judger (
gpt_judger_harmful_binary) as the primary harmful-output judger.
- Auxiliary runs such as
no_attackbaselines are retained only for derived metrics; ABJ assistant-LLM ablations and other non-main attack variants are excluded from leaderboard rankings. - Leaderboard ASR uses only
gpt_judger_harmful_binary. - HarmBench, HarmBench-style GPT-4.1-mini, and prefix judger results are exposed as independent comparison views; they are not averaged together.
ASRis derived from sample-level judger outputs:0means safe.1means unsafe.
- Model rankings use no-defense ASR and matched clean-baseline transition metrics.
- Defense rankings use matched no-defense counterfactuals on shared sample identifiers.
- Attack and defense rankings are split by black-box-compatible and white-box-only method access.
- Utility deltas are read from
experiments/placeholders_utilitywhen available. - Headline matrices do not fill missing no-defense cells from defended runs.