Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .github/workflows/frontend_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,19 @@ jobs:
matrix:
include:
- project: mock
config: playwright.config.ts
shard: 1
total: 2
- project: mock
config: playwright.config.ts
shard: 2
total: 2
- project: seeded
config: playwright.config.ts
shard: 1
total: 1
- project: analytics
config: playwright.analytics.config.ts
shard: 1
total: 1
outputs:
Expand Down Expand Up @@ -119,8 +126,12 @@ jobs:
- name: Install Playwright browsers
run: npx playwright install --with-deps chromium

- name: Build the isolated analytics frontend
if: matrix.project == 'analytics'
run: npm run build

- name: Run E2E shard (no credentials needed)
run: npx playwright test --fail-on-flaky-tests --project=${{ matrix.project }} --shard=${{ matrix.shard }}/${{ matrix.total }}
run: npx playwright test --fail-on-flaky-tests --config=${{ matrix.config }} --project=${{ matrix.project }} --shard=${{ matrix.shard }}/${{ matrix.total }}
env:
CI: true
# Mock specs still use the backend for auth bootstrap and some API calls.
Expand Down
70 changes: 69 additions & 1 deletion doc/gui/0_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,75 @@ To deploy an isolated instance for an external team, see [Deploy a New Instance]

## Views

CoPyRIT has three main views, accessible from the left sidebar: **Technique.execute** (Chat), **Attack History**, and **Target Configuration**. The **Theme** menu is available at the bottom of the sidebar.
Use the left sidebar to open **Home**, **Technique.execute** (Chat), **History**, **Analytics**, **Scanner**,
**Datasets**, or the **Registry**. **Configuration** is available to administrators.
The **Theme** menu is at the bottom of the sidebar.

### Attack Results Analytics

**Analytics** explores saved attack results using the
[SDK analytics API](../code/analytics/0_attack_results.md). Every distinct saved
AttackResult ID counts, including different IDs sharing a conversation and all
result roles. Selecting a scenario filters this same population, including its
retries. It does not switch to the scanner's latest-execution-unit statistics.

Use **Operation**, **Operator**, **Objective target**, or **Add filter** to select
saved metadata. Additional dimensions include model, targeted harm category,
attack type, request or response converters, scenario run, and arbitrary label
keys. Targeted harms describe intended coverage, not harms detected by scorers.
Target options use persisted behavioral identities, not current registry
membership. The content hash in a matching row is for inspecting that saved
target, not a substitute for its filter key.

Filter values load only when their editor is open, with debounced search and
bounded **First values** / **Next values** paging. Selections survive searches
and pages. Values within a filter are alternatives (**ANY**); converters also
support **ALL**. Separate filter chips are combined with **AND**, even when
they select the same dimension. Missing metadata, an empty converter pipeline,
and literal values such as `Unknown` remain distinct.

**Last updated** accepts ISO 8601 timestamps with `Z` (UTC) or an explicit offset.
For example, `2026-11-01T01:30:00-07:00` and `2026-11-01T01:30:00-08:00`
are different instants during a daylight saving transition. After is inclusive;
before is exclusive. Microseconds are retained. These bounds describe last
updates, not attack execution times.

The dashboard offers outcome-composition bars, success-rate bars, and a
two-axis heatmap. **ASR denominator** defaults to decided results (successes plus
failures). All-results ASR includes errors and undetermined outcomes. Both rates
come from the SDK. Counts are explicit: `100%, 3 success / 3 decided, 5 total`
means three successes and two results without a decision, not five successes.
An unavailable decided denominator displays **Unavailable**, not `0%`.
Outcome filters apply to the entire dashboard and mark **ASR*** with an
explanation, including the `100%*` produced by a successes-only cohort.

Click a group, outcome segment, or heatmap cell to append the SDK's drill-down
predicates. Overlapping group totals must not be added together. When the
16-predicate or 500-value budget prevents another drill-down, the SDK's reason
is displayed; the current charts, results, and filter editing remain available.
Group and result pages stay bounded. **Show cell data** provides a textual
heatmap view. Matching result rows open the ordinary saved-attack view with a
click, Enter, or Space.

Use **Reload** for a new report. **Last refreshed** shows the last successful
report time; paging matching results updates only **Page read**, not the report.
Presentation changes preserve pages and do not request new aggregates.
The dashboard does not poll, although the app's shared readiness checks continue.
Errors retain a same-cohort last successful report with a stale warning. Busy
read retries honor the server's cooldown; invalid result cursors can return to
the first results page without recalculating the report.

Copy the browser URL to share filters and presentation. New links put the state
after `#analytics=`, so large filter values are not sent in HTTP request targets
and do not exceed the backend's 8 KiB URL limit on reload. Valid older query
links are migrated to this representation; invalid links require an explicit
reset rather than silently querying a broader cohort. Browser Back and the
Analytics sidebar button retain the exploration. Links contain the selected
metadata values, so share them only with appropriate recipients.

Outcome fills are consistent across summaries, bars, swatches, and result
badges: success is green, failure red, execution error blue, and undetermined
gray. Text and keyboard actions remain available independently of color.

### Themes

Expand Down
55 changes: 51 additions & 4 deletions frontend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,51 @@ a new attack, the draft is kept so you can retry after loading finishes. Copies
and drafts saved to an existing attack remain available during defaults reloads
and keep that attack's attribution.

## Attack results analytics

Open **Analytics** to filter saved results, compare SDK-computed outcome
statistics, and drill into matching attacks. See the
[analytics GUI guide](../doc/gui/0_gui.md#attack-results-analytics) for cohort
semantics, denominators, time bounds, URL sharing, and refresh behavior.
The implementation follows
[part 5 of the analytics roadmap](https://gist.github.com/romanlutz/34230aba87f7fa07996772a4099e9a78).

The shared `AttackResults/OutcomeSummaryBar` accepts `OutcomeStatistics`, not
result rows. This keeps it reusable without counting a paginated sample or
inferring historical scenario breakdowns from retries.

### Isolated analytics browser checks

From `frontend`, run:

```bash
npm run test:e2e:analytics
```

This builds the current frontend and starts the real application routes and
compatibility/readiness middleware over a disposable local SQLite database.
Its synthetic results include repeated conversation IDs, all result roles,
converter memberships, and a saved retry scenario. It uses a private home,
explicit empty environment-file/initializer lists, and no provider calls,
shared database, or Azure credentials. The normal E2E configuration excludes
this fixture-specific spec; its dedicated configuration owns the server.
CI runs it as an additional isolated shard of the required Frontend E2E check,
using the same report upload, merge, and failure gate as the other projects.
Python dependencies must first be available through `uv sync` at the repository
root. `PYRIT_E2E_ANALYTICS_PORT` changes the loopback port when needed.
If Python is managed only by uv, point Vite's existing Python override at that
environment before running the command. In PowerShell:

```powershell
$env:PYRIT_PYTHON = uv run python -c "import sys; print(sys.executable)"
npm run test:e2e:analytics
```

Set `PYRIT_ANALYTICS_SCREENSHOT_DIR` to an artifact directory to also capture the
verified overview in light/dark, a heatmap drill-down, and narrow controls/report views.
The browser checks do not mock analytics APIs. They do not repeat the separate
100,000-result backend performance workload.

## Development

```bash
Expand Down Expand Up @@ -206,10 +251,12 @@ E2E_LIVE_MODE=true npx playwright test

The mock and seeded projects run in the **GitHub Actions** pull-request workflow. The live project is intended for a protected pipeline with an Entra identity or API keys.

CI runs two isolated mock shards alongside one serial seeded job, with one
worker per job. Each job starts its own Vite server and Python backend because
mock specs still depend on backend auth bootstrap and some API calls. The jobs
never share backend state. The required **Frontend E2E Tests** check merges their reports and fails
CI runs two isolated mock shards, one serial seeded job, and one isolated
analytics job, with one worker per job. Mock and seeded jobs start their own
Vite server and Python backend because mock specs still depend on backend auth
bootstrap and some API calls. The analytics job serves its production build
from the disposable SQLite fixture. Jobs never share backend state.
The required **Frontend E2E Tests** check merges their reports and fails
if any shard fails, is cancelled, or is missing a report. Skipped and flaky tests
also fail CI.

Expand Down
Loading
Loading