Add daily grader audit workflow

.github/workflows/daily-grader-audit.md

@@ -0,0 +1,156 @@
+---
+private: true
+emoji: "📊"
+description: Daily audit of workflow runs with deterministic grader results, producing a downloadable 24-hour report artifact
+on:
+  schedule: daily around 8:00
+  workflow_dispatch:
+permissions:
+  contents: read
+  actions: read
+max-ai-credits: 1500
+max-daily-ai-credits: 5000
+engine:
+  id: claude
+  mcp:
+    tool-timeout: 10m
+tools:
+  cli-proxy: true
+  agentic-workflows:
+safe-outputs:
+  upload-artifact:
+    max-uploads: 1
+    retention-days: 1
+    skip-archive: true
+timeout-minutes: 30
+features:
+  gh-aw-detection: true
+imports:
+  - shared/reporting.md
+graders: {}
+evals:
+  - id: grader_runs_audited
+    question: Did the agent inspect workflow runs with deterministic grader results from the last 24 hours?
+  - id: downloadable_report_uploaded
+    question: Did the agent generate and upload a grader audit report artifact retained for 24 hours?
+---
+
+# Daily Grader Audit
+
+You are the grader audit reporter for GitHub Agentic Workflows.
+
+## Mission
+
+Inspect workflow runs from the last 24 full hours that produced deterministic grader results, summarize grader health, and upload a Markdown report artifact retained for 24 hours.
+
+## Context
+
+- Repository: `${{ github.repository }}`
+- Run ID: `${{ github.run_id }}`
+- Window: last 24 full hours ending at workflow start (UTC)
+- Report path: `/tmp/gh-aw/agent/grader-audit-report.md`
+
+## Phase 1: Fetch Grader Runs
+
+Use the agentic-workflows MCP `logs` tool once:
+
+```json
+{
+  "workflow_name": "",
+  "count": 100,
+  "start_date": "-1d",
+  "graders": true,
+  "artifacts": ["graders", "usage"]
+}
+```
+
+Read the returned JSON file path. The logs tool filters to runs with grader results and each `runs[]` entry may include:
+
+- `run_id`
+- `workflow_name`
+- `conclusion`
+- `created_at`
+- `url`
+- `graders.total`
+- `graders.passed`
+- `graders.failed`
+- `graders.error_count`
+- `graders.unavailable_count`
+- `graders.results[]`
+
+If no matching runs are found, still write a report explaining that no grader results were available for the window.
+
+## Phase 2: Analyze Results
+
+For each run with `graders.results[]`, compute:
+
+- Per-run totals: total, passed, failed, errors, unavailable.
+- Per-grader totals grouped by `id`: runs observed, pass count, fail count, error count, unavailable count.
+- Pass rate per grader: `pass count / runs observed * 100`.
+- Notable failures: any grader result where `status` is `fail`, `error`, or `unavailable`.
+
+Do not infer missing graders from workflow files. Only report grader data present in the logs tool output.
+
+## Phase 3: Generate Report
+
+Create `/tmp/gh-aw/agent/grader-audit-report.md` with:
+
+### Summary
+
+One short paragraph covering the window, number of runs, number of workflows, and overall grader health.
+
+### Key Metrics
+
+| Metric | Value |
+|---|---|
+| Runs with grader results | N |
+| Workflows with grader results | N |
+| Grader result rows | N |
+| Passing results | N |
+| Failing results | N |
+| Error results | N |
+| Unavailable results | N |
+
+### Per-Grader Results
+
+| Grader | Runs | Pass | Fail | Error | Unavailable | Pass Rate |
+|---|---:|---:|---:|---:|---:|---:|
+| grader-id | N | N | N | N | N | X% |
+
+### Per-Run Results
+
+| Run | Workflow | Conclusion | Total | Pass | Fail | Error | Unavailable |
+|---|---|---|---:|---:|---:|---:|---:|
+| [§123](https://github.com/owner/repo/actions/runs/123) | workflow-name | success | N | N | N | N | N |
+
+<details>
+<summary>Notable grader failures</summary>
+
+| Run | Workflow | Grader | Status | Value | Threshold | Message |
+|---|---|---|---|---:|---|---|
+| [§123](https://github.com/owner/repo/actions/runs/123) | workflow-name | grader-id | fail | 0.2 | >= 0.8 | explanation |
+
+</details>
+
+### Recommendations
+
+List concrete follow-up actions only when failed, errored, or unavailable grader results exist. Otherwise state that no immediate action is needed.
+
+### Context
+
+- Window: last 24 full hours ending at workflow start (UTC)
+- Source: `gh aw logs --graders --start-date -1d --artifacts graders,usage`
+- Generated by run `${{ github.run_id }}`
+
+## Phase 4: Upload Report
+
+Stage and upload the report:
+
+1. Copy `/tmp/gh-aw/agent/grader-audit-report.md` to `${RUNNER_TEMP}/gh-aw/safeoutputs/upload-artifacts/grader-audit-report.md`.
+2. Call the `upload_artifact` safe-output tool with:
+
+```json
+{ "path": "grader-audit-report.md" }
+```
+
+The workflow safe-output configuration retains the uploaded report for 1 day.

docs/adr/56359-add-graders-run-filter-and-daily-audit-workflow.md

@@ -0,0 +1,49 @@
+# ADR-56359: Add Grader Run Filter and Daily Grader Audit Workflow
+
+**Date**: 2026-08-27
+**Status**: Draft
+**Deciders**: pelikhan, copilot-swe-agent
+
+---
+
+### Context
+
+After grader results were surfaced in the `usage` artifact (see ADR-56066), operators and automation needed two new capabilities: (1) a way to narrow `gh aw logs` downloads to only workflow runs that produced grader results, analogous to the existing `--evals` filter, and (2) a scheduled audit workflow that aggregates 24-hour grader health across all agentic workflows and uploads a downloadable Markdown report. Without a run-level grader filter, every `gh aw logs` call in a grader-focused investigation would download all runs regardless of whether they contained grader output, wasting bandwidth and time. Without a dedicated audit workflow, grader health could only be inspected by manually running ad-hoc CLI queries.
+
+### Decision
+
+We will add `--graders` as a boolean run-level filter to `gh aw logs` (forwarded through the MCP `logs` tool as the `graders` parameter) and introduce a new daily scheduled workflow (`daily-grader-audit.md`) that fetches the last 24 hours of runs with grader results, computes per-grader pass rates, and uploads a Markdown report artifact retained for 24 hours. The filter follows the same pattern as `--evals`: when active it auto-expands the artifact set to include `graders`, then applies `skipByGradersFilter` to exclude runs that lack grader result files. The core `collectProcessedWorkflowRuns` loop was refactored into `fetchAndProcessLogsBatch` and `handleLogsBatchError` helpers as part of this work.
+
+### Alternatives Considered
+
+#### Alternative 1: Post-Download Filtering Inside the Audit Workflow Prompt
+
+Instead of adding a `--graders` CLI flag, the daily audit workflow could download all runs and have the agent discard entries without grader data after downloading.
+
+This would avoid any CLI or MCP changes. It was rejected because it wastes bandwidth and runner time proportional to total run volume, which scales poorly as the number of agentic workflow runs grows. The `--evals` precedent shows that run-level filtering at download time is the correct pattern for optional, artifact-backed features; diverging from that convention for graders would create an inconsistent developer experience.
+
+#### Alternative 2: Extend `gh aw audit` with a Cross-Run Grader Health Section
+
+Retrofit `gh aw audit` to aggregate grader outcomes across multiple runs rather than building a separate scheduled workflow.
+
+This would reuse existing audit infrastructure. It was rejected because `gh aw audit` operates on a single run's downloaded data and its API is not designed for cross-run aggregation. Restructuring it to support a rolling 24-hour window would require significant scope expansion. A standalone scheduled workflow with its own agent prompt is a cleaner fit and keeps `gh aw audit` single-run-scoped.
+
+### Consequences
+
+#### Positive
+- Operators and automated agents can scope `gh aw logs` downloads exclusively to grader-bearing runs, reducing bandwidth and latency for grader-focused investigations.
+- A daily Markdown artifact gives teams a lightweight, downloadable grader health report with per-grader pass rates and notable failures without any custom queries.
+- The implementation follows established conventions (`--evals` pattern, `skipByXxxFilter` chain, artifact-set auto-expansion), keeping the filter layer consistent and predictable.
+
+#### Negative
+- The `collectProcessedWorkflowRuns` function required structural refactoring (extraction of `fetchAndProcessLogsBatch` and `handleLogsBatchError`) to accommodate the new code paths, introducing churn to a core pagination loop.
+- The new scheduled workflow consumes AI credits daily (up to 1,500 per run, capped at 5,000 per day), adding recurring cost even when grader health is stable.
+
+#### Neutral
+- The MCP `logs` tool schema gains a `graders` boolean parameter; this is a backwards-compatible addition that does not break existing callers.
+- The `graders` artifact set is automatically included when `--graders` is active, consistent with how `--evals` auto-includes the evals artifact set.
+- The compiled lock file (`.github/workflows/daily-grader-audit.lock.yml`) is generated by `gh aw compile` from the `.md` source; the lock file itself is not hand-maintained.
+
+---
+
+*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

pkg/cli/logs_artifact_set.go

@@ -362,10 +362,30 @@ func applyEvalsArtifact(artifacts []string, evalsOnly bool) []string {
 	return artifacts
 }
 
+// applyGradersArtifact appends the graders artifact set to artifacts when gradersOnly is true
+// and neither ArtifactSetGraders nor ArtifactSetAll is already present.
+func applyGradersArtifact(artifacts []string, gradersOnly bool) []string {
+	if len(artifacts) == 0 {
+		return artifacts
+	}
+	if gradersOnly &&
+		!slices.Contains(artifacts, string(ArtifactSetGraders)) &&
+		!slices.Contains(artifacts, string(ArtifactSetAll)) {
+		return append(artifacts, string(ArtifactSetGraders))
+	}
+	return artifacts
+}
+
 // isEvalsArtifactRequested reports whether evals were explicitly requested,
 // either via --evals or by including --artifacts evals. Callers use this to
 // decide whether to bypass stale cache entries and trigger legacy dedicated-evals
 // fallback downloads when evals.jsonl is missing from usage artifacts.
 func isEvalsArtifactRequested(evalsOnly bool, artifactSets []string) bool {
 	return evalsOnly || slices.Contains(artifactSets, string(ArtifactSetEvals))
 }
+
+// isGradersArtifactRequested reports whether grader artifacts were explicitly requested,
+// either via --graders or by including --artifacts graders.
+func isGradersArtifactRequested(gradersOnly bool, artifactSets []string) bool {
+	return gradersOnly || slices.Contains(artifactSets, string(ArtifactSetGraders))
+}

pkg/cli/logs_artifact_set_test.go

@@ -293,6 +293,32 @@ func TestApplyEvalsArtifact(t *testing.T) {
 	})
 }
 
+func TestApplyGradersArtifact(t *testing.T) {
+	t.Parallel()
+	t.Run("returns empty slice unchanged when artifact list is empty", func(t *testing.T) {
+		t.Parallel()
+		assert.Empty(t, applyGradersArtifact(nil, true))
+		assert.Empty(t, applyGradersArtifact([]string{}, true))
+	})
+
+	t.Run("appends graders when graders requested and artifact list narrowed", func(t *testing.T) {
+		t.Parallel()
+		assert.Equal(t, []string{"usage", "graders"}, applyGradersArtifact([]string{"usage"}, true))
+	})
+
+	t.Run("does not append graders when already present", func(t *testing.T) {
+		t.Parallel()
+		assert.Equal(t, []string{"graders"}, applyGradersArtifact([]string{"graders"}, true))
+	})
+}
+
+func TestIsGradersArtifactRequested(t *testing.T) {
+	t.Parallel()
+	assert.True(t, isGradersArtifactRequested(true, nil))
+	assert.True(t, isGradersArtifactRequested(false, []string{"graders"}))
+	assert.False(t, isGradersArtifactRequested(false, []string{"usage"}))
+}
+
 func TestIsEvalsArtifactRequested(t *testing.T) {
 	t.Parallel()
 	tests := []struct {

pkg/cli/logs_command.go

@@ -63,6 +63,7 @@ const logsCommandExampleTemplate = `  # Basic usage
   %[1]s logs --ref feature-xyz         # Filter logs by feature branch
   %[1]s logs --filtered-integrity      # Filter logs containing items that were filtered by gateway integrity checks
   %[1]s logs --evals                    # Filter logs from workflows with evals results
+  %[1]s logs --graders                  # Filter logs from workflows with grader results
   %[1]s logs --exclude-staged          # Exclude staged workflow runs from results
 
   # Run ID range filtering
@@ -203,6 +204,7 @@ func loadStdinLogsOptions(cmd *cobra.Command) (StdinLogsOptions, error) {
 		SafeOutputType:    values.SafeOutputType,
 		FilteredIntegrity: values.FilteredIntegrity,
 		EvalsOnly:         values.EvalsOnly,
+		GradersOnly:       values.GradersOnly,
 		Train:             values.Train,
 		Format:            values.Format,
 		ReportFile:        values.ReportFile,
@@ -267,6 +269,7 @@ func loadCommonLogsOptions(cmd *cobra.Command) (LogsDownloadOptions, error) {
 		SafeOutputType:    getStringFlag(cmd, "safe-output"),
 		FilteredIntegrity: getBoolFlag(cmd, "filtered-integrity"),
 		EvalsOnly:         getBoolFlag(cmd, "evals"),
+		GradersOnly:       getBoolFlag(cmd, "graders"),
 		Train:             getBoolFlag(cmd, "train"),
 		Format:            getStringFlag(cmd, "format"),
 		ReportFile:        getStringFlag(cmd, "report-file"),
@@ -277,6 +280,7 @@ func loadCommonLogsOptions(cmd *cobra.Command) (LogsDownloadOptions, error) {
 	}
 	if len(options.ArtifactSets) > 0 {
 		options.ArtifactSets = applyEvalsArtifact(options.ArtifactSets, options.EvalsOnly)
+		options.ArtifactSets = applyGradersArtifact(options.ArtifactSets, options.GradersOnly)
 	}
 	return options, nil
 }
@@ -404,6 +408,7 @@ func addLogsCommandFlags(logsCmd *cobra.Command, validArtifactSets string) {
 	logsCmd.Flags().String("safe-output", "", "Filter to runs containing a specific safe output type (e.g., create-issue, missing-tool, missing-data, noop, report-incomplete)")
 	logsCmd.Flags().Bool("filtered-integrity", false, "Filter to runs containing items that were filtered by gateway integrity checks")
 	logsCmd.Flags().Bool("evals", false, "Filter to runs containing evals results (evals.jsonl); automatically includes the usage artifact (which contains evals)")
+	logsCmd.Flags().Bool("graders", false, "Filter to runs containing deterministic grader results; automatically includes grader artifacts")
 	logsCmd.Flags().Bool("parse", false, "Run JavaScript parsers on agent logs and firewall logs, writing Markdown to log.md and firewall.md")
 	addJSONFlag(logsCmd)
 	logsCmd.Flags().Int("timeout", 0, "Download timeout in minutes (0 = no timeout)")

pkg/cli/logs_filtering_test.go

@@ -17,7 +17,7 @@ func TestLogsCommandFlags(t *testing.T) {
 	cmd := NewLogsCommand()
 
 	// Check that all expected flags are present
-	expectedFlags := []string{"count", "start-date", "end-date", "output", "engine", "ref", "before-run-id", "after-run-id", "filtered-integrity"}
+	expectedFlags := []string{"count", "start-date", "end-date", "output", "engine", "ref", "before-run-id", "after-run-id", "filtered-integrity", "graders"}
 
 	for _, flagName := range expectedFlags {
 		flag := cmd.Flags().Lookup(flagName)
@@ -602,3 +602,23 @@ func TestFilteredIntegrityFlag(t *testing.T) {
 		t.Error("Expected 'filtered-integrity' flag to have usage text")
 	}
 }
+
+// TestGradersFlag verifies the --graders flag is registered correctly.
+func TestGradersFlag(t *testing.T) {
+	t.Parallel()
+
+	cmd := NewLogsCommand()
+
+	flag := cmd.Flags().Lookup("graders")
+	if flag == nil {
+		t.Fatal("Expected flag 'graders' not found in logs command")
+	}
+
+	if flag.DefValue != "false" {
+		t.Errorf("Expected 'graders' default to be 'false', got: %s", flag.DefValue)
+	}
+
+	if !strings.Contains(flag.Usage, "grader results") {
+		t.Errorf("Expected 'graders' usage to mention grader results, got: %s", flag.Usage)
+	}
+}