[daily regulatory] Regulatory Report - 2026-05-03

[github-actions[bot]]

This regulatory audit reviewed 15 daily report discussions created on 2026-05-03 for the github/gh-aw repository. Overall data quality is good — most reports are internally consistent, with well-structured metrics and clear scope documentation. Three notable cross-report discrepancies were identified, of which one (PR count inconsistency between Team Evolution and Copilot Agent Analysis) merits follow-up investigation. The most significant operational concern remains the Smoke Gemini 100% failure (ongoing 30+ days), repeatedly flagged across multiple reports with no visible remediation progress.

The previous regulatory report (#29835, 2026-05-02) is superseded by this report. Note: the close_discussion safe-output tool is not available in this environment; prior reports cannot be automatically closed.

📋 Full Regulatory Report

📊 Reports Reviewed

#	Report	Discussion	Created (UTC)	Status
1	Daily Performance Summary	#30012	21:12	✅ Valid
2	Daily Team Evolution Insights	#30006	20:28	⚠️ See notes
3	Daily Code Metrics	#29994	19:06	✅ Valid
4	Daily Copilot Agent Analysis	#29990	18:34	⚠️ See notes
5	Daily Secrets Analysis	#29977	17:57	✅ Valid
6	Security Observability	#29965	16:51	⚠️ Scope diff
7	Agent Performance Report	#29921	13:04	✅ Valid
8	Daily Experiment Report	#29903	08:41	✅ Valid
9	Copilot Session Insights	#29897	08:02	✅ Valid
10	Safe Output Health	#29882	05:32	✅ Valid
11	Daily Compiler Code Quality	#29867	03:44	✅ Valid
12	Daily Firewall Report	#29861	03:39	⚠️ Scope diff
13	Auto-Triage Report	#29849	01:20	✅ Valid
14	Daily Sentrux Report	#29847	00:11	✅ Valid
15	Observability Coverage	#29846	00:09	✅ Valid

🔍 Data Consistency Analysis

Cross-Report Metrics Comparison

Metric	Source	Value	Scope	Status
Open Issues	Daily Performance	100	90-day window	✅
Open Issues	Team Evolution	30 active	Last 24h subset	i️ Different scope
PRs Merged (90d)	Daily Performance	82	90d	✅
PRs Merged (today)	Team Evolution	19	Last 24h	✅ Consistent trend
PRs Today (all)	Team Evolution	28 opened	Last 24h	⚠️ See discrepancy
PRs Today (Copilot)	Copilot Agent Analysis	54 total	Last 24h, Copilot-only	⚠️ See discrepancy
Avg PR merge time	Daily Performance	1.7h	90d avg	✅
Avg PR merge time	Team Evolution	1.6h	Today	✅ Consistent
Firewall block rate	Daily Firewall Report	46.7%	7d, 15 runs	⚠️ Scope diff
Firewall block rate	Security Observability	2.2%	7d, 11 runs (filtered)	⚠️ Scope diff
AWF coverage	Observability Coverage	98.6%	24h snapshot	✅
MCP coverage	Observability Coverage	100%	24h snapshot	✅
Safe output success	Safe Output Health	87%	Last 24h	✅
Workflows w/ secrets	Daily Secrets	209	All compiled	✅
Redaction coverage	Daily Secrets	100%	All workflows	✅

Consistency Score

• Overall Consistency: ~87% (13 of 15 metrics agree within scope)
• Critical Discrepancies: 1 (PR count inconsistency)
• Scope-based differences (expected): 2 (firewall block rates)
• Minor notes: 2

⚠️ Issues and Anomalies

Critical Issues

1. PR Count Discrepancy — Team Evolution vs Copilot Agent Analysis
   • Affected Reports: [daily-team-evolution] 🌱 Daily Team Evolution Insights - 2026-05-03 #30006 (Team Evolution), [copilot-agent-analysis] Daily Copilot Agent Analysis - 2026-05-03 #29990 (Copilot Agent Analysis)
   • Metric: PRs opened on 2026-05-03
   • Team Evolution: 28 PRs opened today
   • Copilot Agent Analysis: 54 Copilot-only PRs today (38 merged, 10 closed, 6 open)
   • Issue: Copilot PRs (54) exceed total PRs (28) — logically impossible if Copilot PRs are a subset
   • Scope Analysis: Team Evolution may source data from git commit history or a filtered view; Copilot Agent Analysis queries PR API with author:app/github-copilot. Possible explanation: Team Evolution counts PRs from a time-bounded window that doesn't cover the full day, while Copilot Agent Analysis includes PRs from prior days that are still "active" today.
   • Severity: Medium
   • Recommended Action: Align data sources and time window definitions between Team Evolution and Copilot Agent Analysis reports. Both should query the GitHub PR API with identical date ranges.

Warnings

1. Smoke Gemini 100% Failure — Ongoing 30+ Days

   • Details: Smoke Gemini workflow has a 100% failure rate due to API_KEY_INVALID. Active issues [aw-failures] [P0] Smoke Engine Failures: Gemini API_KEY_INVALID + Crush EROFS Install #29459 and [aw-failures] [aw] P0: Smoke Gemini — Gemini CLI proxy architecture blocks all agent traffic (localhost:8080 not reachable) #29852 are open. No remediation progress visible across multiple daily reports.
   • Impact: Gemini engine is untested. Risk of undetected regressions in Gemini-based workflows.

2. Firewall Block Rate Apparent Discrepancy

   • Details: Daily Firewall Report shows 46.7% block rate (15 runs, 7d); Security Observability shows 2.2% (11 workflows, 7d). Both cover the same period.
   • Root Cause: Firewall Report includes Smoke Gemini (301 localhost blocks representing 86% of all blocks); Security Observability appears to exclude these or use a different workflow filter.
   • Impact: Low — discrepancy is explainable, but report consumers may be confused.
   • Recommended Action: Both reports should document their workflow inclusion criteria explicitly.

3. Agent Average Duration Spike

   • Details: Copilot Agent Analysis reports avg duration jumped from 44 min (May 2) to 137 min (May 3) — a 211% increase.
   • Impact: Potential cost increase; may indicate workflow complexity growth or CI infrastructure issues.

4. Sentrux Quality Signal Below Threshold

   • Details: First run shows quality signal of 5249/10000 (healthy threshold: 8000). 821 complex functions detected.
   • Impact: High complexity functions increase refactoring risk. No baseline for trend comparison.

Data Quality Notes

• The Observability Coverage report notes its analysis window timed out on the initial 100-run fetch; results are from a partial artifact set (2026-05-02T18:04–23:57 UTC window only). Coverage metrics may not reflect the full day.
• The Daily Sentrux Report is a first-run baseline — no week-over-week comparison is possible.
• The Experiment Report notes that the caveman experiment's no variant has 0 samples due to branch-scoped cache preventing state sharing. Experiment is structurally blocked from completing.

📈 Trend Analysis

Key Metrics — Day over Day

Metric	May 3	May 2	Change
Copilot PRs	54	60	-10%
Copilot PRs merged	38	48	-21%
Copilot success rate	79.2%	81.4%	-2.2pp
Copilot avg duration	137 min	44 min	+211% ⚠️
Safe output success	87%	—	—

Notable Trends

• PR merge velocity remains high but showing slight day-over-day decline in Copilot agent success rate
• Average agent duration spiked significantly on May 3 — likely related to more complex changes (796-line additions noted in Copilot session insights)
• Codebase LOC growing at ~7,700 lines/day on average — rapid feature development pace

📝 Per-Report Analysis

Daily Performance Summary (#30012)

Time Period: Last 90 days (90d window ending 2026-05-03)
Quality: ✅ Valid

Metric	Value
Total PRs	100
Merged PRs	82 (82%)
Open PRs	1
Total Issues	1,000
Open Issues	100 (10%)
Avg PR Merge Time	1.7h
Unique Contributors	3

Notes: Internally consistent. 0% discussion answer rate noted — discussions are used for general conversation, not Q&A.

Daily Firewall Report (#29861)

Time Period: Last 7 days (2026-04-26 to 2026-05-03)
Quality: ⚠️ Scope clarification needed

Metric	Value
Runs analyzed	15
Total requests	752
Allowed	401 (53.3%)
Blocked	351 (46.7%)
Unique blocked domains	11

Notes: High block rate is dominated by Smoke Gemini localhost traffic (301 of 351 blocks). Firewall is working correctly; the metric is misleading without context about Smoke Gemini's expected failure pattern.

Security Observability (#29965)

Time Period: Last 7 days
Quality: ⚠️ Scope differs from Firewall Report

Metric	Value
Firewall-enabled workflows	11
Total requests	91
Blocked	2 (2.2%)
DIFC filtered events	3

Notes: Scope is narrower than the Daily Firewall Report — excludes Smoke Gemini's high-volume localhost blocks.

Observability Coverage (#29846)

Time Period: 2026-05-02T18:04–23:57 UTC (partial)
Quality: ✅ Valid (with caveats)

Metric	Value
Runs analyzed	81
AWF Firewall coverage	98.6% (68/69)
MCP coverage	100% (77/77)
Missing firewall log	1 (Smoke CI)

Safe Output Health (#29882)

Time Period: Last 24 hours
Quality: ✅ Valid

Metric	Value
Workflow runs	48
Safe output jobs	23
Failed	3 (13%)
Success rate	87%
Error clusters	2

Notes: Failures are explained and root-caused (branch deletion race condition, missing permission). No systemic issues.

Daily Copilot Agent Analysis (#29990)

Time Period: Last 24 hours
Quality: ⚠️ PR count inconsistency

Metric	Value
Total Copilot PRs	54
Merged	38 (70.4%)
Avg duration	137 min
Success rate	79.2%

Notes: PR total (54) exceeds Team Evolution's reported total PRs opened (28) — see Critical Issue #1.

Daily Sentrux Report (#29847)

Time Period: First run (no baseline)
Quality: ✅ Valid

Metric	Value
Quality signal	5249/10000
Complex functions	821
Architectural cycles	2
Files analyzed	4,335

💡 Recommendations

Process Improvements

1. Standardize PR count data sources: Both Team Evolution and Copilot Agent Analysis should use identical GitHub API queries with the same date ranges to eliminate inconsistencies.
2. Document scope explicitly in all reports: Each report should include a "Data Scope" section declaring time window, filters applied, and any exclusions (e.g., Smoke Gemini excluded from block rate calculation).
3. Add cross-report metric IDs: Reference scratchpad/metrics-glossary.md metric codes in report tables to enable automated cross-report validation.

Data Quality Actions

1. Fix experiment state sharing: The caveman experiment on smoke-copilot is structurally blocked due to branch-scoped cache. Implement a shared cache strategy (e.g., repository-level cache key) so variant counts persist across PR branches.
2. Add Sentrux historical baseline: Now that the first run exists, ensure the report runs daily to build trend data. Set alert thresholds at 8000 (healthy) and 4000 (critical).
3. Extend Observability Coverage window: The timeout on the 100-run fetch suggests the report needs optimization or pagination to capture a full 24h window reliably.

Workflow Suggestions

1. Escalate Smoke Gemini API key: The 30+ day ongoing failure is a significant gap in test coverage. Escalate the API key renewal immediately.
2. Investigate agent duration spike: The 211% increase in Copilot agent duration from May 2 to May 3 warrants investigation to identify whether it reflects legitimate complexity or infrastructure degradation.

📊 Regulatory Metrics

Metric	Value
Reports Reviewed	15
Reports Passed	11
Reports with Issues/Caveats	4
Reports Failed	0
Critical Discrepancies	1
Warnings	4
Overall Health Score	87%

---

Report generated automatically by the Daily Regulatory workflow
Data sources: Daily report discussions from github/gh-aw (2026-05-03)
Metric definitions: scratchpad/metrics-glossary.md
Previous report: #29835 (2026-05-02)
Note: close_discussion safe-output is unavailable; prior reports were not auto-closed.

References:

• Workflow Run §25291265701

Generated by Daily Regulatory Report Generator · ● 1.5M · ◷

• expires on May 6, 2026, 9:33 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Daily Regulatory Report Generator.

A newer discussion is available at Discussion #30227.