Skip to content

E2E performance report — 2026-09-05 (gh-aw timing) #14187

Description

@dsyme

Run: https://github.com/githubnext/gh-aw-test/actions/runs/33941175759  ·  Generated: 2026-09-05T04:50:26Z

Performance history for copilot-create-issue.md

Run, job & step times (main, using inference)

75 successful runs. Regressions shown below are limited to the last six weeks.

Run and job times for main, using inference

Run or job Samples Median P90
Workflow complete 75 337.0s 440.2s
Workflow start to proxy step 75 111.0s 142.6s
Proxy step to first reasoning/sample 75 21.0s 28.1s
Copilot phase — AWF startup 75 13.9s 18.6s
Copilot phase — harness startup 75 2.2s 4.9s
Copilot phase — Copilot process 75 7.9s 10.6s
Job activation 75 47.0s 73.2s
Job agent 75 89.0s 170.8s
Job detection 75 73.0s 96.0s
Job safe_outputs 75 38.0s 63.2s
Job conclusion 75 42.0s 59.2s
Major step Execute GitHub Copilot CLI 75 28.0s 113.6s
Major step Set up job 75 17.0s 21.6s
Major step Install ripgrep 6 14.0s 19.0s
Major step Download container images 75 11.0s 18.6s
Major step Start MCP Gateway 75 6.0s 11.0s
Major step Install GitHub Copilot CLI 75 4.0s 9.0s
Major step Setup Scripts 74 3.0s 5.0s
Major step Download activation artifact 37 2.0s 2.0s
Major step Upload agent artifacts 25 2.0s 2.0s
Major step Checkout repository 16 2.0s 2.0s
Major step Install AWF binary 5 2.0s 2.0s
Major step Stop MCP Gateway 9 2.0s 2.0s
Major step Print firewall logs 1 2.0s 2.0s
Major step Audit pre-agent workspace 1 2.0s 2.0s
Major step Upload agent output fallback artifact 1 2.0s 2.0s

Major step times for job activation (main, using inference)

Major step times for activation, main, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-25 to 2026-07-27 Set up job 75.0s 29.5s 154% #325 v0.83.3-17-g11d9ea9de7 / 11d9ea9de729
R2 2026-08-08 Set up job 63.0s 34.5s 83% #341 v0.86.1-57-gba0a9f9589 / ba0a9f958976
R3 2026-08-08 Setup Scripts 18.0s 3.0s 500% #341 v0.86.1-57-gba0a9f9589 / ba0a9f958976
R4 2026-08-17 Set up job 48.0s 31.5s 52% #352 v0.87.0-133-g2b2cf3fb01 / 2b2cf3fb01ee

Major step times for job agent (main, using inference)

Major step times for agent, main, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-07 Download container images 24.0s 11.0s 118% #340 v0.86.1-3-ge1e298d64b / e1e298d64bfa
R2 2026-08-07 Execute GitHub Copilot CLI 38.0s 22.5s 69% #340 v0.86.1-3-ge1e298d64b / e1e298d64bfa
R3 2026-08-14 Copilot phase — AWF startup 23.2s 12.9s 80% #349 v0.86.2-73-gc35faf436c / c35faf436c79
R4 2026-08-14 Execute GitHub Copilot CLI 38.0s 23.0s 65% #349 v0.86.2-73-gc35faf436c / c35faf436c79
R5 2026-09-02 Copilot phase — AWF startup 27.9s 14.9s 87% #470 bde2c79aa0 / bde2c79aa0d6
R6 2026-09-02 Download container images 24.0s 9.5s 153% #470 bde2c79aa0 / bde2c79aa0d6
R7 2026-09-02 Execute GitHub Copilot CLI 47.0s 24.0s 96% #470 bde2c79aa0 / bde2c79aa0d6
R8 2026-09-05 Install GitHub Copilot CLI 20.0s 9.0s 122% #481 5473143ca3 / 5473143ca3ae

Major step times for job detection (main, using inference)

Major step times for detection, main, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-18 Execute GitHub Copilot CLI 50.0s 31.5s 59% #353 v0.87.1-4-g4845f00caf / 4845f00caf46
R2 2026-08-24 Download container images 12.0s 1.0s 1100% #437 5f0cc8dcc8 / 5f0cc8dcc819

Major step times for job safe_outputs (main, using inference)

Major step times for safe_outputs, main, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-25 Set up job 61.0s 25.5s 139% #325 v0.83.3-17-g11d9ea9de7 / 11d9ea9de729
R2 2026-08-01 to 2026-08-03 Set up job 89.0s 34.5s 158% #334 v0.84.3-58-g53baccde53 / 53baccde5390
R3 2026-08-03 Setup Scripts 41.0s 4.5s 811% #334 v0.84.3-58-g53baccde53 / 53baccde5390
R4 2026-08-14 to 2026-08-16 Set up job 69.0s 29.0s 138% #349 v0.86.2-73-gc35faf436c / c35faf436c79
R5 2026-08-25 Download agent output artifact 12.0s 1.0s 1100% #445 818e3a3863 / 818e3a386376

Major step times for job conclusion (main, using inference)

Major step times for conclusion, main, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-30 to 2026-08-02 Set up job 59.0s 29.5s 100% #333 v0.84.2-95-gf5bd99245e / f5bd99245e40
R2 2026-08-03 Setup Scripts 14.0s 4.0s 250% #334 v0.84.3-58-g53baccde53 / 53baccde5390
R3 2026-08-17 Upload usage artifact 11.0s 1.0s 1000% #352 v0.87.0-133-g2b2cf3fb01 / 2b2cf3fb01ee
R4 2026-08-19 Set up job 48.0s 28.0s 71% #354 v0.87.1-44-g5d5e0af5c4 / 5d5e0af5c46c

Run, job & step times (released, using inference)

37 successful runs. Regressions shown below are limited to the last six weeks.

Run and job times for released, using inference

Run or job Samples Median P90
Workflow complete 37 220.0s 332.6s
Workflow start to proxy step 37 63.0s 90.0s
Proxy step to first reasoning/sample 34 21.4s 28.8s
Copilot phase — AWF startup 34 13.5s 18.5s
Copilot phase — harness startup 34 1.9s 5.0s
Copilot phase — Copilot process 34 7.8s 8.8s
Job activation 37 17.0s 43.2s
Job agent 37 74.0s 157.4s
Job detection 37 54.0s 78.2s
Job safe_outputs 37 12.0s 17.6s
Job conclusion 37 15.0s 21.4s
Major step Execute GitHub Copilot CLI 37 30.0s 115.4s
Major step Download container images 37 12.0s 17.4s
Major step Install ripgrep 3 12.0s 14.4s
Major step Start MCP Gateway 37 6.0s 9.4s
Major step Install GitHub Copilot CLI 37 4.0s 9.4s
Major step Set up job 35 3.0s 4.0s
Major step Setup Scripts 34 3.0s 4.0s
Major step Stop MCP Gateway 6 2.0s 2.0s
Major step Install AWF binary 3 2.0s 2.0s
Major step Download activation artifact 10 2.0s 2.0s
Major step Upload agent artifacts 7 2.0s 2.0s
Major step Checkout repository 10 2.0s 2.1s

Major step times for job activation (released, using inference)

Major step times for activation, released, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-31 Checkout .github and .agents folders 17.0s 2.0s 750% #462 v0.87.10 / ff62cdbec362
R2 2026-08-31 Set up job 83.0s 3.0s 2667% #462 v0.87.10 / ff62cdbec362
R3 2026-08-31 Setup Scripts 16.0s 3.0s 433% #462 v0.87.10 / ff62cdbec362

Major step times for job agent (released, using inference)

Major step times for agent, released, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-15 Copilot phase — AWF startup 23.5s 13.2s 79% #432 v0.86.3 / 6062cd2238b6
R2 2026-08-31 Copilot phase — AWF startup 35.6s 14.7s 142% #462 v0.87.10 / ff62cdbec362
R3 2026-08-31 Copilot phase — harness startup 13.0s 2.2s 495% #462 v0.87.10 / ff62cdbec362
R4 2026-08-31 Download container images 46.0s 9.5s 384% #462 v0.87.10 / ff62cdbec362
R5 2026-08-31 Execute GitHub Copilot CLI 62.0s 25.0s 148% #462 v0.87.10 / ff62cdbec362
R6 2026-08-31 Set up job 29.0s 3.0s 867% #462 v0.87.10 / ff62cdbec362
R7 2026-08-31 Start MCP Gateway 23.0s 6.5s 254% #462 v0.87.10 / ff62cdbec362

Major step times for job detection (released, using inference)

Major step times for detection, released, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-07 Install GitHub Copilot CLI 15.0s 4.0s 275% #430 v0.86.1 / 475927dfc6d1
R2 2026-08-15 Execute GitHub Copilot CLI 36.0s 23.5s 53% #432 v0.86.3 / 6062cd2238b6
R3 2026-08-31 Set up job 18.0s 2.0s 800% #462 v0.87.10 / ff62cdbec362

Major step times for job safe_outputs (released, using inference)

Major step times for safe_outputs, released, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-07 Setup Scripts 20.0s 2.0s 900% #430 v0.86.1 / 475927dfc6d1
R2 2026-08-31 Set up job 37.0s 3.0s 1133% #462 v0.87.10 / ff62cdbec362

Major step times for job conclusion (released, using inference)

Major step times for conclusion, released, using inference

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-08-31 Set up job 27.0s 2.5s 980% #462 v0.87.10 / ff62cdbec362

Run, job & step times (main, using samples)

69 successful runs. Regressions shown below are limited to the last six weeks.

Run and job times for main, using samples

Run or job Samples Median P90
Workflow complete 69 194.0s 246.2s
Workflow start to proxy step 69 98.0s 122.2s
Proxy step to first reasoning/sample 69 0.0s 0.2s
Job activation 69 46.0s 65.4s
Job agent 69 45.0s 57.0s
Job detection 0 n/a n/a
Job safe_outputs 69 31.0s 44.2s
Job conclusion 69 36.0s 55.2s
Major step Download container images 69 9.0s 12.2s
Major step Set up job 69 7.0s 12.0s
Major step Start MCP Gateway 69 6.0s 11.0s
Major step Install GitHub Copilot CLI 69 4.0s 7.4s
Major step Setup Scripts 68 2.0s 4.3s
Major step Checkout repository 11 2.0s 2.0s
Major step Install AWF binary 3 2.0s 2.0s
Major step Download activation artifact 22 2.0s 2.0s
Major step Upload agent artifacts 13 2.0s 2.0s
Major step Stop MCP Gateway 7 2.0s 2.0s
Major step Upload agent output fallback artifact 1 2.0s 2.0s

Major step times for job activation (main, using samples)

Major step times for activation, main, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-27 Set up job 46.0s 28.0s 64% #266 55b5181bb8 / 55b5181bb855
R2 2026-09-01 Set up job 71.0s 32.0s 122% #467 4a88fd99c3 / 4a88fd99c35f
R3 2026-09-05 Set up job 73.0s 33.0s 121% #482 5473143ca3 / 5473143ca3ae

Major step times for job agent (main, using samples)

Major step times for agent, main, using samples

Candidate regressions (last six weeks)

No candidate regressions in the last six weeks.

Major step times for job detection (main, using samples)

Major step times for detection, main, using samples

Candidate regressions (last six weeks)

No candidate regressions in the last six weeks.

Major step times for job safe_outputs (main, using samples)

Major step times for safe_outputs, main, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-29 to 2026-08-02 Set up job 37.0s 20.5s 80% #279 a9137e1445 / a9137e144504
R2 2026-08-24 Set up job 54.0s 27.5s 96% #438 5f0cc8dcc8 / 5f0cc8dcc819
R3 2026-09-04 to 2026-09-05 Set up job 48.0s 24.5s 96% #478 76182db3ee / 76182db3eedf

Major step times for job conclusion (main, using samples)

Major step times for conclusion, main, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-27 to 2026-07-29 Set up job 49.0s 17.0s 188% #270 acc797bbab / acc797bbab36
R2 2026-08-28 Setup Scripts 15.0s 3.0s 400% #454 76f6ea7c22 / 76f6ea7c2220
R3 2026-09-04 Set up job 42.0s 24.5s 71% #478 76182db3ee / 76182db3eedf

Run, job & step times (released, using samples)

114 successful runs. Regressions shown below are limited to the last six weeks.

Run and job times for released, using samples

Run or job Samples Median P90
Workflow complete 114 119.5s 161.7s
Workflow start to proxy step 114 64.0s 95.1s
Proxy step to first reasoning/sample 114 0.0s 1.0s
Job activation 114 17.5s 34.8s
Job agent 114 41.0s 57.7s
Job detection 0 n/a n/a
Job safe_outputs 114 12.0s 20.7s
Job conclusion 114 16.0s 22.7s
Major step Download container images 114 9.0s 16.7s
Major step Install ripgrep 8 9.0s 18.0s
Major step Start MCP Gateway 114 7.0s 12.0s
Major step Install GitHub Copilot CLI 114 4.0s 10.0s
Major step Set up job 96 3.0s 5.5s
Major step Setup Scripts 103 2.0s 4.0s
Major step Install AWF binary 10 2.0s 2.1s
Major step Download activation artifact 38 2.0s 2.0s
Major step Upload agent artifacts 23 2.0s 2.0s
Major step Stop MCP Gateway 14 2.0s 2.0s
Major step Checkout repository 7 2.0s 2.0s
Major step Upload agent output fallback artifact 1 2.0s 2.0s

Major step times for job activation (released, using samples)

Major step times for activation, released, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-23 to 2026-07-27 Set up job 21.0s 3.0s 600% #264 v0.83.4 / bbb804287845
R2 2026-07-25 Setup Scripts 18.0s 2.5s 620% #268 v0.83.3 / 7e728ffd6fc9
R3 2026-08-25 Setup Scripts 14.0s 2.5s 460% #452 v0.87.5 / 654cf351a595
R4 2026-08-31 Set up job 58.0s 3.0s 1833% #463 v0.87.10 / ff62cdbec362
R5 2026-08-31 Setup Scripts 21.0s 2.0s 950% #463 v0.87.10 / ff62cdbec362
R6 2026-09-03 Set up job 14.0s 3.0s 367% #475 v0.88.2 / 8e30bcd8897f

Major step times for job agent (released, using samples)

Major step times for agent, released, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-24 to 2026-07-25 Download container images 24.0s 8.0s 200% #263 v0.83.3 / 7e728ffd6fc9
R2 2026-07-27 Set up job 28.0s 3.0s 833% #269 v0.83.4 / bbb804287845
R3 2026-08-11 Download container images 27.0s 12.0s 125% #398 v0.86.2 / 48e5fa3ff522
R4 2026-08-11 Install GitHub Copilot CLI 15.0s 4.0s 275% #398 v0.86.2 / 48e5fa3ff522
R5 2026-08-22 Install GitHub Copilot CLI 20.0s 9.0s 122% #399 v0.87.4 / 83d6315352f7
R6 2026-08-31 Download container images 20.0s 9.0s 122% #465 v0.87.10 / ff62cdbec362
R7 2026-08-31 Install GitHub Copilot CLI 28.0s 9.5s 195% #468 v0.87.10 / ff62cdbec362
R8 2026-08-31 Set up job 13.0s 2.0s 550% #463 v0.87.10 / ff62cdbec362

Major step times for job detection (released, using samples)

Major step times for detection, released, using samples

Candidate regressions (last six weeks)

No candidate regressions in the last six weeks.

Major step times for job safe_outputs (released, using samples)

Major step times for safe_outputs, released, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-25 Setup Scripts 20.0s 2.0s 900% #268 v0.83.3 / 7e728ffd6fc9
R2 2026-07-27 Set up job 29.0s 3.0s 867% #264 v0.83.4 / bbb804287845
R3 2026-08-31 Set up job 14.0s 2.0s 600% #463 v0.87.10 / ff62cdbec362
R4 2026-09-03 Set up job 18.0s 2.5s 620% #475 v0.88.2 / 8e30bcd8897f

Major step times for job conclusion (released, using samples)

Major step times for conclusion, released, using samples

Candidate regressions (last six weeks)

Label Episode Step Peak Prior median Increase Run gh-aw version / commit
R1 2026-07-27 Set up job 16.0s 2.5s 540% #264 v0.83.4 / bbb804287845
R2 2026-08-31 Set up job 67.0s 2.0s 3250% #463 v0.87.10 / ff62cdbec362
R3 2026-09-03 Set up job 34.0s 2.0s 1600% #475 v0.88.2 / 8e30bcd8897f

Method

Each section fixes both independent dimensions: gh-aw source (main or combined stable/pre-release released) and execution mode (inference or samples). Only overall-successful workflow_dispatch runs with a successful agent job are included. Candidate regression baselines use up to ten preceding observations from the same section and step; displayed regression episodes are limited to the six weeks before report generation. A step is graphed when it has a sustained cost, recent slowdown, or recent regression. Runs with missing compiler metadata remain in CSV/JSON but are excluded from graphs.

For inference runs, Execute GitHub Copilot CLI is additionally split using timestamped runtime markers: AWF startup is step start to the AWF agent-container entrypoint, harness startup is that entrypoint to the first Copilot process start, and Copilot process is the first process start through the final process close (including retries and retry delays). Cleanup after process close remains visible only in the full step duration, while unavailable markers produce no phase value.

Previous performance reports (closed by this run)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    e2e-status-reportAutomated cross-ref E2E status report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions