Skip to content

Add configurable company stress testing and PR workload calculator - #19972

Merged
pelikhan merged 5 commits into
mainfrom
pelikhan/company-stress-testing
Oct 30, 2026
Merged

pelikhan merged 5 commits into
mainfrom
pelikhan/company-stress-testing

Conversation

@pelikhan

@pelikhan pelikhan commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Company-scale workload planning needs a fast way to exercise real ingestion without waiting for thousands of agent executions. This adds configurable synthetic company scenarios and a side-effect-free dashboard calculator for PR-driven AW volume.

Approach

  • Stream completed-AW-shaped evidence directly into the existing hash-verified ingestion pipeline, then measure PostgreSQL retention and uncached HTTP queries. Agent execution, completion callbacks, collection subprocesses, webhooks, and GitHub quotas are deliberately outside this benchmark.
  • Provide small, medium, and large presets, JSON scenarios, and CLI overrides for repository/developer counts, PR rate, AW fan-out, duration, AIC per run, seed, and timestamp distribution. The large preset runs daily on main at 06:37 UTC through the existing full-validation workflow, with its outcome included in the aggregate daily report. Manual configuration remains available; the small smoke case joins existing service-backed CI.
  • Add Simulators > Pull request using Dashboard Language queries and existing controls, with worker/server parity coverage and experimental-page gating. Repository count distributes activity rather than multiplying PR volume.

Requested scenario

The large preset models 10,000 repositories, 10,000 developers, 10 PRs/developer/day, five AW runs/PR, and 100 AIC/run: 100,000 PRs, 500,000 AW records, and 50 million modeled AIC per day.

The full 500,000-record run completed successfully. Production retention remains unchanged: 200,000 runs are retained, 300,000 are evicted, and query provenance correctly reports partial coverage. Reports distinguish generated volume, retained evidence, modeled credits, and measured timings; no capacity or billing guarantee is inferred. Disposable enterprise PostgreSQL instances need additional lock-table headroom, configured in the reusable stress workflow.

POSTGRES_URL='postgres://postgres@127.0.0.1:5432/cao_test?sslmode=disable' \
  npm run test:stress:company -- --preset large

Validation

  • Full-size real PostgreSQL run, multi-day burst and zero-volume cases, replay/cancellation checks, and focused race-enabled Go tests.
  • 403 dashboard unit suites passed: 3,887 tests passed, one skipped.
  • Dashboard browser regressions covered all 309 cases; four contention-related timeouts passed on a bounded-worker rerun. Four additional populated full-site desktop/mobile checks passed and captured screenshots.
  • Root/dashboard lint, typechecks, Go vet/format checks, dashboard/corpus validation, focused workflow/launcher/build contracts, workflow compilation, and documentation build passed.
  • Daily scheduling, large-preset selection, main-revision gating, reusable-workflow permissions, and aggregate report coverage are covered by 24 focused passing contract tests.
  • Lighthouse completed all scenarios but the CFO full-data desktop CLS budget failed at 0.118 versus 0.1 for cold and PWA runs. An isolated build of unchanged HEAD reproduces both failures identically; this PR does not change that baseline issue.

Desktop and mobile screenshots are attached in a PR comment.

Generate synthetic AW evidence for configurable company workloads, exercise real PostgreSQL ingestion and query paths, and expose a declarative dashboard calculator with opt-in CI coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator Author

Populated full-site Pull request calculator at desktop (1440px) and mobile (390px) widths, using the default 10,000-developer workload. These are screenshots of the rendered dashboard, not mocked page content.

Desktop Pull request calculator with modeled company workload totals

Mobile Pull request calculator with modeled company workload totals

@github-actions

Copy link
Copy Markdown
Contributor

GitHub Actions lint failures

Failed scanners: actionlint

Workflow run: https://github.com/githubnext/gh-cao/actions/runs/41065495475

Generated .lock.yml files were excluded.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor
Dashboard dead views

Dead element views (0):
Dead dashboard pages (0):
Analyzed 8 element renderers and 70 dashboard pages across 1 dashboard.json file(s) from dashboard/site/src/components/ui-elements.js.

⚠️ This reachability report is informational and does not block the pull request.
🔎 View the workflow run and report artifact.

Reuse the existing main-only daily schedule and include company stress results in the aggregate report while preserving manual scenario configuration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

GitHub Actions lint failures

Failed scanners: actionlint

Workflow run: https://github.com/githubnext/gh-cao/actions/runs/41065802253

Generated .lock.yml files were excluded.

Use a Bash arithmetic loop so the retry counter is consumed while preserving the 60-attempt readiness bound.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review October 30, 2026 13:47
@pelikhan
pelikhan requested a review from mnkiefer as a code owner October 30, 2026 13:47
@pelikhan

Copy link
Copy Markdown
Collaborator Author

Full-site calculator regression evidence after separating company repository parameters from the database simulator. Both desktop (1440px) and mobile (390px) Playwright scenarios passed against the populated synthetic Pages deployment.

Company calculator at desktop width using synthetic data

Company calculator at mobile width using synthetic data

pelikhan and others added 2 commits October 30, 2026 06:59
Assign the company stress contracts to tooling so fail-closed area selection can run. Separate company repository inputs from database simulator defaults, regenerate the agent catalog, and cover both default bindings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use root-confined fixture and generated-evidence reads and compare cancellation errors with errors.Is.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan merged commit bb53a9b into main Oct 30, 2026
50 checks passed
@pelikhan
pelikhan deleted the pelikhan/company-stress-testing branch October 30, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant