Skip to content

test(e2e): stabilize real flow run - #401

Draft
rksharma-owg wants to merge 1 commit into
vxcontrol:mainfrom
rksharma-owg:fix/e2e-network-change-race
Draft

test(e2e): stabilize real flow run#401
rksharma-owg wants to merge 1 commit into
vxcontrol:mainfrom
rksharma-owg:fix/e2e-network-change-race

Conversation

@rksharma-owg

Copy link
Copy Markdown

Description of the Change

Problem

The scheduled Tier-2 flow test is intermittent on unchanged commits. In the September 1 run, both attempts created and ran their backend flows, but Chromium aborted the in-flight createFlow GraphQL request with net::ERR_NETWORK_CHANGED when Docker created the nested sandbox network endpoint. The page remained on /flows/new and timed out.

A clean local run also exposed the adjacent fixture race: the deterministic mock can issue its terminal command and final messages before the newly routed flow page has attached its live subscriptions. This leaves persisted input visible while the streamed stdout or final result is absent.

Solution

  • Run only the fixed Tier-2 worker sandbox with PentAGI's existing DOCKER_NETWORK=host mode, avoiding bridge-interface creation while Chromium is connected.
  • Let mock rules declare a bounded response delay.
  • Delay the scenario's only terminal command by one second so the flow page can route and attach its log subscriptions before the deterministic transcript completes.

Production Compose defaults, CLI/API behavior, schemas, dependencies, and persisted data are unchanged.

No issue is closed by this PR. I found no open issue or PR covering this nightly failure.

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Configuration change
  • 🧪 Test update
  • 🛡️ Security update

Areas Affected

  • Core Services (Frontend UI/Backend API)
  • AI Agents (Researcher/Developer/Executor)
  • Security Tools Integration
  • Memory System (Vector Store/Knowledge Base)
  • Monitoring Stack (Grafana/OpenTelemetry)
  • Analytics Platform (Langfuse)
  • External Integrations (LLM/Search APIs)
  • Documentation
  • Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: main @ ea665308baaff015b226f308438a68d929d0f29b
Docker Version: 29.6.2; Compose v5.3.1
Host OS: macOS arm64 (Darwin)
Node: 24.17.0
pnpm: 11.8.0
Go: 1.26.5
LLM Provider: deterministic Tier-2 mock
Enabled Features: none

Test Steps

  1. Run CI=1 ./frontend/e2e/tools/run-local-tier.sh --grep 'runs a flow end-to-end' on unmodified main and inspect the failed Playwright traces and Compose log.
  2. Apply this change and repeat the focused run from a fresh stack.
  3. Run the full Tier-2, frontend unit, browser, formatting, lint, type-check, build, Compose validation, and cross-build checks listed below.

Test Results

  • Baseline focused flow run: failed on both attempts; the backend completed the flow while the UI missed the final streamed result.
  • Focused flow run after the fix: 4 consecutive fresh-stack runs passed (2/2 setup + flow tests each time; flow assertion ~5.1s).
  • Full Tier-2: 13/13 passed.
  • pnpm run prettier: passed.
  • pnpm run lint: passed with zero warnings.
  • pnpm run typescript: passed.
  • pnpm run test: 78 files, 1,365/1,365 tests passed when run alone. One timing-sensitive WebSocket-mock test failed during an earlier concurrent gate run and passed on the isolated rerun.
  • pnpm run build: passed.
  • Full mock Playwright tier: 181/182 passed; the unrelated template-create test timed out once. Its isolated rerun passed 3/3.
  • docker compose ... config --quiet: passed; resolved DOCKER_NETWORK: host.
  • Linux/amd64 and Linux/arm64 CGO_ENABLED=0 backend builds: passed.
  • CGO_ENABLED=0 go test ./pkg/docker -run '^TestRunContainerHostNetworkSkipsPortPublishing$' -count=1: passed.
  • Full native macOS go test ./...: not clean because the existing github.com/shoenig/go-m1cpu CGo initializer crashes before affected packages run; disabling CGo then exposes expected go-sqlite3 stub failures and missing untracked installer assets. No Go files are changed here.

Security Considerations

host networking reduces isolation for the worker container, so the override is deliberately confined to docker-compose.e2e.yml. The Tier-2 scenario uses a fixed prompt, a local deterministic mock, a dummy provider key, and executes only uname -a; the runner also blocks .env loading and cleans its sandbox afterward. Production workers retain their existing network configuration. No credentials, external model calls, or new dependencies are introduced.

Performance Impact

The mock-only terminal rule waits one second. This adds about one second to the local Tier-2 flow path; it does not affect production. The fixed flow assertion completed in about 5.1 seconds versus 90-second assertion timeouts before the change.

Documentation Updates

No user documentation changes are required because commands and production configuration are unchanged. Inline comments document both non-obvious test choices.

  • README.md updates
  • API documentation updates
  • Configuration documentation updates
  • GraphQL schema updates
  • Other: inline E2E rationale only

Deployment Notes

None. This only changes the isolated Tier-2 test override and mock fixture.

Checklist

Code Quality

  • My code follows the project's coding standards
  • I have added/updated necessary documentation
  • I have added tests to cover my changes
  • All new and existing tests pass
  • I have run go fmt and go vet (for Go code)
  • I have run pnpm run lint (for TypeScript/JavaScript code)

Security

  • I have considered security implications
  • Changes maintain or improve the security model
  • Sensitive information has been properly handled

Compatibility

  • Changes are backward compatible
  • Breaking changes are clearly marked and documented
  • Dependencies are properly updated

Documentation

  • Documentation is clear and complete
  • Comments are added for non-obvious code
  • API changes are documented

Additional Notes

The public failing scheduled run used the same main SHA as alternating successful runs: https://github.com/vxcontrol/pentagi/actions/runs/33466020033

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant