From 433bd9fdec89cd0fd18f49c691f88a6ed3ed3015 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 3 Jul 2026 18:39:09 +0000 Subject: [PATCH 1/3] docs: plan for sharding e2e CI across multiple runners MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With the concurrent-runner limit raised to 20, revisit the matrix-sharding idea that e2e-speedup-plan.md deferred ("Considered and skipped", item 2) solely because of the old ~3-runner throttle. Grounds the plan in measured step-level timings from a green PR run: both e2e lanes are execution-dominated (full 636s exec / 4.6m overhead; quickstart 823s exec / 4.2m overhead), and quickstart (~18m) — not full (~15m) — is the real critical path. Proposes reshaping the 2-lane matrix into ~6 balanced legs (label-filter split of the functional suite + break up the sequential quickstart chain), each on its own runner + k3d cluster, projecting the e2e gate from ~18m to ~8-10m. Covers the details that must be handled: per-shard coverage merge via Codecov flags, an e2e-complete aggregator so branch protection is matrix- agnostic, keeping fail-fast:false, and why per-cluster isolation makes sharding strictly safer than raising --procs on the singleton controller. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/design/e2e-ci-runner-sharding-plan.md | 202 +++++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 docs/design/e2e-ci-runner-sharding-plan.md diff --git a/docs/design/e2e-ci-runner-sharding-plan.md b/docs/design/e2e-ci-runner-sharding-plan.md new file mode 100644 index 00000000..b548ea63 --- /dev/null +++ b/docs/design/e2e-ci-runner-sharding-plan.md @@ -0,0 +1,202 @@ +# E2E CI Runner Sharding Plan + +## Question + +We are now allowed **up to 20 concurrent GitHub-hosted runners** (previously this +repo was throttled at ~3). Can we split the e2e suite across more runners to cut +wall-clock CI time? An earlier attempt "didn't fully work" — this doc explains +why, what changed, and a concrete plan. + +## TL;DR + +- **Yes, and the one thing that blocked it before is now gone.** The only reason + matrix sharding was rejected in [e2e-speedup-plan.md](e2e-speedup-plan.md) + ("Considered and skipped", item 2) was the concurrent-runner throttle: extra + shards *queued* instead of running in parallel, so they burned CPU minutes for + no wall-clock win. With 20 runners that constraint is lifted. +- **Sharding across runners is strictly *safer* than raising `--procs` on one + cluster.** Each shard gets its own k3d cluster, so the full-suite shared-state + races documented in + [e2e-full-suite-shared-state-investigation.md](e2e-full-suite-shared-state-investigation.md) + **cannot cross a shard boundary**, and `Serial` specs stop contending with + everything else in the run. +- **The economics work now**: in both current e2e lanes, *test execution* + dominates *fixed cluster-bringup overhead* (see the measured breakdown below), + so splitting execution across runners actually moves the needle. +- **The real critical path today is the `quickstart` lane (~18 min), not `full` + (~15 min)** — so a good plan splits *both*, not just `full`. + +## Measured baseline (run 28649519807, PR e2e, 2026-07-03) + +Both e2e lanes run in parallel today on 2 `ubuntu-latest` runners. Step-level +timings: + +| Lane | Fixed overhead¹ | Test execution | Total | +|---|---|---|---| +| `full` | ~4.6 min | **636 s (10.6 min)** | ~15.2 min | +| `quickstart` | ~4.2 min | **823 s (13.7 min)** | ~18.0 min | + +¹ Fixed overhead = free-disk + artifact download + docker load + "Bring up +cluster + Flux services" (140 s / 95 s) + coverage collect + report. It is paid +**once per runner/cluster** and is the tax every new shard adds. + +Two facts drive the whole plan: + +1. **Execution ≫ overhead** (10.6 vs 4.6; 13.7 vs 4.2). Amdahl is on our side: + splitting the execution slice across `S` shards gives, per shard, + `wall ≈ overhead + execution / S`. +2. **`quickstart` is the longer lane.** Sharding only `full` below 15 min buys + nothing until `quickstart` is also shortened — the whole-CI e2e gate is + `max(all lanes)`. + +### What each lane actually runs + +- **`full`** = `go run ginkgo --procs=4 --label-filter='!image-refresh'` — the + ~46-spec functional suite on one cluster. `Serial` specs + (`restart-reconcile`, `crd_lifecycle`, `playground`) are interleaved by Ginkgo; + everything else runs 4-wide. This is the only lane that collects e2e coverage + (config-dir install carries the `GOCOVERDIR` overlay). +- **`quickstart`** = **four heavy operations, strictly sequential on one cluster**: + 1. `test-e2e-quickstart-helm` — Helm install validation + 2. `cleanup-installs.sh` + 3. `test-image-refresh` (`Serial`) — local **rebuild** + k3d reload + rollout + 4. `test-e2e-quickstart-manifest` — plain-manifest install validation + + Steps 1/3/4 each install/reinstall the controller, so they can't overlap *on + one cluster* — but on **separate clusters** they are independent. + +## Why "it didn't fully work" before + +Historical context, not a blocker anymore: + +- **Runner throttle** (the real one): shards queued, so 3 shards × 15 min ran + serially ≈ 45 min. Recorded in [e2e-speedup-plan.md](e2e-speedup-plan.md). +- **Shared-state flakes** within one cluster (stale `GitTarget`/`WatchRule` + writing to a shared repo/branch — see the + [shared-state investigation](e2e-full-suite-shared-state-investigation.md)). + Sharding *removes* this class between shards (separate clusters) and the + per-target isolation fix + ([gittarget-isolation-on-rule-change.md](../finished/gittarget-isolation-on-rule-change.md)) + already raised the safe in-cluster parallelism from `procs=1` → `procs=4`. + +## Why sharding beats "just raise `--procs`" + +The controller is a **singleton** per cluster. On a stock `ubuntu-latest` runner +it is CPU-starved past `procs=4` (a different spec timed out each run — see +Phase 2.5 of the speedup plan). More Ginkgo procs on one cluster all fight for +one controller's CPU. **A new shard gets its own controller, its own cluster, and +a whole runner's CPU** — it sidesteps the singleton ceiling entirely instead of +pushing against it. + +## Proposal + +Reshape the `e2e` matrix in [ci.yml](../../.github/workflows/ci.yml) from **2 +lanes → ~6 balanced legs**, each its own runner + cluster. Partition functional +specs with Ginkgo `--label-filter` (labels already exist — see the `Label(...)` +on each `Describe`), and break the `quickstart` chain into independent legs. + +### Target matrix + +| Leg | Selector | Source (today) | Notes | +|---|---|---|---| +| `manager` | `--label-filter='manager'` | part of `full` | Big bucket of many small specs; includes `Serial` `crd_lifecycle`. Collects coverage. | +| `signing-agg-bidi` | `'signing \|\| aggregated-api \|\| bi-directional'` | part of `full` | Three slow-ish independent suites. Collects coverage. | +| `audit-restart` | `'audit-consumer \|\| restart-reconcile'` | part of `full` | Slow audit specs + the slow `Serial` restart spec, alone so it can't contend. Collects coverage. | +| `quickstart-helm` | `'quickstart-framework'`, `E2E_QUICKSTART_MODE=helm` | `quickstart` step 1 | Needs release bundle artifact. | +| `quickstart-manifest` | `'quickstart-framework'`, `E2E_QUICKSTART_MODE=plain-manifests-file` | `quickstart` step 4 | Needs release bundle artifact. | +| `image-refresh` | `'image-refresh'`, local build (no `PROJECT_IMAGE`) | `quickstart` step 3 | `Serial` rebuild/reload/rollout; its own runner. | + +Six legs uses **6 of 20 runners** — comfortable headroom. + +### Projected wall-clock + +Per leg `≈ overhead + execution/slice`. Splitting `full`'s 636 s three ways and +`quickstart`'s 823 s three ways, with ~4.5 min overhead each: + +- functional legs: ~4.5 + ~3.5 ≈ **~8 min** each +- quickstart legs: ~4.5 + (helm/manifest/refresh slice) ≈ **~7–10 min** each + +**New e2e gate ≈ 8–10 min, down from ~18 min — roughly halved.** The new critical +leg is whichever single slice is largest (likely `image-refresh`'s local docker +rebuild or the `manager` bucket); rebalance from data (next section). + +### Cost + +~6 legs × ~8 min ≈ **~48 runner-minutes** vs today's 2 × ~16.5 ≈ ~33 — about +**1.5× the CPU minutes for ~2× the speed**. Acceptable given the granted capacity; +tune the leg count to trade minutes for latency. + +## Details that must be handled + +1. **Balance from real data, don't guess.** Each run already uploads + `e2e-ginkgo-reports-*` artifacts, and `test/e2e/tools/spec-timings` ranks + specs by duration. Pull a green run's per-spec CI timings and pack the label + groups to equalize slices. Start with the table above; adjust once measured. +2. **Coverage merge.** Today only `full` collects e2e coverage. Each functional + shard must run the **config-dir** install (so it carries the `GOCOVERDIR` + overlay), run `task e2e-coverage-collect`, and upload with `flags: e2e`. + Codecov unions same-flag uploads, so the merged `e2e` coverage is the union of + what each shard exercised — non-regression ratchet still holds. (The + quickstart/manifest/image-refresh legs don't carry the overlay and collect no + coverage, same as today.) +3. **Required status checks / branch protection.** Job name is + `E2E (${{ matrix.name }})`, so adding legs changes the required-check set. Add + a tiny aggregator job `e2e-complete` (`needs: [e2e]`, succeeds iff all legs + pass) and make **that** the single required check — then the matrix can grow or + shrink without editing branch protection each time. +4. **Keep `fail-fast: false`** (already set) so one leg's flake doesn't cancel the + others mid-run and lose their reports. +5. **Serial registry stays authoritative.** [e2e-serial-registry.md](e2e-serial-registry.md) + still governs *within* a shard. Sharding doesn't let us drop any `Serial` + marker — it just means a `Serial` spec only blocks its own (smaller) shard. +6. **Disk pressure is per-shard.** Each fresh runner has its own disk, so the + DiskPressure eviction mitigations (PR #187) and the `Free runner disk` step + apply per leg — no worse per shard. +7. **Artifact fan-out.** Each shard re-downloads the CI container + project image + tarballs (~35 s + ~44 s load). That's part of the per-shard overhead already + counted; watch total artifact egress if leg count grows large. + +## Phased rollout + +- **Phase 1 — split `full` into 3 label legs** (lowest risk, no chain surgery). + Wire per-shard coverage upload. Land the `e2e-complete` aggregator + update + branch protection. Verify green over ~5–10 runs (watch for any residual + cross-spec flake *inside* a shard — should be none, separate clusters). +- **Phase 2 — split the `quickstart` chain into 3 legs** + (`quickstart-helm`, `quickstart-manifest`, `image-refresh`). This is where the + critical-path win actually lands, since quickstart is the long pole. +- **Phase 3 — rebalance** leg membership from measured per-spec CI timings; merge + or split legs to flatten the slowest leg. Consider whether a 4th functional leg + is worth the extra ~4.5 min overhead. + +## Rollback + +Every change here lives in the `e2e` matrix of `ci.yml` (plus one aggregator job +and a branch-protection edit). Reverting to the 2-lane matrix is a one-file revert +— no product code, no test code, no cluster/state changes. + +## Open questions / to validate during Phase 1 + +- Do `playground` / `demo` labels currently execute in the `full` lane or + self-skip without their env gates? Confirm before assigning them to a leg (they + should stay out of the functional shards). +- Exact per-slice split of `quickstart`'s 823 s across helm / cleanup / refresh / + manifest — measure to decide whether `image-refresh` (local rebuild) needs to + be the sole occupant of the slowest leg or can share. +- Whether one designated leg should run the *whole* suite for a single clean + coverage number instead of per-shard union (simpler Codecov story vs. one extra + full-length leg). Recommendation: start with per-shard union; revisit only if + the merged number looks lossy. + +## References + +- [e2e-speedup-plan.md](e2e-speedup-plan.md) — Phases 0–3 already shipped; this + plan is its "Revisit if we get real concurrent runner capacity" item. +- [e2e-serial-registry.md](e2e-serial-registry.md) — the authoritative `Serial` set. +- [e2e-full-suite-shared-state-investigation.md](e2e-full-suite-shared-state-investigation.md) + — the cross-spec races sharding isolates away. +- [gittarget-isolation-on-rule-change.md](../finished/gittarget-isolation-on-rule-change.md) + — the fix that raised safe in-cluster parallelism to `procs=4`. +- [ci-overview.md](../ci-overview.md) — trust-zone CI design the matrix lives in. + + From d0d257b496beabe376e65bdbaa195da250eef1ad Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 3 Jul 2026 19:44:39 +0000 Subject: [PATCH 2/3] fix(ci): repair release publish (immutable releases + cosign chart auth) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 0.29.1 release — the first to exercise the new sign/attest + retag pipeline — failed in both final publish jobs: - publish-manifest could not attach sbom.spdx.json or append notes: release-please publishes the GitHub release and GitHub now enforces immutable releases, which reject post-publish asset/body edits. Create the release as a draft, attach every asset while it is still a draft, and add a publish-release job that flips it to published only after both publish jobs succeed (a failure now leaves an unpublished draft, not a half-populated immutable release). - publish-helm's cosign chart signature failed with UNAUTHORIZED: `helm registry login` writes helm's own registry config, but cosign reads ~/.docker/config.json. Add an explicit `cosign login` before signing. Also refresh the README badges: - add the OpenSSF Best Practices badge (closes #189) - drop the retired Go Report Card badge (closes #183) Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/release.yml | 40 +++++++++++++++++++++++++++++++++++ README.md | 2 +- release-please-config.json | 1 + 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9315b137..f3e2d7da 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -175,6 +175,11 @@ jobs: uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3.0.1 with: tag_name: ${{ needs.release-please.outputs.tag_name }} + # Keep the release a draft: GitHub's immutable-releases enforcement + # forbids adding assets once a release is published, so every asset + # must land while it is still a draft. The publish-release job flips + # it to published after all assets are attached. + draft: true append_body: true files: sbom.spdx.json body: | @@ -260,6 +265,15 @@ jobs: - name: Install cosign uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 + - name: Log in to registry for cosign + # `helm registry login` writes helm's own registry config; cosign reads + # ~/.docker/config.json, so it needs its own login or the signature + # upload fails with UNAUTHORIZED. + env: + COSIGN_TOKEN: ${{ secrets.GITHUB_TOKEN }} + COSIGN_USER: ${{ github.actor }} + run: cosign login "${REGISTRY}" --username "${COSIGN_USER}" --password "${COSIGN_TOKEN}" + - name: Sign the Helm chart (cosign keyless) run: cosign sign --yes "${CHART_REGISTRY}/gitops-reverser@${{ steps.chart.outputs.digest }}" @@ -267,6 +281,32 @@ jobs: uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3.0.1 with: tag_name: ${{ needs.release-please.outputs.tag_name }} + # Attach to the still-draft release; publish-release publishes it once + # every asset is in place (immutable releases reject post-publish + # uploads). + draft: true files: dist/install.yaml env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + # Publish the release only after every asset is attached. release-please + # creates the GitHub release as a draft (see release-please-config.json); + # GitHub's immutable-releases enforcement freezes a release the moment it is + # published, so publish-manifest and publish-helm attach their assets to the + # draft and this job flips it to published at the very end. If either publish + # job fails, the release stays an unpublished draft instead of a + # half-populated immutable release. + publish-release: + name: Publish GitHub Release + runs-on: ubuntu-latest + timeout-minutes: 10 + needs: [release-please, publish-manifest, publish-helm] + if: needs.release-please.outputs.release_created == 'true' + permissions: + contents: write # publish the draft release (creates the tag) + steps: + - name: Publish the draft release + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG_NAME: ${{ needs.release-please.outputs.tag_name }} + run: gh release edit "${TAG_NAME}" --draft=false --repo "${{ github.repository }}" diff --git a/README.md b/README.md index 4ad39662..9d126666 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ [![CI](https://github.com/ConfigButler/gitops-reverser/actions/workflows/release.yml/badge.svg?branch=main)](https://github.com/ConfigButler/gitops-reverser/actions/workflows/release.yml) [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/ConfigButler/gitops-reverser/badge)](https://scorecard.dev/viewer/?uri=github.com/ConfigButler/gitops-reverser) +[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13468/badge)](https://www.bestpractices.dev/projects/13468) [![Release](https://img.shields.io/github/v/release/ConfigButler/gitops-reverser?sort=semver)](https://github.com/ConfigButler/gitops-reverser/releases) [![License](https://img.shields.io/github/license/ConfigButler/gitops-reverser)](https://www.apache.org/licenses/LICENSE-2.0) [![Go](https://img.shields.io/badge/go-1.26-blue?logo=go)](go.mod) [![Platforms](https://img.shields.io/badge/platforms-amd64%20%7C%20arm64-2ea44f?logo=docker)](#) -[![Go Report Card](https://goreportcard.com/badge/github.com/ConfigButler/gitops-reverser)](https://goreportcard.com/report/github.com/ConfigButler/gitops-reverser) [![codecov](https://codecov.io/gh/ConfigButler/gitops-reverser/graph/badge.svg)](https://codecov.io/gh/ConfigButler/gitops-reverser) [![Container](https://img.shields.io/badge/container-ghcr.io%2Fconfigbutler%2Fgitops--reverser-2ea44f?logo=docker)](https://github.com/ConfigButler/gitops-reverser/pkgs/container/gitops-reverser) [![Open Issues](https://img.shields.io/github/issues/ConfigButler/gitops-reverser)](https://github.com/ConfigButler/gitops-reverser/issues) diff --git a/release-please-config.json b/release-please-config.json index 3ba69bac..a3043bf4 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -4,6 +4,7 @@ "release-type": "simple", "package-name": "gitops-reverser", "bump-minor-pre-major": true, + "draft": true, "changelog-sections": [ { "type": "feat", From d7efd72ac2732f61b076271cbb3d0507afdc3aee Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 3 Jul 2026 20:00:14 +0000 Subject: [PATCH 3/3] ci(e2e): shard the e2e suite across 4 runners (2x full, 2x quickstart) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With up to 20 concurrent runners now available, split both e2e lanes to cut the wall-clock gate (was max(full ~15m, quickstart ~18m)). Per docs/design/e2e-ci-runner-sharding-plan.md this is Phases 1+2 at a 2-way split: - full -> full-manager (`manager`) + full-core (`!manager && !image-refresh`). The two filters are exact complements of the old `!image-refresh` full filter; a ginkgo --dry-run confirms 37 + 18 = 55 specs, so no spec is added, dropped, or double-run. Both shards run config-dir installs and collect e2e coverage; Codecov unions same-flag uploads, so merged coverage is unchanged. - quickstart -> quickstart-install (helm + manifest install validations on one cluster) + image-refresh (the local rebuild/reload/rollout chain on its own runner — the plan's likely-slowest slice). test-e2e gains E2E_LABEL_FILTER / E2E_REPORT_NAME (defaults preserve a plain `task test-e2e`); CI forwards them by env-name so the filter's `&&` never reaches the shell. e2e is not a required status check, so no branch-protection change is needed. Leg membership is a provisional first cut — rebalance from the e2e-ginkgo-reports-* per-spec timings once a green run produces them. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 81 +++++++++++++++++++++++++++------------- test/e2e/Taskfile.yml | 9 ++++- 2 files changed, 63 insertions(+), 27 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index deff3894..1445963b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -710,41 +710,62 @@ jobs: fail-fast: false matrix: include: - - name: quickstart - # test-image-refresh validates the local build → k3d load → - # rollout restart chain, so it must run against a locally built - # image: unset PROJECT_IMAGE (the CI-built image doesn't rebuild - # from source) and force load delivery. Fork-safe: the chain never - # touches a registry. - # - # cleanup-installs.sh between the two: the helm quickstart just - # installed with the *provided* image ref, and install stamps do - # not invalidate on an image-ref change — without the reset the - # refresh suite would keep restarting a deployment that still - # points at the provided tag and never runs the local rebuilds. + # Functional suite, sharded across two runners (was the single `full` + # lane). The two label filters are exact complements within the old + # `!image-refresh` filter — `manager` ∪ `!manager && !image-refresh` + # equals what `full` ran — so no spec is added or dropped, the work is + # just split across two clusters. The membership is a provisional + # first cut; rebalance from the e2e-ginkgo-reports-* per-spec timings + # (docs/design/e2e-ci-runner-sharding-plan.md, "balance from real data"). + # Each shard runs config-dir installs (GOCOVERDIR overlay) and collects + # coverage; Codecov unions same-flag `e2e` uploads, so merged coverage + # is the union of both shards — the non-regression ratchet still holds. + - name: full-manager + script: "task test-e2e" + e2e_label_filter: "manager" + e2e_report_name: "full-manager" + needs_artifact: false + coverage: "1" + # Bumped 1 -> 4: the full suite was serialized to avoid the + # cross-target rule-change snapshot coupling, now fixed (per-target + # effective watch plan — docs/design/gittarget-isolation-on-rule-change.md). + e2e_ginkgo_procs: "4" + k3d_agent_count: "0" + - name: full-core + script: "task test-e2e" + e2e_label_filter: "!manager && !image-refresh" + e2e_report_name: "full-core" + needs_artifact: false + coverage: "1" + e2e_ginkgo_procs: "4" + k3d_agent_count: "0" + # Quickstart chain, sharded across two runners (was the single + # `quickstart` lane). quickstart-install runs the two install-mode + # validations on one cluster (cleanup-installs.sh resets the namespace + # between them, since install stamps don't invalidate on an image-ref + # change). image-refresh gets its own runner: it validates the local + # build → k3d load → rollout restart chain, so it must run against a + # locally built image — unset PROJECT_IMAGE and force load delivery. + # Fork-safe: neither leg pushes to a registry. Helm/manifest installs + # don't carry the GOCOVERDIR overlay, so these legs collect no coverage. + - name: quickstart-install script: >- export INSTALL_MODE=helm HELM_CHART_SOURCE=./gitops-reverser.tgz && task test-e2e-quickstart-helm && bash hack/cleanup-installs.sh k3d-gitops-reverser-test-e2e - && env -u PROJECT_IMAGE IMAGE_DELIVERY_MODE=load task test-image-refresh && export INSTALL_MODE=plain-manifests-file && task test-e2e-quickstart-manifest needs_artifact: true + coverage: "" e2e_ginkgo_procs: "2" k3d_agent_count: "0" - # Helm/manifest installs don't carry the GOCOVERDIR overlay, so no - # coverage is collected here. + - name: image-refresh + script: >- + export INSTALL_MODE=helm HELM_CHART_SOURCE=./gitops-reverser.tgz + && env -u PROJECT_IMAGE IMAGE_DELIVERY_MODE=load task test-image-refresh + needs_artifact: true coverage: "" - - name: full - script: "task test-e2e" - needs_artifact: false - # config-dir install carries the GOCOVERDIR overlay; collect e2e coverage. - coverage: "1" - # Bumped 1 -> 4: the full suite was serialized to avoid the - # cross-target rule-change snapshot coupling, now fixed (per-target - # effective watch plan — docs/design/gittarget-isolation-on-rule-change.md). - # Validated locally at --procs=4; trialing the same in CI. - e2e_ginkgo_procs: "4" + e2e_ginkgo_procs: "2" k3d_agent_count: "0" env: PROJECT_IMAGE: ${{ needs.build.outputs.image }} @@ -859,6 +880,14 @@ jobs: " - name: Run E2E tests in CI container + # E2E_LABEL_FILTER / E2E_REPORT_NAME are passed through by name (not + # value) so the shell never sees the filter's spaces or `&&` — GitHub + # sets them as process env vars and `docker run -e VAR` forwards them. + # They're only consumed by `task test-e2e` (the functional shards); the + # quickstart legs leave them empty and ignore them. + env: + E2E_LABEL_FILTER: ${{ matrix.e2e_label_filter }} + E2E_REPORT_NAME: ${{ matrix.e2e_report_name }} run: | docker run --rm \ --network host \ @@ -868,6 +897,8 @@ jobs: -e PROJECT_IMAGE=${{ env.PROJECT_IMAGE }} \ -e IMAGE_DELIVERY_MODE=${{ env.IMAGE_DELIVERY_MODE }} \ -e E2E_GINKGO_PROCS=${{ matrix.e2e_ginkgo_procs }} \ + -e E2E_LABEL_FILTER \ + -e E2E_REPORT_NAME \ -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ -e HOST_PROJECT_PATH=${{ github.workspace }} \ -e KUBECONFIG=${{ env.CI_WORKDIR }}/.stamps/cluster/k3d-gitops-reverser-test-e2e/.kube/config \ diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index d11ac638..a7ebc7cb 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -218,10 +218,15 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" + # E2E_LABEL_FILTER lets CI shard the suite across runners (e.g. + # 'manager' vs '!manager && !image-refresh'); it defaults to the whole + # non-image-refresh suite so a plain `task test-e2e` still runs it all. + # E2E_REPORT_NAME keeps each shard's Ginkgo JSON report distinct. go run github.com/onsi/ginkgo/v2/ginkgo \ --procs={{.E2E_GINKGO_PROCS}} --timeout="{{.E2E_FULL_TIMEOUT}}" -v \ - --label-filter='!image-refresh' \ - --output-dir="{{.CS}}/{{.NAMESPACE}}" --json-report=ginkgo-report-full.json \ + --label-filter='{{.E2E_LABEL_FILTER | default "!image-refresh"}}' \ + --output-dir="{{.CS}}/{{.NAMESPACE}}" \ + --json-report=ginkgo-report-{{.E2E_REPORT_NAME | default "full"}}.json \ ./test/e2e/ e2e-coverage-collect: