Skip to content

Raise the e2e timeout to 75 minutes - #121

Merged
sscarduzio merged 5 commits into
masterfrom
ci/fit-the-e2e-timeout
Sep 10, 2026
Merged

Raise the e2e timeout to 75 minutes#121
sscarduzio merged 5 commits into
masterfrom
ci/fit-the-e2e-timeout

Conversation

@sscarduzio

@sscarduzio sscarduzio commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

The problem

timeout_minutes was 45, and it is per attempt. The suite needs more. Run 33853781633 was killed on spec 27 of 27.

The change

75 minutes, at all three retry steps: both jobs in all-e2e-tests.yml and the one in targeted-e2e-tests.yml.

What the number is based on

The nick-fields/retry action waits after every failed attempt, the last one included. With max_attempts: 2 a double timeout therefore costs 2×45+4 = 94 min at the old cap, and 2×75+4 = 154 at the new one.

45 was too short, but it is not what made these legs red. Of the 82 E2E legs that failed between 26 Aug and 5 Sep, exactly one reached the 94-minute floor. The other 81 finished below it, so at most one of their attempts timed out. The common shape is the 75–78 minute cluster: attempt 1 fails on its own at about 29 min, then attempt 2 is cut off at 45. Only the last attempt's error is reported, which is why sampling recent failures showed Timeout of 2700000ms hit — it hides why attempt 1 failed, and it denies attempt 2 the chance to finish.

Sizing: Tenancy.cy.ts alone measured 19:35, and the ES repo runs this same 27-spec suite under a 120-minute cap. Cost: each job is 4 versions × 3 envs, so the worst case moves from 94 to 154 min per leg, up to +720 runner-minutes per run. Neither job sets timeout-minutes, so two attempts fit inside the 360-minute default. max_attempts stays 2 — the retry is for the transients named at each step, not for the timeouts.

This does not turn the matrix green

It makes the suite report the real failure instead of a timeout, and lets a second attempt run to completion. The redness is spec flakiness, addressed in #118 and made visible by #120.

Today's runs show both halves of that, side by side. #124 carries develop into master, so its legs still run the 45-minute cap:

🔬 E2E Tests (released plugins) (9.4.6, docker)
##[error]Final attempt failed. Timeout of 2700000ms hit

This branch, at 75, reports the actual failure instead:

✖  Tenancy.cy.ts     02:07     17     1     4     -     12
✖  1 of 27 failed (4%)     30:25     82     52     4     14     12
error Command failed with exit code 4

Same leg, same day. One says "timeout", the other names the spec and the four tests. That is the whole point of the change.

Review notes

  • Base branch is now master. docs/dev/branching.md in [RORDEV-2220] add branching, code style and writing style docs #125, case 4: a pipeline, workflow or runner change targets master, because master runs the nightly and the full matrix. Rebased onto master, so this PR carries only its own commits.
  • The comments moved to the code they govern. The file used to open with an essay covering the history of the change and the drift between the two old copies. docs/dev/code-style.md asks for the current state at the code that enforces it, and leaves the story of the change to the commit message and the PR description — which is where the measurements above now live. Each retry step states only what the reader cannot see from the line: the cap is per attempt, the retry waits after the last attempt too, and the eck envs need the larger number.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Increased automated end-to-end test time limits to accommodate slower test environments and longer-running test suites.
    • Updated workflow guidance to clarify expected execution times and retry behavior.

@coderabbitai

This comment was marked as outdated.

sscarduzio

This comment was marked as outdated.

@sscarduzio

This comment was marked as outdated.

@10hexdev

This comment was marked as outdated.

@10hexdev

This comment was marked as outdated.

10hexdev[bot]

This comment was marked as outdated.

@sscarduzio
sscarduzio requested a review from coutoPL September 5, 2026 17:00
@sscarduzio

This comment was marked as outdated.

@sscarduzio

This comment was marked as outdated.

@10hexdev

This comment was marked as outdated.

@10hexdev

This comment was marked as off-topic.

10hexdev[bot]

This comment was marked as outdated.

10hexdev[bot]

This comment was marked as outdated.

@sscarduzio sscarduzio changed the title Give the e2e suite a cap it fits inside Raise the e2e timeout to 75 minutes Sep 6, 2026
@sscarduzio
sscarduzio requested a review from Dzuming September 10, 2026 06:08

@Dzuming Dzuming left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread .github/workflows/all-e2e-tests.yml Outdated
sscarduzio and others added 5 commits September 10, 2026 09:20
The nightly matrix is red in 30 of its last 37 finished runs. The reason is not
a failing test: of the E2E legs that failed between 26 Aug and 5 Sep, every one
sampled ended in

  ##[error]Final attempt failed. Timeout of 2700000ms hit

with no failing spec reported at all.

The durations say the same thing. Successful legs sit at a median of 28.4 min,
while failed legs START at 40.9 min and run to 94.2 - which is 45 twice, plus
setup, because max_attempts is 2. The cap is what ends them.

Run 33853781633 was killed while running Tenancy.cy.ts, spec 27 of 27: it had
got through the whole suite and was cut off in the last one.

45 was already a raise from 35, and the comment justifying it says Tenancy.cy.ts
takes "~9 minutes". It does not: 19:35 in one measured run. The ES repo runs this
same 27-spec suite under a 120-minute cap and its legs take 32-56 min, so 45 was
below the suite's own working range, not a generous bound on it.

75 per attempt covers the observed worst case with headroom. These jobs declare
no timeout-minutes of their own, so two attempts still fit well inside the
360-minute default.

This does not make the slow runs fast. Specs that retry three times inflate the
run, and the flake behind that is a separate fix. It stops the cap from turning
a suite that finished into a suite that reports nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review findings on this PR.

**targeted-e2e-tests.yml guards the same suite and was left at 60**, with
max_attempts 1 so nothing catches it. By this PR's own numbers - failed legs
start at 40.9 min, Tenancy.cy.ts measured at 19:35 - 60 sits inside the band
that produced the timeouts. Raised to 75, with the reason next to it.

**The cost is now stated.** "Both attempts fit inside the 360-minute default"
answers whether the job dies, not what it costs. Worst case per leg goes from
2x45+2 = 92 min to 2x75+2 = 152, and each job is 4 versions x 3 envs, so up to
+720 runner-minutes per run when everything times out.

max_attempts stays 2. The retry is there for the transients the comment above it
names - npm 429, a missing dev image - not for the timeouts, and with a cap that
fits the suite a timeout should be the rare branch rather than the usual one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@10hexdev is right, and the correction is bigger than the arithmetic.

timeout_minutes is per attempt, and retryWait runs after every failed attempt
including the last (src/index.ts, both the timeout and the error branch), so two
attempts cost 2x45+4 = 94 min, not 92. A 40.9-minute failure therefore cannot be
a timeout at all.

Checking that against all 82 failed E2E legs from 26 Aug to 5 Sep: exactly ONE
reached 94 min. The other 81 finished below the double-timeout floor, so at most
one of their attempts timed out. My "every failure sampled ended in Timeout" came
from five recent legs, and only the last attempt's error is reported - so a leg
whose attempt 1 failed for a real reason and whose attempt 2 was then cut off
reports a timeout and hides the actual failure.

Run 33853781633, measured attempt by attempt: attempt 1 08:57:24 to ~09:42
(timed out), attempt 2 09:44:24 to 10:31 (timed out) = 94 min. That is the one
outlier, not the pattern. The pattern is the 75-78 min cluster: about 29 min of
real failure, then 45 min of cap.

So this raise does not turn the matrix green, and the PR should never have
implied it would. What it buys is a suite that reports the real failure instead
of a timeout, and a second attempt allowed to finish. The redness is spec
flakiness, addressed in #118 and made visible by #120.

Cost arithmetic corrected with it: 94 -> 154 min worst case per leg. The +720
runner-minutes delta is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed claim

@10hexdev caught that I corrected only ONE of the two copies. The dev-e2e-tests
block still said "45 was still too short... every failure sampled ended in
Timeout... failed legs start at 40.9 min" - the exact claim the PR body
retracts - and still had 2x45+2 = 92 instead of 94.

That is the duplication problem itself, not a typo: the rationale was written
out twice and the copies drifted the moment one was fixed. So it is now stated
ONCE, above `on:`, and both retry steps carry a three-line pointer to it.

targeted-e2e-tests.yml cited the disproven reason too. The raise there is still
right, but for the real reason: a genuine pass can run past 60 min - the suite
reaches ~77 min in the slow shape - and max_attempts is 1, so nothing catches it.

Verified both jobs still emit (2, 75) and the retracted wording is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file opened with an essay about why the cap changed and how the two
copies of the old comment drifted. docs/dev/code-style.md asks for the
current state at the code that enforces it, and leaves the story of the
change to the commit message and the PR description.

So each retry step now states what the reader cannot see from the line:
the cap is per attempt, the retry waits after the last attempt too, and
the eck envs need the larger number. The measurements that chose 75 move
to the PR description.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sscarduzio
sscarduzio force-pushed the ci/fit-the-e2e-timeout branch from 0918642 to da7aa5c Compare September 10, 2026 09:21
@sscarduzio
sscarduzio requested a review from coutoPL September 10, 2026 09:29
Comment thread .github/workflows/all-e2e-tests.yml
@sscarduzio

Copy link
Copy Markdown
Contributor Author

Merge order: this one first

#124 is red right now, and this PR is what unblocks it.

#124 carries develop into master, so its head still has the 45-minute cap — this raise is an open PR, not on either long-lived branch yet. Both of its failing legs are 9.4.6, docker, and they fail in the two different ways this PR is about:

leg ends with
🔬 released plugins Final attempt failed. Timeout of 2700000ms hit
🧪 pre-build plugins 4 of 27 failed (15%), exit code 4

2,700,000 ms is 45 minutes. So one leg reports the cap and says nothing about what went wrong, while the other, which got far enough to finish, names the specs.

Merging this PR to master first, then re-running #124, turns the first leg into a real result — a pass, or the name of the spec that failed. Merging #124 first leaves both branches on 45 and we learn nothing new from the re-run.

To be clear about what this does not do: Tenancy.cy.ts fails for its own reasons on the pre-build leg, and this PR does not fix that. It only stops the cap from hiding the evidence. The redness is spec flakiness, addressed in #118 and made visible by #120.

@sscarduzio
sscarduzio requested a review from coutoPL September 10, 2026 10:09
@sscarduzio
sscarduzio merged commit 4d95bbb into master Sep 10, 2026
15 of 16 checks passed
@coutoPL
coutoPL deleted the ci/fit-the-e2e-timeout branch September 10, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants