Raise the e2e timeout to 75 minutes - #121
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as off-topic.
This comment was marked as off-topic.
38c8965 to
0918642
Compare
The nightly matrix is red in 30 of its last 37 finished runs. The reason is not a failing test: of the E2E legs that failed between 26 Aug and 5 Sep, every one sampled ended in ##[error]Final attempt failed. Timeout of 2700000ms hit with no failing spec reported at all. The durations say the same thing. Successful legs sit at a median of 28.4 min, while failed legs START at 40.9 min and run to 94.2 - which is 45 twice, plus setup, because max_attempts is 2. The cap is what ends them. Run 33853781633 was killed while running Tenancy.cy.ts, spec 27 of 27: it had got through the whole suite and was cut off in the last one. 45 was already a raise from 35, and the comment justifying it says Tenancy.cy.ts takes "~9 minutes". It does not: 19:35 in one measured run. The ES repo runs this same 27-spec suite under a 120-minute cap and its legs take 32-56 min, so 45 was below the suite's own working range, not a generous bound on it. 75 per attempt covers the observed worst case with headroom. These jobs declare no timeout-minutes of their own, so two attempts still fit well inside the 360-minute default. This does not make the slow runs fast. Specs that retry three times inflate the run, and the flake behind that is a separate fix. It stops the cap from turning a suite that finished into a suite that reports nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review findings on this PR. **targeted-e2e-tests.yml guards the same suite and was left at 60**, with max_attempts 1 so nothing catches it. By this PR's own numbers - failed legs start at 40.9 min, Tenancy.cy.ts measured at 19:35 - 60 sits inside the band that produced the timeouts. Raised to 75, with the reason next to it. **The cost is now stated.** "Both attempts fit inside the 360-minute default" answers whether the job dies, not what it costs. Worst case per leg goes from 2x45+2 = 92 min to 2x75+2 = 152, and each job is 4 versions x 3 envs, so up to +720 runner-minutes per run when everything times out. max_attempts stays 2. The retry is there for the transients the comment above it names - npm 429, a missing dev image - not for the timeouts, and with a cap that fits the suite a timeout should be the rare branch rather than the usual one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@10hexdev is right, and the correction is bigger than the arithmetic. timeout_minutes is per attempt, and retryWait runs after every failed attempt including the last (src/index.ts, both the timeout and the error branch), so two attempts cost 2x45+4 = 94 min, not 92. A 40.9-minute failure therefore cannot be a timeout at all. Checking that against all 82 failed E2E legs from 26 Aug to 5 Sep: exactly ONE reached 94 min. The other 81 finished below the double-timeout floor, so at most one of their attempts timed out. My "every failure sampled ended in Timeout" came from five recent legs, and only the last attempt's error is reported - so a leg whose attempt 1 failed for a real reason and whose attempt 2 was then cut off reports a timeout and hides the actual failure. Run 33853781633, measured attempt by attempt: attempt 1 08:57:24 to ~09:42 (timed out), attempt 2 09:44:24 to 10:31 (timed out) = 94 min. That is the one outlier, not the pattern. The pattern is the 75-78 min cluster: about 29 min of real failure, then 45 min of cap. So this raise does not turn the matrix green, and the PR should never have implied it would. What it buys is a suite that reports the real failure instead of a timeout, and a second attempt allowed to finish. The redness is spec flakiness, addressed in #118 and made visible by #120. Cost arithmetic corrected with it: 94 -> 154 min worst case per leg. The +720 runner-minutes delta is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed claim @10hexdev caught that I corrected only ONE of the two copies. The dev-e2e-tests block still said "45 was still too short... every failure sampled ended in Timeout... failed legs start at 40.9 min" - the exact claim the PR body retracts - and still had 2x45+2 = 92 instead of 94. That is the duplication problem itself, not a typo: the rationale was written out twice and the copies drifted the moment one was fixed. So it is now stated ONCE, above `on:`, and both retry steps carry a three-line pointer to it. targeted-e2e-tests.yml cited the disproven reason too. The raise there is still right, but for the real reason: a genuine pass can run past 60 min - the suite reaches ~77 min in the slow shape - and max_attempts is 1, so nothing catches it. Verified both jobs still emit (2, 75) and the retracted wording is gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file opened with an essay about why the cap changed and how the two copies of the old comment drifted. docs/dev/code-style.md asks for the current state at the code that enforces it, and leaves the story of the change to the commit message and the PR description. So each retry step now states what the reader cannot see from the line: the cap is per attempt, the retry waits after the last attempt too, and the eck envs need the larger number. The measurements that chose 75 move to the PR description. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0918642 to
da7aa5c
Compare
Merge order: this one first#124 is red right now, and this PR is what unblocks it. #124 carries
2,700,000 ms is 45 minutes. So one leg reports the cap and says nothing about what went wrong, while the other, which got far enough to finish, names the specs. Merging this PR to To be clear about what this does not do: |
The problem
timeout_minuteswas 45, and it is per attempt. The suite needs more. Run 33853781633 was killed on spec 27 of 27.The change
75 minutes, at all three retry steps: both jobs in
all-e2e-tests.ymland the one intargeted-e2e-tests.yml.What the number is based on
The
nick-fields/retryaction waits after every failed attempt, the last one included. Withmax_attempts: 2a double timeout therefore costs 2×45+4 = 94 min at the old cap, and 2×75+4 = 154 at the new one.45 was too short, but it is not what made these legs red. Of the 82 E2E legs that failed between 26 Aug and 5 Sep, exactly one reached the 94-minute floor. The other 81 finished below it, so at most one of their attempts timed out. The common shape is the 75–78 minute cluster: attempt 1 fails on its own at about 29 min, then attempt 2 is cut off at 45. Only the last attempt's error is reported, which is why sampling recent failures showed
Timeout of 2700000ms hit— it hides why attempt 1 failed, and it denies attempt 2 the chance to finish.Sizing:
Tenancy.cy.tsalone measured 19:35, and the ES repo runs this same 27-spec suite under a 120-minute cap. Cost: each job is 4 versions × 3 envs, so the worst case moves from 94 to 154 min per leg, up to +720 runner-minutes per run. Neither job setstimeout-minutes, so two attempts fit inside the 360-minute default.max_attemptsstays 2 — the retry is for the transients named at each step, not for the timeouts.This does not turn the matrix green
It makes the suite report the real failure instead of a timeout, and lets a second attempt run to completion. The redness is spec flakiness, addressed in #118 and made visible by #120.
Today's runs show both halves of that, side by side. #124 carries
developintomaster, so its legs still run the 45-minute cap:This branch, at 75, reports the actual failure instead:
Same leg, same day. One says "timeout", the other names the spec and the four tests. That is the whole point of the change.
Review notes
master.docs/dev/branching.mdin [RORDEV-2220] add branching, code style and writing style docs #125, case 4: a pipeline, workflow or runner change targetsmaster, becausemasterruns the nightly and the full matrix. Rebased ontomaster, so this PR carries only its own commits.docs/dev/code-style.mdasks for the current state at the code that enforces it, and leaves the story of the change to the commit message and the PR description — which is where the measurements above now live. Each retry step states only what the reader cannot see from the line: the cap is per attempt, the retry waits after the last attempt too, and the eck envs need the larger number.🤖 Generated with Claude Code
Summary by CodeRabbit