test-e2e carries continue-on-error: true in ci.yml. The comment beside it
says what it was for, and it is honest:
continue-on-error on purpose: a flaky E2E job everyone ignores is worth less than no job
at all. To be removed once it has proven itself over a few dozen runs.
It has now had the other kind of run. On main at e4ed5d9,
Test E2E (ubuntu-22.04) failed on a real data-loss bug — a note keeping its
title and losing the body typed after it — and the workflow was reported as
success. gh run list says success. The badge says passing. The failure sat
there across three merges and was found only because it was looked for by hand.
So the flag is no longer buying a quiet CI; it is buying a CI that says the wrong
thing. Whatever replaces it has to make a failing E2E job visible.
What to decide
- Make it blocking — drop
continue-on-error entirely. Simplest, and it is what
the flag was always meant to become. The cost is that one genuinely flaky run
blocks a merge until it is re-run.
- Keep it non-blocking but loud — leave the flag and add a step that fails the
run summary, or post the result as a check of its own, so a red E2E leg cannot
be read as green.
Done when
- A failing E2E job can no longer be mistaken for a green run, from
gh run list,
from the PR checks list and from the badge.
- The decision and its reason are written down in
ci.yml beside whatever is left,
and in docs/architecture.md where the suite is described.
Worth doing first
The suite should earn it. #106 fixes the defect the Ubuntu leg was reporting; a
handful of consecutive green runs after it lands is what makes "blocking" a
reasonable thing to ask for.
test-e2ecarriescontinue-on-error: trueinci.yml. The comment beside itsays what it was for, and it is honest:
It has now had the other kind of run. On
mainate4ed5d9,Test E2E (ubuntu-22.04)failed on a real data-loss bug — a note keeping itstitle and losing the body typed after it — and the workflow was reported as
success.gh run listsayssuccess. The badge says passing. The failure satthere across three merges and was found only because it was looked for by hand.
So the flag is no longer buying a quiet CI; it is buying a CI that says the wrong
thing. Whatever replaces it has to make a failing E2E job visible.
What to decide
continue-on-errorentirely. Simplest, and it is whatthe flag was always meant to become. The cost is that one genuinely flaky run
blocks a merge until it is re-run.
run summary, or post the result as a check of its own, so a red E2E leg cannot
be read as green.
Done when
gh run list,from the PR checks list and from the badge.
ci.ymlbeside whatever is left,and in
docs/architecture.mdwhere the suite is described.Worth doing first
The suite should earn it. #106 fixes the defect the Ubuntu leg was reporting; a
handful of consecutive green runs after it lands is what makes "blocking" a
reasonable thing to ask for.