Skip to content

A failing E2E job is reported as a green run #108

Description

@vmillet-dev

test-e2e carries continue-on-error: true in ci.yml. The comment beside it
says what it was for, and it is honest:

continue-on-error on purpose: a flaky E2E job everyone ignores is worth less than no job
at all. To be removed once it has proven itself over a few dozen runs.

It has now had the other kind of run. On main at e4ed5d9,
Test E2E (ubuntu-22.04) failed on a real data-loss bug — a note keeping its
title and losing the body typed after it — and the workflow was reported as
success. gh run list says success. The badge says passing. The failure sat
there across three merges and was found only because it was looked for by hand.

So the flag is no longer buying a quiet CI; it is buying a CI that says the wrong
thing. Whatever replaces it has to make a failing E2E job visible.

What to decide

  • Make it blocking — drop continue-on-error entirely. Simplest, and it is what
    the flag was always meant to become. The cost is that one genuinely flaky run
    blocks a merge until it is re-run.
  • Keep it non-blocking but loud — leave the flag and add a step that fails the
    run summary, or post the result as a check of its own, so a red E2E leg cannot
    be read as green.

Done when

  • A failing E2E job can no longer be mistaken for a green run, from gh run list,
    from the PR checks list and from the badge.
  • The decision and its reason are written down in ci.yml beside whatever is left,
    and in docs/architecture.md where the suite is described.

Worth doing first

The suite should earn it. #106 fixes the defect the Ubuntu leg was reporting; a
handful of consecutive green runs after it lands is what makes "blocking" a
reasonable thing to ask for.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    changeBehaviour change to something that already existed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions