Keep the stack logs when the suite fails

[sscarduzio]

The problem

Seven runs between 12 August and 4 September collapsed with every spec red while it_linux was green, and not one of them left a single line of Kibana or Elasticsearch log behind. There was nothing to diagnose them with.

The stack does write a log, elk-ror.log, but start.sh writes it only from its ERR trap — that is, only when the stack fails to come up. These seven runs are the opposite case: the stack came up, the healthchecks passed, and the Cypress suite then collapsed with every spec stuck on /login. By then the file that would have explained it was never written.

print-logs.sh is not a substitute either. It cats that same elk-ror.log, so on a test failure it has nothing to print.

The change

runner.sh takes the suite's exit status by hand and, on failure, calls a new dump-logs.sh for the environment before it exits:

• environments/elk-ror/dump-logs.sh — container status, exit codes and per-container logs. It reads the container list straight from docker rather than from docker compose, so it needs neither the compose file list that start.sh assembles nor the working directory it assembles them in.
• environments/eck-ror/dump-logs.sh — the twin for ECK: pod logs, pod status and cluster events, one file per pod. Every kubectl call goes through docker exec on the kind control-plane node, because the host does not necessarily have a kubectl that can reach that cluster. It queries all namespaces: ES and Kibana live in default, but on an ECK failure the operator's log is usually the one that explains it, and that lives in elastic-system.

Both write into results/stack-logs. The ES repo already uploads "$E2E_TESTS_DIR/results" to S3 on failure, next to the Cypress videos and screenshots, so no caller has to change to start receiving the logs.

Neither collector can fail its caller. They run on a path that has already failed, and a missing log must not replace the real error with an error from collecting logs.

The run summary

Cypress prints its per-spec table thousands of lines into the step log, where nobody finds it — in run 34421845288 it starts at line 13,072. runner.sh now repeats that table on the run's summary page, on a pass as well as on a failure.

It tees the suite's output, strips the colour codes and copies the block from Run Finished onward into $GITHUB_STEP_SUMMARY. Verified against the real log of the failing 9.4.6 leg: 121 lines, the table and the totals, no escapes left. GITHUB_STEP_SUMMARY is unset outside Actions, so a local run is unchanged.

One collector, two failure paths

print-logs.sh and the new dump-logs.sh were unrelated mechanisms. They are now called from one function, used by the ERR trap and by the suite-failure path:

collect_logs() {
  ./environments/"$ENV_NAME"/print-logs.sh || true
  ./environments/"$ENV_NAME"/dump-logs.sh results/stack-logs || true
}

They are not redundant. print-logs.sh cats elk-ror.log, which exists only when the stack failed to come up. The dump covers the opposite case. Calling both from one place means neither path can lose its logs again.

Limit

This does not fix the collapse. It makes the next one possible to diagnose.

Review notes

• Base branch is now master. docs/dev/branching.md in [RORDEV-2220] add branching, code style and writing style docs #125, case 4: a pipeline, workflow or runner change targets master. Rebased onto master, so this PR carries only its own two commits.
• The comments were rewritten. docs/dev/code-style.md asks a comment to explain the current state and to avoid naming other files. The headers did the opposite: they counted the seven collapses, walked through what start.sh and print-logs.sh each write and when, and named another repository's S3 upload. All of that is above instead. The comments keep only what a reader cannot get from the code — which failure the script runs on, why the container list comes from docker and not from compose, why kubectl goes through the control-plane node, and why the collector must never fail its caller.

🤖 Generated with Claude Code

[coutoPL]

ok, but I still have a doublt. We collect logs in the dump-logs.sh script differently that in the print-logs.sh. I think these are related things. Maybe we can replace both by some common script. Or do sth else to make it more common?