Skip to content

seed steps are killed by SIGPIPE on their first byte of output #23

Description

@luthermonson

run_seed spawns each seed: command with both stdout and stderr set to Stdio::piped() and then calls .status():

// src/deployer.rs
let status = Command::new("sh")
    .args(["-c", cmd])
    .current_dir(&workdir)
    .env("PREVIEW_URL", preview_url)
    .env("PREVIEW_HOST", hostname)
    .env("PR", pr_number.to_string())
    .stdout(Stdio::piped())
    .stderr(Stdio::piped())
    .status()
    .await;

tokio's Command::status() drops the parent's stdin/stdout/stderr handles before awaiting the child, so both pipes' read ends close immediately. The first byte the seed step writes to fd 1 or 2 therefore raises SIGPIPE and kills it.

The failure is invisible: the step dies in single-digit milliseconds with exit 141, no output is captured anywhere (the pipes were dropped), and the only trace is

WARN switchboard::deployer: seed step failed — continuing step=2 command=...

Reproduced on a preview node against ephpm/wordpress-sample:

$ cd /srv/ephpm/sites/ephpm-wordpress-sample-pr-4
$ sh -c "DOCROOT=. HOST=x bash seed/install.sh" | true ; echo ${PIPESTATUS[0]}
141
$ sh -c "DOCROOT=. HOST=x bash seed/install.sh" >/dev/null 2>&1 ; echo $?
2      # runs normally, fails later for its own unrelated reason

That step has failed on every deploy of PRs #1, #3 and #4 of wordpress-sample, always in 5-11 ms, always before it reaches the network — it dies on its first echo. It looked like an application bug for weeks.

Build steps are only half affected: run_build uses stdout(Stdio::null()) (safe) but stderr(Stdio::piped()) with the same .status(), so a build step that writes to stderr dies the same way.

Suggested fix

Use .output().await and log the captured stdout/stderr on failure — which is what the piping presumably intended in the first place, and would have made this self-diagnosing. Alternatively Stdio::null() for both if the output genuinely is not wanted.

The captured output would be valuable on its own: today a failing seed step gives an operator nothing but the command string.

Workaround

Apps must redirect every seed step's output, e.g. - "bash seed/foo.sh >> .seed.log 2>&1". Done in ephpm/wordpress-sample#5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions