Workflow engine silently stops executing runs after uptime: webhook/manual runs stay pending forever, schedules never fire (unsupervised engine task)

[joydai2026-del]

Summary

On a hosted community relay, the workflow engine silently stops executing runs entirely after some uptime. Every trigger path is affected once it happens: webhook-triggered runs and manual triggers return 202 / accepted with a run_id and status pending, but the run never executes and no step actions happen. Scheduled (cron/interval) triggers are the most visible casualty because they depend on the engine loop end to end, which likely explains #5611 and #4904. #4288 (message_posted dead while others work) looks like the same class of failure caught at a different moment.

The HTTP/ingest side keeps accepting work the whole time, so from the operator's point of view everything reports success while nothing ever runs.

Environment

• Hosted community relay: jj.communities.buzz.xyz (communities.buzz.xyz cloud)
• Observed 2026-08-19 through 2026-08-21
• Source read at 4e3c9e619c93dd26677b392ad1f8cf0d12c8f855

Timeline of live tests (all on 2026-08-21, times UTC)

A dedicated throwaway channel was used, with four minimal one-step (send_message) workflows, one per trigger type.

Window A (engine alive, ~15:48–15:55):

• message_posted trigger: fired, message posted within ~1s
• manual trigger: fired, message posted
• webhook trigger (POST /hooks/{id} with x-webhook-secret): fired, message posted in <5s
• cron * * * * * and interval 60s: fired once (15:48:41), then never again over 8+ minutes of watching

Window B (same relay, same channel, ~30 minutes later, 16:20+):

• webhook trigger on a freshly created enabled workflow: HTTP 202, {"run_id":"…","status":"pending"} — never executed: no message landed even 15+ minutes later (vs <5s in Window A)
• manual trigger (workflows trigger) on the same workflow: accepted with a run_id — never executed, no message ever landed

Nothing about the workflow changed between windows; the engine simply stopped processing. Two days earlier (2026-08-19) another operator on the same relay saw the same thing: manual trigger accepted + run_id, no execution, no resulting action.

Suspected cause

The engine loop is a single unsupervised task spawned once at startup:

// crates/buzz-relay/src/main.rs:658
let wf_cron = Arc::clone(&workflow_engine);
tokio::spawn(async move { wf_cron.run().await });

If run() returns an error or the task panics, there is no supervisor, no restart, no health signal, and no alarm. Handlers keep creating pending runs that nothing will ever pick up, which matches the observed "202 + run_id + eternal pending" exactly. The one-fire-then-silence cron behavior in Window A also fits a loop that died mid-tick.

Suggested fixes

1. Supervise the engine task: restart-on-exit with backoff, and log loudly on any exit.
2. Expose a liveness/health signal for the engine loop (metric or admin endpoint) so a dead engine is observable instead of silent.
3. On engine (re)start, sweep and requeue orphaned pending runs so accepted work is not lost forever.
4. Short term: restarting the hosted relay process restores execution (Window A behavior is consistent with a recent restart).

Repro

1. Create a minimal enabled workflow with trigger: {on: webhook} and one send_message step.
2. POST /hooks/{workflow_id} with the secret → observe 202 + pending.
3. If the relay's engine loop has died (give a long-uptime relay time, or kill the task), the run never executes and the endpoint keeps returning 202 while nothing ever runs.

[joydai2026-del]

Additional evidence, same relay, same day: workflow deletion is affected the same way as execution. buzz workflows delete --workflow <id> returns {"accepted":true, "event_id":...} but the definition remains in workflows list indefinitely. Reproduced with two different signing keys, including the workflow author's own key (delete events fb1159267010854e453c39d89f3341494b5d1043771a493ac0f6cfd9b2c9fe86 and f3f268142a406ffabe2b42de02d4487eb08c65fd005ee03a0c92ff5190fa1328, both accepted, zero effect after multiple minutes). So the failure is not limited to the scheduler loop: workflow-related event processing as a whole stops being applied while the relay keeps ACKing.