Skip to content

fix: audit long task replacements without an endless Slack retry loop - #29

Merged
SergioTCG merged 1 commit into
mainfrom
fix/team-payload-audit-loop
Oct 8, 2026
Merged

SergioTCG merged 1 commit into
mainfrom
fix/team-payload-audit-loop

Conversation

@SergioTCG

Copy link
Copy Markdown
Owner

Summary

A queued task replaced with an instruction longer than about 4,000 characters could never be dispatched, and SAB retried its failing Slack card update every 3 seconds, forever.

This PR posts oversized revisions as new cards and backs off failing card updates.

Root cause

  • The instruction card is posted in both channels with chat.postMessage, which accepts long text.
  • A sab team replace revision updates both cards through chat.update, which rejects text above roughly 4,000 characters with msg_too_long.
  • The 3-second reconciliation sweep retried the failing update indefinitely.
  • Dispatch requires a fully audited revision, so the replaced task could never be claimed.

In the live daemon, one replaced (later cancelled) task produced:

  • 34,192 failures between two restarts, two Slack calls per sweep, about 40 chat.update calls a minute;
  • about 210,000 log lines in total.

Successful replacements in the journal were at most 3,309 characters; the failing one is 4,703.

Fix

The card updater moves to daemon/team-audit.mjs as createPayloadAuditUpdater(); teamTaskPayloadText() moves unchanged.

  • Oversized revisions: the revision is posted once, in full, as a new card in each channel. It is recorded per side in payloadRevisionCards, so a retry never posts it twice. The original card is edited into a short pointer to it. The same fallback applies when Slack reports msg_too_long for shorter text.
  • Backoff: a failing update backs off per task, from 30 s doubling up to 30 min. A newer revision is attempted immediately.
  • Unchanged guarantees:
    • the rendered revision is captured first and recorded only after both cards succeed;
    • a missing card is a failure;
    • updates stay serialized per task;
    • dispatch can never cross a mixed-card state.

Tests (test/team-audit.test.mjs, fake Slack enforcing the update limit)

  • A 4,703-character replacement is audited, and the revised queued task can then be claimed.
  • A short revision is still edited in place.
  • Slack's msg_too_long below the local threshold falls back to a revision card.
  • A failed pointer edit is retried without reposting the revision.
  • A permanently failing card makes at most 6 attempts over 10 simulated minutes, against 200 before.
  • A missing card fails, and the rendered revision, not a newer one, is recorded.

Each mutation was applied on its own:

Mutation Result
Old in-place-only behaviour 3 red
No backoff 1 red
Repost on every attempt 2 red
Record the latest version instead of the rendered one 1 red
No fallback on a short msg_too_long 1 red

Validation

  • npm test: 591 pass. npm run check, npm run audit (0), node --check, Pi extension and shellcheck all pass. Every daemon named import resolves.

Deployment notes

  • A daemon restart is needed. There is no dependency or state migration; payloadRevisionCards is additive.
  • After the restart, the one stuck (cancelled) task's revision is posted as a new card in its two channels, and the loop stops.

🤖 Generated with Claude Code

https://claude.ai/code/session_011Xs8AVBVrTGfpzUMiCQp2R

A queued task's instruction is shown on a Slack card in both channels. A
replacement must update both cards before the task may be dispatched. The
cards are posted with chat.postMessage, which accepts long text. Updates
went through chat.update, which rejects text above roughly 4,000 characters
with msg_too_long.

A replacement of about 4,700 characters therefore failed on every 3-second
reconciliation sweep, indefinitely. That cost two Slack calls per sweep,
tens of thousands of log lines a day, and much of the chat.update rate
budget. Dispatch requires a fully audited revision, so the replaced task
could never be claimed.

The card updater moves to daemon/team-audit.mjs
(createPayloadAuditUpdater, with the unchanged teamTaskPayloadText):
- A revision too long to edit in place is posted once, in full, as a new
  card in each channel. It is recorded per side in payloadRevisionCards so
  a retry never posts it twice. The original card is edited into a short
  pointer to it. The same fallback applies when Slack reports msg_too_long
  below the local threshold.
- A failing update backs off per task, from 30 s doubling up to 30 min.
  A newer revision is attempted at once.
- The rendered revision is still captured first and recorded only after
  both cards succeed, and updates remain serialized per task, so dispatch
  never crosses a mixed-card state.

Tests (test/team-audit.test.mjs) use a fake Slack that enforces the update
limit:
- a 4,703-character replacement is audited, and the revised task can be
  claimed;
- a short revision is still edited in place;
- the threshold fallback works;
- a partial failure is retried without reposting;
- a permanently failing card makes at most 6 attempts over 10 simulated
  minutes, against 200 before;
- the rendered version is the one recorded.
Restoring the old in-place-only behaviour, or removing the backoff, the
repost guard, the version capture or the fallback, each turns the tests
red. Full suite: 591 pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Xs8AVBVrTGfpzUMiCQp2R
@SergioTCG
SergioTCG merged commit 171706c into main Oct 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants