Skip to content

fix(purchase): the fire sweep starts new purchases without checking the Lambda time left #1582

Closed

Description

@IAmJSD

#1580 turns the fire_scheduled_purchases sweep on by default (D33). FireScheduledDelayedPurchases (internal/purchase/scheduled_fire.go) fires every due row one after another in a single invocation and never checks the remaining time. The 3m30s + 30s + 30s budget (#1578) covers one purchase, not a whole sweep.

With several slow due purchases, a later one can start near the 300s Lambda limit and be cut off partway. The row stays running until the reaper fails it, and operators get the "outcome unknown, verify before retrying" alert. The stable idempotency keys make a double purchase unlikely, but the record can disagree with what was bought.

Fix options: stop claiming new rows when the time left is below the per-purchase budget (the rest fire on the next tick), or fire one row per run on Lambda. Add a test with a deadline-limited context.

Found by both independent reviews of #1580.

Activity

  1. cristim commented on Nov 4, 2026

    @cristim
    Member

    Claimed by orchestrator run 2026-11-04 (platform-burndown). The plan has passed two adversarial review passes; implementation follows in a PR that closes this issue.

  2. cristim commented on Nov 4, 2026

    @cristim
    Member

    Fixed by #1673, merged as 4dde27a. The fire sweep now defers a due row before claiming it when less than the 270s purchase budget remains, logs an Error when nothing could fire, and the AKS and Fargate task_timeout floors are 300 so a deadline below the budget cannot be configured. Verified on real Postgres with the fire sweep and a fake provider: with 10s left, 3 due rows stay scheduled and none reach the provider (main bought all 3); with no deadline all 3 fire; a row past 24h is still failed with no_submission. Note: an external deployment with task_timeout between 60 and 299 now gets a plan-time validation error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions