Skip to content

fix(worker): rate-limited enrichment jobs never actually defer (missing DelayedError) - #38

Open
AlaganAk wants to merge 1 commit into
SkxOverKill:mainfrom
AlaganAk:fix/rate-limited-jobs-never-defer
Open

fix(worker): rate-limited enrichment jobs never actually defer (missing DelayedError)#38
AlaganAk wants to merge 1 commit into
SkxOverKill:mainfrom
AlaganAk:fix/rate-limited-jobs-never-defer

Conversation

@AlaganAk

@AlaganAk AlaganAk commented Sep 6, 2026

Copy link
Copy Markdown

Bug

The enrichment worker in src/worker.ts defers a job when a provider quota is exhausted:

const limited = outcomes.find((o) => o.status === "rate_limited");
if (limited && limited.status === "rate_limited") {
  await job.moveToDelayed(Date.now() + Math.max(1000, limited.retryAfterMs));
  return { deferred: true, retryAfterMs: limited.retryAfterMs };
}

Per the BullMQ contract, a processor that calls moveToDelayed() must exit by throwing new DelayedError() (official docs, "Process Step jobs"). If the processor returns normally instead, the worker treats the run as a normal completion and finalizes the job as completed - the delayed marker is discarded and the deferred retry never runs. Returning normally after moveToDelayed is also the known cause of Missing lock for job errors, because the worker tries to move the already-relocated job again (bullmq#3295, SO: "Missing lock" after moveToDelayed).

Impact

Every enrichment job that hits a provider quota is silently marked completed with {deferred: true}. The intended "reschedule past the reset" behavior (the comment in the code explicitly says a rate-limited job is not a failure - reschedule it past the reset) never happens. On the free tiers this platform is designed around - VirusTotal 4 req/min / 500/day, AbuseIPDB 1000/day, keyless NVD 5/30s, all routinely exhausted during bulk imports - any indicator whose enrichment was rate-limited is silently never enriched until a later manual or bulk pass touches it again.

Fix

Throw new DelayedError() after moveToDelayed(), exactly as the official BullMQ pattern prescribes. The worker then leaves the job in the delayed set (without counting it as a failure) and it re-runs after the provider quota resets. One-line behavioral fix plus the corresponding import.

Verification

  • bullmq ^6.3.2 exports DelayedError (stable API since v3).
  • Behavior matches the documented pattern: worker sees DelayedError, leaves the job delayed, no attempt burned, no Missing lock log noise.

Co-authored-by: Pair Extraordinaire alenakinfo404@gmail.com

…obs actually defer

The enrichment worker defers a job when a provider quota is exhausted:

  await job.moveToDelayed(Date.now() + retryAfterMs);
  return { deferred: true, ... };

Per the BullMQ contract, a processor that calls moveToDelayed() must exit by
throwing DelayedError. If the processor returns normally instead, the worker
treats the run as a normal completion and finalizes the job as *completed* -
the delayed marker is discarded, and the deferred retry never runs. This is
the documented pattern (docs.bullmq.io "Process Step jobs"), and skipping the
throw is also the known cause of "Missing lock for job" errors, because the
worker tries to move the already-relocated job again when the processor ends.

Impact: every enrichment job that hits a provider quota (VirusTotal 4 req/min /
500/day, AbuseIPDB 1000/day, NVD 5/30s keyless - all routinely exhausted on the
free tiers this platform is designed around) is silently marked completed with
{deferred: true}. The intended "reschedule past the reset" behavior never
happens, so indicators that were rate-limited are never enriched until some
later manual/bulk pass happens to touch them again - exactly the spin-loop the
comment above the code says this path exists to avoid.

Fix: throw new DelayedError() after moveToDelayed(), per the official pattern.
The worker then leaves the job in the delayed set without counting it as a
failure, and it re-runs after the provider quota resets.

Co-authored-by: Pair Extraordinaire <alenakinfo404@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant