fix(worker): rate-limited enrichment jobs never actually defer (missing DelayedError) - #38
Open
AlaganAk wants to merge 1 commit into
Open
Conversation
…obs actually defer
The enrichment worker defers a job when a provider quota is exhausted:
await job.moveToDelayed(Date.now() + retryAfterMs);
return { deferred: true, ... };
Per the BullMQ contract, a processor that calls moveToDelayed() must exit by
throwing DelayedError. If the processor returns normally instead, the worker
treats the run as a normal completion and finalizes the job as *completed* -
the delayed marker is discarded, and the deferred retry never runs. This is
the documented pattern (docs.bullmq.io "Process Step jobs"), and skipping the
throw is also the known cause of "Missing lock for job" errors, because the
worker tries to move the already-relocated job again when the processor ends.
Impact: every enrichment job that hits a provider quota (VirusTotal 4 req/min /
500/day, AbuseIPDB 1000/day, NVD 5/30s keyless - all routinely exhausted on the
free tiers this platform is designed around) is silently marked completed with
{deferred: true}. The intended "reschedule past the reset" behavior never
happens, so indicators that were rate-limited are never enriched until some
later manual/bulk pass happens to touch them again - exactly the spin-loop the
comment above the code says this path exists to avoid.
Fix: throw new DelayedError() after moveToDelayed(), per the official pattern.
The worker then leaves the job in the delayed set without counting it as a
failure, and it re-runs after the provider quota resets.
Co-authored-by: Pair Extraordinaire <alenakinfo404@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
The enrichment worker in
src/worker.tsdefers a job when a provider quota is exhausted:Per the BullMQ contract, a processor that calls
moveToDelayed()must exit by throwingnew DelayedError()(official docs, "Process Step jobs"). If the processor returns normally instead, the worker treats the run as a normal completion and finalizes the job as completed - the delayed marker is discarded and the deferred retry never runs. Returning normally aftermoveToDelayedis also the known cause ofMissing lock for joberrors, because the worker tries to move the already-relocated job again (bullmq#3295, SO: "Missing lock" after moveToDelayed).Impact
Every enrichment job that hits a provider quota is silently marked
completedwith{deferred: true}. The intended "reschedule past the reset" behavior (the comment in the code explicitly says a rate-limited job is not a failure - reschedule it past the reset) never happens. On the free tiers this platform is designed around - VirusTotal 4 req/min / 500/day, AbuseIPDB 1000/day, keyless NVD 5/30s, all routinely exhausted during bulk imports - any indicator whose enrichment was rate-limited is silently never enriched until a later manual or bulk pass touches it again.Fix
Throw
new DelayedError()aftermoveToDelayed(), exactly as the official BullMQ pattern prescribes. The worker then leaves the job in the delayed set (without counting it as a failure) and it re-runs after the provider quota resets. One-line behavioral fix plus the corresponding import.Verification
bullmq^6.3.2 exportsDelayedError(stable API since v3).DelayedError, leaves the job delayed, no attempt burned, noMissing locklog noise.Co-authored-by: Pair Extraordinaire alenakinfo404@gmail.com