Skip to content

Handle errors from unbatched completeJob/failJob - #643

Open
cpruijsen wants to merge 1 commit into
graphile:mainfrom
cpruijsen:fix/issue-641-2ef856b0
Open

cpruijsen wants to merge 1 commit into
graphile:mainfrom
cpruijsen:fix/issue-641-2ef856b0

Conversation

@cpruijsen

Copy link
Copy Markdown

Fixes #641.

With the default completeJobBatchDelay/failJobBatchDelay of -1, the unbatched completeJob/failJob now handle a failed outcome write the same way the batched path does. Before, they returned the promise from batchCompleteJobs/batchFailJobs, which the worker does not await. If that query rejected with an error withRetries does not retry (for example Connection terminated unexpectedly, which has no code), nothing logged it or emitted an event. The job stayed locked until the stale-lock reset, and the process exited on unhandledRejection if the app had no listener.

In src/main.ts the batched error handlers move, with unchanged bodies, into handleCompleteJobsError/handleFailJobsError. The batched path passes them to batch() as before, and the unbatched path attaches them with .then(null, ...). With or without batching, a failure to record a job's outcome is now:

  • emitted as pool:fatalError with action: "completeJob" or "failJob",
  • logged with the job id(s),
  • followed by a graceful pool shutdown, with workerPool.promise rejecting with the error.
flowchart LR
  W[worker calls completeJob/failJob] --> D{batch delay >= 0?}
  D -- yes --> B["batch() errorHandler"]
  D -- "no (default -1)" --> U["batchCompleteJobs / batchFailJobs<br/>.then(null, handler)"]
  B --> H[handleCompleteJobsError / handleFailJobsError]
  U --> H
  H --> E[pool:fatalError + logger.error + gracefulShutdown]
Loading

I kept the graceful shutdown because one failed unbatched write leaves the queue inconsistent in the same way as a failed batch. If you would rather log and emit only and keep the pool running, that is a small change.

Side effect: the unbatched completeJob/failJob now resolve after the handler runs instead of rejecting. Their declared return type is void and the worker ignores the value.

__tests__/main.unbatchedOutcomeErrors.test.ts makes the outcome query reject once with an error that has no code, then checks the pool:fatalError action, the logged job id, and the workerPool.promise rejection. Without the fix both cases fail: pool:fatalError never fires and Jest reports the unhandled rejection.

The default (unbatched) completeJob/failJob functions returned promises that nobody awaited or attached a rejection handler to, so a non-retryable error recording a job's outcome became an unhandled rejection. Attach the same error handler as the batched path so the failure is logged with the job id, emitted as pool:fatalError, and the pool shuts down gracefully.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unhandled rejection when recording a job's outcome fails with unbatched completeJob/failJob

1 participant