Description
We are getting transient errors from GCP when running task with Deferrable=True, which cause trigger to fail.
Async function get_batch from hooks/dataproc.py can takes param retry: AsyncRetry, but there is currently no way how to pass it from operator to trigger.
Use case/motivation
Extend operator params so it is possible to pass retries also for async get_batch in trigger. We are generating unique batch_id for each task run, so we can't use task-level retries.
Related issues
#73675
Are you willing to submit a PR?
Code of Conduct
Description
We are getting transient errors from GCP when running task with Deferrable=True, which cause trigger to fail.
Async function get_batch from hooks/dataproc.py can takes param
retry: AsyncRetry, but there is currently no way how to pass it from operator to trigger.Use case/motivation
Extend operator params so it is possible to pass retries also for async get_batch in trigger. We are generating unique batch_id for each task run, so we can't use task-level retries.
Related issues
#73675
Are you willing to submit a PR?
Code of Conduct