Add durable execution to SnowparkContainerJobOperator

[justinpakzad]

---

Snowpark container services jobs are often long running jobs and the operator polls Snowflake until a terminal status is reached. As per the ResumableJobMixin documentation this operator is a textbook candidate for durable execution. If the worker is restarted or crashed mid-way through, the next retry reconnects to the already running SPCS job instead of submitting a duplicate. If the job has already completed, then polling is skipped and the job finalized. If the prior job had failed or no longer exists, a fresh job is submitted. By default the operator would now run in durable mode. It only applies to the synchronous polling path and has no effect with wait_for_completion=False or deferrable=True.

Couple of changes worth mentioning:

• Both poll_until_complete and get_job_result can finalize the job (check status, fetch logs, and drop the service) depending on the state of the job upon reconnect, so a flag was introduced (_poll_until_complete_ran) to avoid duplicate finalization. Similar approach to the SnowflakeSqlApiOperator.
• The status fetch was split into _describe_status which is used by the polling loop so Snowflake errors still surface, and get_job_status, which is used only for the reconnect decision. get_job_status catches an "object does not exist" error (errno 2003) and returns a not-found sentinel, so a non-existent service resubmits fresh instead of raising.
• Since ResumableJobMixin only exists on Airflow 3.3+, the operator falls back to a stub on older versions so it stays importable and always submits fresh (same as the other operators that have durable execution).

The existing operator tests were updated for the new methods and new tests cover the durable paths. This was also tested end to end against a Snowflake instance (please see screenshots in the comment below). A durable section was also added to the docs.

Was generative AI tooling used to co-author this PR?

• Yes (Claude Opus 4.8)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[justinpakzad]

In order to test the reconnect paths, I subclassed the operator and intentionally raised an error where
try_number=1, after submit_job was called.

Reconnects to a running job:

Job already completed (polling skipped and job is finalized):

Job no longer exists (submits fresh):