ddel error terminates entire ddel job

[carbocation]

In dsub v0.5.2, I'm running into the following when trying to delete a large job. My guess is that at some point we run into a job that has already finished (not actually sure here), but for some reason this terminates the entire ddel process instead of being handled gracefully:

$ ddel --provider google-batch --project [X] --location us-central1 --jobs '[X]' --users '[X]'
Delete running jobs:
  user:
    {'[X]'}

  job-id:
    ['[X]']


Found 4288 tasks to delete.
Traceback (most recent call last):
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/dsub/providers/google_base.py", line 377, in _cancel_batch
    batch.add(cancel_fn(name=op_name, body={}), request_id=op_name)
TypeError: BatchServiceClient.cancel_job() got an unexpected keyword argument 'body'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/grpc/_interceptor.py", line 277, in __call__
    response, ignored_call = self._with_call(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/grpc/_interceptor.py", line 332, in _with_call
    return call.result(), call
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/grpc/_channel.py", line 439, in result
    raise self
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/grpc/_interceptor.py", line 315, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/grpc/_channel.py", line 1193, in with_call
    return _end_unary_response_blocking(state, call, True, None)
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.ABORTED
        details = "the job cannot be cancelled in its current state"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:[X] {created_time:"2026-06-21T[X]", grpc_status:10, grpc_message:"the job cannot be cancelled in its current state"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/jupyter/bin/ddel", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/dsub/commands/ddel.py", line 147, in main
    deleted_tasks = ddel_tasks(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/dsub/commands/ddel.py", line 199, in ddel_tasks
    deleted_tasks, error_messages = provider.delete_jobs(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/dsub/providers/google_batch.py", line 1022, in delete_jobs
    return google_base.cancel(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/dsub/providers/google_base.py", line 418, in cancel
    batch_canceled, batch_messages = _cancel_batch(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/dsub/providers/google_base.py", line 380, in _cancel_batch
    batch.add(cancel_fn(name=op_name), request_id=op_name)
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/google/cloud/batch_v1/services/batch_service/client.py", line 1251, in cancel_job
    response = rpc(
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
    return func(*args, **kwargs)
  File "/opt/conda/envs/jupyter/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.Aborted: 409 the job cannot be cancelled in its current state [resource_name: "projects/[X]/locations/us-central1/jobs/[X]-6005-1"
, request_id: "[X]"
]