When running a plan with blueapi, it completes and then forever blocks

[oliwenmandiamond]

When trying to use BlueapiClient, I ran into an issue with plans running but never returning. The setup is below:

>>> from blueapi.client import BlueapiClient
>>> bc = BlueapiClient.from_config_file("/home/xol73553/Documents/BLUESKY/i09_blueapi_client_config.yaml")
>>> for device in bc.devices:
...     print(device)
... 
<Response [200]>
dcm
lakeshore
psi2
psj2
psi1
psk1
fsj1
psj1
jphase
pgm
jgap
iid
ienergy_order
fsi1
source_selector
synchrotron
smpm
iidenergy
jid
ienergy
dual_fast_shutter
dual_energy_source
jidcontroller
jpolarisation
ew4000
jidenergy
jenergy
>>> devices = bc.devices
>>> plans = bc.plans
<Response [200]>

Then when trying to run a simple move plan:

>>> plans.move({devices.fsj1: "Out"})
<Response [201]>
<Response [200]>

it successfully moves the shutter. However, it is still forever running and blocking so I'm unable to run any more plans. In the logs, I can see the following in BlueAPI for i09

[11:59 AM]2026-03-20 11:57:34,051     INFO blueapi.worker.task_worker Submitting: task_id='4fa02cbf-1794-45b5-9232-4c29fd32345b' task=Task(name='move', params={'moves': {'fsj1': 'Out'}}, metadata={'user': 'xol73553', 'instrument_session': 'cm12345-1'}) request_id=None is_complete=False is_pending=True errors=[] outcome=None
2026-03-20 11:57:34,052     INFO blueapi.worker.task_worker Got new task: task_id='4fa02cbf-1794-45b5-9232-4c29fd32345b' task=Task(name='move', params={'moves': {'fsj1': 'Out'}}, metadata={'user': 'xol73553', 'instrument_session': 'cm12345-1'}) request_id=None is_complete=False is_pending=True errors=[] outcome=None
2026-03-20 11:57:34,052     INFO blueapi.worker.task Asked to run plan move with {'moves': {'fsj1': 'Out'}} and metadata {'user': 'xol73553', 'instrument_session': 'cm12345-1'} for all runs
2026-03-20 11:57:34,052     INFO bluesky Executing plan <generator object move at 0x7fc9bb3db100>
2026-03-20 11:57:34,053     INFO bluesky.RE.state Change state on <bluesky.run_engine.RunEngine object at 0x7fc9c0de66d0> from 'idle' -> 'running'
2026-03-20 11:57:34,065     INFO blueapi.service.main 10.36.0.21:60270 PUT /worker/task 200
2026-03-20 11:57:34,136     INFO bluesky.RE.state Change state on <bluesky.run_engine.RunEngine object at 0x7fc9c0de66d0> from 'running' -> 'idle'
2026-03-20 11:57:34,137     INFO bluesky Cleaned up from plan <generator object move at 0x7fc9bb3db100>
2026-03-20 11:57:34,137     INFO blueapi.worker.task_worker Task ran successfully - returned: (<AsyncStatus, device: fsj1, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7fc9bb104340>, done>,)
2026-03-20 11:57:34,140    ERROR blueapi.worker.task_worker Type is not JSON serializable: AsyncStatus
TypeError: Type <class 'ophyd_async.core._status.AsyncStatus'> not serializable
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/blueapi/worker/task_worker.py", line 424, in _cycle_with_error_handling
    self._cycle()
  File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/worker/task_worker.py", line 485, in _cycle
    self._report_status()
  File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/worker/task_worker.py", line 570, in _report_status
    self._worker_events.publish(event, correlation_id)
  File "/app/.venv/lib/python3.11/site-packages/blueapi/core/event.py", line 80, in publish
    callback(event, correlation_id)
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/interface.py", line 121, in forward_message
    stomp_client_ref.send(
  File "/app/.venv/lib/python3.11/site-packages/bluesky_stomp/messaging.py", line 213, in send
    serialized_message = self._serializer(obj)
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/bluesky_stomp/serdes.py", line 35, in serialize_message
    return orjson.dumps(
           ^^^^^^^^^^^^^
TypeError: Type is not JSON serializable: AsyncStatus
2026-03-20 11:57:34,141     INFO blueapi.worker.task_worker Awaiting task

So it seems to be trying to serialise the AsyncStatus and then fails as this isn't serializable.

Acceptance Criteria

• Plan runs successfully, doesn't block once finished.

[Villtord]

I can confirm the same issue when I tried to run move motor via terminal in blueapi pod and got same error

2026-03-20 14:36:24,774    ERROR blueapi.worker.task_worker Type is not JSON serializable: WatchableAsyncStatus
TypeError: Type <class 'ophyd_async.core._status.WatchableAsyncStatus'> not serializable
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/blueapi/worker/task_worker.py", line 424, in _cycle_with_error_handling
    self._cycle()
  File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/worker/task_worker.py", line 485, in _cycle
    self._report_status()
  File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/worker/task_worker.py", line 570, in _report_status
    self._worker_events.publish(event, correlation_id)
  File "/app/.venv/lib/python3.11/site-packages/blueapi/core/event.py", line 80, in publish
    callback(event, correlation_id)
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/interface.py", line 121, in forward_message
    stomp_client_ref.send(
  File "/app/.venv/lib/python3.11/site-packages/bluesky_stomp/messaging.py", line 213, in send
    serialized_message = self._serializer(obj)
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/bluesky_stomp/serdes.py", line 35, in serialize_message
    return orjson.dumps(
           ^^^^^^^^^^^^^
TypeError: Type is not JSON serializable: WatchableAsyncStatus

[Villtord]

BlueApi version

dependencies:
  - name: blueapi
    version: "1.12.1"
    repository: "oci://ghcr.io/diamondlightsource/charts"

[oliwenmandiamond]

Tested using BlueAPI version 1.12.0, 1.12.1, and 1.12.2 now. Get same issue on all.

[oliwenmandiamond]

Could this be linked to change in AsyncStatus behaviour?

https://diamondlightsource.slack.com/archives/C04TCA0AF2P/p1770130660214499

[oliwenmandiamond]

Just for more info, if I try and exit out and then try to run another plan I get this error

>>> plans.move({devices.fsj1: "In"})
<Response [201]>
<Response [200]>
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/client.py", line 150, in __call__
    self._client.run_task(req)
  File "/scratch/bluesky_development/blueapi/venv/lib64/python3.12/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/client.py", line 510, in run_task
    return complete.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 451, in result
    self._condition.wait(timeout)
  File "/usr/lib64/python3.12/threading.py", line 355, in wait
    waiter.acquire()
KeyboardInterrupt
>>> plans.move({devices.fsj1: "Out"})
<Response [201]>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/client.py", line 150, in __call__
    self._client.run_task(req)
  File "/scratch/bluesky_development/blueapi/venv/lib64/python3.12/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/client.py", line 509, in run_task
    self._rest.update_worker_task(WorkerTask(task_id=task_id))
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/rest.py", line 202, in update_worker_task
    return self._request_and_deserialize(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/venv/lib64/python3.12/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/rest.py", line 271, in _request_and_deserialize
    raise exception
blueapi.client.rest.BlueskyRemoteControlError: (500, '<Response [500]>') 

With logs

2026-03-20 15:17:18,258     INFO blueapi.service.main 10.36.0.21:32962 POST /tasks 201
2026-03-20 15:17:18,280    ERROR uvicorn.error Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 81, in collapse_excgroups
  |     yield
  |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 192, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1160, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/applications.py", line 107, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/app/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 811, in __call__
    |     await self.app(scope, otel_receive, otel_send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/app/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/fastapi/__init__.py", line 307, in __call__
    |     await self.app(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    |     with recv_stream, send_stream, collapse_excgroups():
    |   File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 592, in log_request_details
    |     response = await call_next(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    |     raise app_exc from app_exc.__cause__ or app_exc.__context__
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    |     with recv_stream, send_stream, collapse_excgroups():
    |   File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 620, in inject_propagated_observability_context
    |     response = await call_next(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    |     raise app_exc from app_exc.__cause__ or app_exc.__context__
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    |     with recv_stream, send_stream, collapse_excgroups():
    |   File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 574, in add_api_version_header
    |     response = await call_next(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    |     raise app_exc from app_exc.__cause__ or app_exc.__context__
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/app/.venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    |     await self.app(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
    |     await route.handle(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
    |     await self.app(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 130, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 116, in app
    |     response = await f(request)
    |                ^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 670, in app
    |     raw_response = await run_endpoint_function(
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 326, in run_endpoint_function
    |     return await run_in_threadpool(dependant.call, **values)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/starlette/concurrency.py", line 32, in run_in_threadpool
    |     return await anyio.to_thread.run_sync(func)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 63, in run_sync
    |     return await get_async_backend().run_sync_in_worker_thread(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
    |     return await future
    |            ^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 986, in run
    |     result = context.run(func, *args)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    |     return func(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 378, in set_active_task
    |     active_task = runner.run(interface.get_active_task)
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    |     return func(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.11/site-packages/blueapi/service/runner.py", line 136, in run
    |     return self._subprocess.apply(
    |            ^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 360, in apply
    |     return self.apply_async(func, args, kwds).get()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 774, in get
    |     raise self._value
    | multiprocessing.pool.MaybeEncodingError: Error sending result: 'TrackableTask(task_id='ff095f31-8031-41bb-9bc3-c0e7f374422f', task=Task(name='move', params={'moves': {'fsj1': 'In'}}, metadata={'user': 'xol73553', 'instrument_session': 'cm12345-1'}), request_id=None, is_complete=True, is_pending=False, errors=["Error(s) publishing event: state=<WorkerState.IDLE: 'IDLE'> task_status=TaskStatus(task_id='ff095f31-8031-41bb-9bc3-c0e7f374422f', result=TaskResult(outcome='success', result=(<AsyncStatus, device: fsj1, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7f0b54ce3e40>, done>,), type='tuple'), task_complete=True, task_failed=False) errors=[] warnings=[] (1 sub-exception)"], outcome=TaskResult(outcome='success', result=(<AsyncStatus, device: fsj1, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7f0b54ce3e40>, done>,), type='tuple'))'. Reason: 'TypeError("cannot pickle '_asyncio.Task' object")'
    +------------------------------------
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1160, in __call__
    await super().__call__(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/applications.py", line 107, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/app/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 811, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/app/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/fastapi/__init__.py", line 307, in __call__
    await self.app(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 592, in log_request_details
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    raise app_exc from app_exc.__cause__ or app_exc.__context__
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 620, in inject_propagated_observability_context
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    raise app_exc from app_exc.__cause__ or app_exc.__context__
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 574, in add_api_version_header
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    raise app_exc from app_exc.__cause__ or app_exc.__context__
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/app/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/.venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 130, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 116, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 670, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 326, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/starlette/concurrency.py", line 32, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 63, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 986, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/main.py", line 378, in set_active_task
    active_task = runner.run(interface.get_active_task)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py", line 151, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/blueapi/service/runner.py", line 136, in run
    return self._subprocess.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 360, in apply
    return self.apply_async(func, args, kwds).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: 'TrackableTask(task_id='ff095f31-8031-41bb-9bc3-c0e7f374422f', task=Task(name='move', params={'moves': {'fsj1': 'In'}}, metadata={'user': 'xol73553', 'instrument_session': 'cm12345-1'}), request_id=None, is_complete=True, is_pending=False, errors=["Error(s) publishing event: state=<WorkerState.IDLE: 'IDLE'> task_status=TaskStatus(task_id='ff095f31-8031-41bb-9bc3-c0e7f374422f', result=TaskResult(outcome='success', result=(<AsyncStatus, device: fsj1, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7f0b54ce3e40>, done>,), type='tuple'), task_complete=True, task_failed=False) errors=[] warnings=[] (1 sub-exception)"], outcome=TaskResult(outcome='success', result=(<AsyncStatus, device: fsj1, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7f0b54ce3e40>, done>,), type='tuple'))'. Reason: 'TypeError("cannot pickle '_asyncio.Task' object")'

[oliwenmandiamond]

Okay, I've narrowed this down a bit

>>> bc.instrument_session = "cm12345-1"
>>> plans.set_absolute(devices.fsj1, "Out")
<Response [201]>
<Response [200]>
>>> plans.set_absolute(devices.fsj1, "In")
<Response [201]>
<Response [200]>
>>> plans.set_absolute(devices.fsj1, "Out")
<Response [201]>
<Response [200]>
>>> plans.set_absolute(devices.fsj1, "In")
<Response [201]>
<Response [200]>
>>> plans.move({devices.fsj1: "Out"})
<Response [201]>
<Response [200]>

Dodal set_absolute plan seems to be working fine, moved the shutter and returned instantly. However, as soon as I use dodal move plan, it moves shutter and then doesn't return and results in logs above!

The plans defined are:

def set_absolute(
    movable: Movable[T], value: T, group: Group | None = None, wait: bool = False
) -> MsgGenerator:
    """Set a device, wrapper for `bp.abs_set`.

    Args:
        movable (Movable[T]): The device to set.
        value (T): The new value.
        group (Group | None, optional): The message group to associate with the setting,
            for sequencing. Defaults to None.
        wait (bool, optional): The group should wait until all setting is complete (e.g.
            a motor has finished moving). Defaults to False.

    Returns:
        MsgGenerator: Plan.

    Yields:
        Iterator[MsgGenerator]: Bluesky messages.
    """
    return (yield from bps.abs_set(movable, value, group=group, wait=wait))

...

def move(moves: Mapping[Movable[T], T], group: Group | None = None) -> MsgGenerator:
    """Move a device, wrapper for `bp.mv`.

    Args:
        moves (Mapping[Movable, T]): Mapping of Movables to target positions.
        group (Group | None, optional): The message group to associate with the setting,
            for sequencing. Defaults to None.

    Returns:
        MsgGenerator: Plan.

    Yields:
        Iterator[MsgGenerator]: Bluesky messages.
    """
    return (
        yield from bps.mv(*itertools.chain.from_iterable(moves.items()), group=group)
    )

[oliwenmandiamond]

Okay, I understand why it fails now. So the difference between the two is this

(function) def abs_set(
    obj: Movable[Unknown],
    ...,
    group: Hashable | None = None,
    wait: bool = False
) -> MsgGenerator[Status]

(function) def mv(
    *args: Movable[Unknown] | Any,
    group: Hashable | None = None,
    timeout: float | None = None,
    **kwargs: Unknown
) -> MsgGenerator[tuple[Status, ...]]

BlueAPI sees the tuple and not that it contains AsyncStatus inside so tries to serialise it

[dan-fernandes]

@oliwenmandiamond and I dug into this on I09. We were able to replicate it on my laptop locally using the system test BlueAPI environment. The issue does appear to be as Oli has mentioned above.