What happens
Every so often in production, await AudioSource.aclose() never returns. aclose is async, but all it does is call self._ffi_handle.dispose(), a synchronous FFI drop that never yields. When the drop hangs, the whole asyncio loop hangs with it. About 60 s later the agents worker's watchdog kills the job as unresponsive, and the room ends.
Sequence
- We create an
AudioSource, publish a LocalAudioTrack from it, and push frames with capture_frame in a pump task.
- After a while we cancel the pump, run
await local_participant.unpublish_track(sid), which returns, and then run await source.aclose() about 50 ms later.
aclose() never returns.
faulthandler dump taken when the watchdog fired (main thread):
stop_ringback
-> livekit/rtc/audio_source.py aclose
-> livekit/rtc/_ffi_client.py dispose
-> livekit_ffi_drop_handle
Asks
- Add a sync close function: DataStream and DataTrack have them, can add it to AudioStream for parity and so we can close it in a sync context out of the event loop.
- Fix the hang. Seems like something is deadlocking.
- Make
aclose() not block the loop.: Move the sync operation off the main event loop.
Right now the only way to keep the loop safe is:
def _dispose(source):
asyncio.run(source.aclose()) # aclose never suspends, so a throwaway loop is fine
await asyncio.wait_for(asyncio.to_thread(_dispose, source), timeout=2)
If the drop still hangs, that worker thread stays stuck.
BackgroundAudioPlayer.aclose in livekit-agents disposes its source the same way, so it is probably exposed to the same hang when a call ends.
What happens
Every so often in production,
await AudioSource.aclose()never returns.acloseisasync, but all it does is callself._ffi_handle.dispose(), a synchronous FFI drop that never yields. When the drop hangs, the whole asyncio loop hangs with it. About 60 s later the agents worker's watchdog kills the job as unresponsive, and the room ends.Sequence
AudioSource, publish aLocalAudioTrackfrom it, and push frames withcapture_framein a pump task.await local_participant.unpublish_track(sid), which returns, and then runawait source.aclose()about 50 ms later.aclose()never returns.faulthandler dump taken when the watchdog fired (main thread):
Asks
aclose()not block the loop.: Move the sync operation off the main event loop.Right now the only way to keep the loop safe is:
If the drop still hangs, that worker thread stays stuck.
BackgroundAudioPlayer.aclosein livekit-agents disposes its source the same way, so it is probably exposed to the same hang when a call ends.