Skip to content

AudioSource.aclose() can hang forever in livekit_ffi_drop_handle` after the track is unpublished, and blocks the event loop while it does #832

Description

@ziv-nagish

What happens

Every so often in production, await AudioSource.aclose() never returns. aclose is async, but all it does is call self._ffi_handle.dispose(), a synchronous FFI drop that never yields. When the drop hangs, the whole asyncio loop hangs with it. About 60 s later the agents worker's watchdog kills the job as unresponsive, and the room ends.

Sequence

  1. We create an AudioSource, publish a LocalAudioTrack from it, and push frames with capture_frame in a pump task.
  2. After a while we cancel the pump, run await local_participant.unpublish_track(sid), which returns, and then run await source.aclose() about 50 ms later.
  3. aclose() never returns.

faulthandler dump taken when the watchdog fired (main thread):

stop_ringback
  -> livekit/rtc/audio_source.py  aclose
  -> livekit/rtc/_ffi_client.py   dispose
  -> livekit_ffi_drop_handle

Asks

  1. Add a sync close function: DataStream and DataTrack have them, can add it to AudioStream for parity and so we can close it in a sync context out of the event loop.
  2. Fix the hang. Seems like something is deadlocking.
  3. Make aclose() not block the loop.: Move the sync operation off the main event loop.

Right now the only way to keep the loop safe is:

def _dispose(source):
    asyncio.run(source.aclose())  # aclose never suspends, so a throwaway loop is fine

await asyncio.wait_for(asyncio.to_thread(_dispose, source), timeout=2)

If the drop still hangs, that worker thread stays stuck.

BackgroundAudioPlayer.aclose in livekit-agents disposes its source the same way, so it is probably exposed to the same hang when a call ends.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions