Feishu frontend cannot recover when the SDK event loop stalls while the process stays alive #825

Activity

1. aron-intframe commented on Oct 4, 2026

   

   aron-intframe

   on Oct 4, 2026

   More actions

   Reproduced this against real lark-oapi 1.6.8 and 1.7.3, and the stall has a single site: ws.Client._get_conn_url() is a synchronous requests.post with no timeout=, called from inside the _connect() coroutine, so it parks the event-loop thread rather than only that coroutine.

   Startup phase, against a local endpoint that accepts the TCP connection and never answers. A heartbeat coroutine scheduled onto lark_oapi.ws.client.loop measures loop progress:

   t+ 4.0s  start()_returned=False  raised=None  loop_ticks=0  loop_running=True  process_alive=True
   t+12.0s  start()_returned=False  raised=None  loop_ticks=0  loop_running=True  process_alive=True
   
     File ".../lark_oapi/ws/client.py", line 162, in start
       loop.run_until_complete(self._connect())
     File ".../lark_oapi/ws/client.py", line 194, in _connect
       conn_url = self._get_conn_url()
     File ".../lark_oapi/ws/client.py", line 237, in _get_conn_url
       response = requests.post(
     File ".../urllib3/connectionpool.py", line 793, in urlopen
       response = self._make_request(
     File ".../urllib3/connectionpool.py", line 540, in _make_request
       response = conn.getresponse()
     File "/usr/lib/python3.12/socket.py", line 707, in readinto
       return self._sock.recv_into(b)
   

   Reconnect phase, which is the case you describe: one event handled, connection dropped, reconnect discovery hangs.

   [10:40:26] handler ran, events=1
   [10:40:27] reconnecting -> endpoint discovery POST
   t+ 2.0s events=1 loop_ticks=  1 start()_returned=False raised=None conn=None process_alive=True
   t+12.0s events=1 loop_ticks=  1 start()_returned=False raised=None conn=None process_alive=True
   

   So the structural claim holds. cli.start() stays inside loop.run_until_complete(_select()), fsapp.main()'s while True never runs again, and the process stays alive with loop_ticks frozen. The missing timeout is identical in 1.7.3 (current latest), so an SDK upgrade does not remove the need for an application-level watchdog:

   lark-oapi 1.6.8 _get_conn_url has timeout= False
   lark-oapi 1.7.3 _get_conn_url has timeout= False
   

   I then ran a watchdog whose first rule is the same as #826's (loop.call_later beat, monotonic staleness check on a separate thread, os._exit) against the real SDK in both phases, with thresholds shortened to check=1s / stall=6s, plus two more scenarios:

   [watchdog] event loop made no progress for 6s -> exit 1
   startup-stall: watchdog_fired=True start()_still_parked=True start()_raised=None process_alive=True
   [watchdog] event loop made no progress for 6s -> exit 1
   reconnect-stall: watchdog_fired=True start()_still_parked=True start()_raised=None process_alive=True
   [watchdog] reconnect started 6s ago and never completed -> exit 1
   deaf: watchdog_fired=True start()_still_parked=True start()_raised=None process_alive=True
   healthy-idle: watchdog_fired=False start()_still_parked=True start()_raised=None process_alive=True
   

   Loop progress covers both stall phases. The deaf row is the gap I would fold into #826 before merge, and it is reachable with the frontend's current settings (auto_reconnect defaults to True). If reconnect-time endpoint discovery returns a non-retryable client code, _try_connect re-raises it, _reconnect propagates, and the entire _receive_message_loop task dies. Local endpoint answering HTTP 200 with {"code":99991663,...}:

   lark_oapi.ws.exception.ClientException: 99991663: app ticket is invalid
   t+ 3.0s start()_returned=False raised=None loop_ticks=  6 conn=None process_alive=True
   t+ 9.0s start()_returned=False raised=None loop_ticks= 18 conn=None process_alive=True
   

   loop_ticks keeps advancing at full rate while _conn stays None forever, so a loop-progress watchdog reports healthy on a frontend that can no longer receive anything. Your own docstring already says a live loop does not prove a healthy connection, and the SDK gives you the signal for free in both 1.6.8 and 1.7.3: cli.on_reconnecting / cli.on_reconnected. Chaining them onto beat() is a few lines:

   prev_ing, prev_ed = cli.on_reconnecting, cli.on_reconnected
   recon = [None]
   cli.on_reconnecting = lambda: (recon.__setitem__(0, time.monotonic()), prev_ing())
   cli.on_reconnected = lambda: (recon.__setitem__(0, None), prev_ed())
   # inside watch(), next to the last_beat check:
   if recon[0] and time.monotonic() - recon[0] >= timeout:
       ...  # reconnect started and never completed

   That is what the deaf row fired on, and healthy-idle (connected, zero events for 14s against a 6s threshold) stayed quiet, so it does not violate the idle requirement in your proposed scope.

   One more freeze that no progress-based check can see, as an upstream note rather than something for this PR: _connect() acquires self._lock and then returns early when self._conn is not None, before entering the try whose finally releases it.

   B after _connect() early return: lock_held=True
   B send: blocked on the leaked lock (timed out after 2s)
   B disconnect: blocked on the leaked lock (timed out after 2s)
   

   I reached that by awaiting _connect() directly with _conn already set, so I have not proven that a live session races into it. If it does, the loop keeps ticking while every send and _disconnect() blocks forever.

   All of the above needs no Feishu credentials: a local socket that accepts and never answers, a local HTTP handler returning the error envelope, and a small fake conn object whose recv() raises.

   The shape here (process alive, supervisor satisfied, work stopped) is the failure class our own scheduled agents keep producing, and the check we use reports process state and log staleness together instead of answering "is it running" (ours): https://github.com/aron-intframe/agent-guards
2. ytahml commented on Oct 4, 2026

   

   ytahml

   on Oct 4, 2026

   Last edited by ytahml

   Author

   More actions

   @aron-intframe Thanks for the detailed reproduction! The startup and reconnect stall results are helpful confirmation for #826.

   For this PR, I’m keeping the scope limited to event-loop stalls. The deaf case you found is useful additional context, but detecting it would introduce a separate reconnect recovery policy, so I’d prefer to discuss that independently if the maintainers are interested.