Skip to content

fix: add depth guard to sync_node_execution to prevent unbounded recursion#3441

Open
falloficaruss wants to merge 1 commit into
flyteorg:masterfrom
falloficaruss:fix/sync-node-execution-depth-guard
Open

fix: add depth guard to sync_node_execution to prevent unbounded recursion#3441
falloficaruss wants to merge 1 commit into
flyteorg:masterfrom
falloficaruss:fix/sync-node-execution-depth-guard

Conversation

@falloficaruss

@falloficaruss falloficaruss commented Jun 18, 2026

Copy link
Copy Markdown

Tracking issue

Closes flyteorg/flyte#7338

Why are the changes needed?

FlyteRemote.sync_node_execution recursively syncs launched-LP executions and parent-node children with no depth limit. A workflow graph with
deeply nested launched LPs triggers RecursionError, crashing the Python process. Every pyflyte run --remote call against such an execution
blows up.

What changes were proposed in this pull request?

  • Added _depth and _max_depth parameters to _sync_execution and sync_node_execution (default _max_depth=50)
    • Both methods raise FlyteAssertion at entry when _depth > _max_depth instead of overflowing the Python call stack
    • Updated all 6 recursive call sites to pass _depth + 1: the per-node sync in _sync_execution, the launched-LP path, dynamic subworkflow
      children, static subworkflow children, and both branch-node child paths

How was this patch tested?

  • Added tests/flytekit/unit/remote/test_recursion_guard.py with 4 tests:
    • test_max_depth_raises_flyte_assertion — verifies FlyteAssertion is raised when depth exceeds limit
    • test_reasonable_depth_does_not_raise — verifies default depth allows normal operation
    • test_nested_under_default_limit — verifies nested calls under the limit succeed
    • test_sync_execution_depth_guard — verifies the guard on _sync_execution independently
    • All 44 existing tests in tests/flytekit/unit/remote/ pass with no regressions

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

    - Add _depth and _max_depth parameters (default 50) to _sync_execution and sync_node_execution
    - Raise FlyteAssertion when nesting exceeds limit instead of crashing with RecursionError
    - Add unit tests verifying depth guard and regression safety

    Fixes #7338

Signed-off-by: Abhishek Shinde <norizzabhii@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] FlyteRemote.sync_node_execution: unbounded recursion (no depth check, no cycle detection)

1 participant