feat(aorta): collect per-node traces, survive partial node failure - #331
Open
speriaswamy-amd wants to merge 1 commit into
Open
feat(aorta): collect per-node traces, survive partial node failure#331speriaswamy-amd wants to merge 1 commit into
speriaswamy-amd wants to merge 1 commit into
Conversation
Two problems with profiler artifacts on a multi-node run. First, each node wrote its torch_profiler/ tree to its own filesystem, so the host parser only ever saw the head node's ranks. _collect_multi_node_traces() consolidates every node's trees into <aorta_path>/combined_traces/node_<rank>/: a local copy when the orchestrator shares the head's filesystem, rsync over SSH otherwise, scp -r where rsync is absent. Per-node failures are logged and skipped rather than aborting the collection. The trace-discovery scan skips anything under combined_traces/ so the stale per-node originals cannot shadow the consolidated set, and seeds trace_mtime from the collected tree - without that seed the existing freshest-trace comparison raises UnboundLocalError once trace_dir can be pre-set. Second, run() returned as soon as any node failed, before collecting anything. One flaky node in a multi-hour run therefore discarded every surviving node's traces, forcing a full rerun or a manual TraceLensParser salvage. Collection and artifact discovery now run unconditionally; the run is still reported FAILED or TIMEOUT with the offending nodes named in error_message, but artifacts come from whatever the survivors produced. Co-Authored-By: Claude <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 6/6 — splits #171. Base: #330. Final PR; the tip of this stack is identical to #171's tree.
Why
Two problems with profiler artifacts on a multi-node run.
torch_profiler/tree to its own filesystem, so the host parser only ever saw the head node's ranks.run()returned as soon as any node failed, before collecting anything. One flaky node in a multi-hour run discarded every surviving node's traces, forcing a full rerun or a manualTraceLensParsersalvage.What changed
_collect_multi_node_traces()— consolidates every node's trees into<aorta_path>/combined_traces/node_<rank>/: local copy when the orchestrator shares the head's filesystem, rsync over SSH otherwise,scp -rwhere rsync is absent. Per-node failures are logged and skipped rather than aborting the collection.combined_traces/so the stale per-node originals cannot shadow the consolidated set, and seedstrace_mtimefrom the collected tree — without that seed the existing freshest-trace comparison raisesUnboundLocalErroroncetrace_dircan be pre-set.FAILED/TIMEOUTwith the offending nodes named inerror_message, but artifacts come from whatever the survivors produced.Test
ruffclean. Unit tests 631 → 638 (7 new:combined_tracespath predicate, local copy-tree with recursion guard, collection layout, and partial-failure-still-collects).Live validation from #171 still applies unchanged — this stack's tip is byte-identical to that branch for every non-test file:
cvs run test_aortaon a real 2-node cluster (g17u19 + f16u13, 16×MI300X), 5/5 pytest cases in 148s, traces from both nodes, host parser produced metrics for all 16 ranks.