fix(pssh): retry once with a fresh client on SessionError - #317
Open
speriaswamy-amd wants to merge 3 commits into
Open
fix(pssh): retry once with a fresh client on SessionError#317speriaswamy-amd wants to merge 3 commits into
speriaswamy-amd wants to merge 3 commits into
Conversation
Long-running suites reuse a single ParallelSSHClient for the whole module. If the underlying session goes stale — common after a benchmark step that keeps the connection idle for tens of minutes — the next run_command raises SessionError and the test fails on a trivial follow-up command rather than on anything it was actually validating. exec() and exec_cmd_list() now route through _run_command_with_session_retry, which logs, rebuilds the client via the extracted _recreate_parallel_client, and retries exactly once. A second failure propagates unchanged, so a genuinely broken host still fails fast. _recreate_parallel_client is the client-construction block lifted out of __init__ verbatim; __init__ now calls it, so there is one construction path. Tests: the two setUp methods move from a decorator-scoped patch to patcher.start()/addCleanup, because the retry constructs a client during the test body, not just during setUp. Adds coverage for retry-then-succeed (with and without a timeout) and retry-then-fail on both exec paths. Co-Authored-By: Claude <noreply@anthropic.com>
…sh-session-retry (bring branch up to date)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4 of 12 in a stack that replaces #184. Base: #316.
Why
Long-running suites reuse a single
ParallelSSHClientfor the whole module. If the underlying session goes stale — common after a benchmark step that leaves the connection idle for tens of minutes — the nextrun_commandraisesSessionErrorand the test fails on a trivial follow-up command rather than on anything it was validating.What changed
_recreate_parallel_client()— the client-construction block lifted out of__init__verbatim;__init__now calls it, so there is one construction path._run_command_with_session_retry()— logs, rebuilds the client, retries once. A second failure propagates unchanged, so a genuinely broken host still fails fast.exec()andexec_cmd_list()route theirrun_commandcalls through it.Tests
The two
setUpmethods move from a decorator-scoped@patchtopatcher.start()+addCleanup, because the retry constructs a client during the test body, not just duringsetUp. New coverage: retry-then-succeed (with and without a timeout) and retry-then-fail, on both exec paths.