bug: get_public_key() has no retry unlike sign_message() — transient failure at startup hangs validator waiting for manual intervention

[pplmaverick]

Environment

• arc-node version: v0.7.3 (90d71dc, HEAD of main as of 2026-08-16)
• Affected crate: arc-remote-signer (crates/remote-signer)
• Signing config: SigningConfig::Remote (consensus node configured to use an external gRPC signing service — the HSM/KMS-backed signing path)
• Component: crates/malachite-app (Node::consensus_identity())

Description

RemoteSignerClient exposes two RPCs to the remote signing service, but only one is retried.

• sign_message() → sign_message_with_retry() (crates/remote-signer/src/client.rs:172-215) wraps the gRPC call in backon's RetryableWithContext, using the configured RetryConfig (default: 3 retries, exponential backoff 100ms → 5s).
• get_public_key() (client.rs:127-130) calls get_public_key_internal() (client.rs:133-170) exactly once. Any transient failure returns Err(RemoteSigningError::Status(...)) immediately, with zero retry.

get_public_key() is called exactly once, on the consensus startup path, from Node::consensus_identity() when SigningConfig::Remote is configured:

// crates/malachite-app/src/node.rs:373-376
let public_key = remote_provider
    .public_key()
    .await
    .wrap_err("Failed to get public key from remote signer")?;

RemoteSigningProvider::public_key() (provider.rs:73-108) is a cache-then-fetch wrapper around client.get_public_key() — it does not add its own retry either.

If this single call fails, the error propagates via ? through setup_node_identity() → start() → run():

// crates/malachite-app/src/node.rs:769-781
Err(e) => {
    let startup_error = e.wrap_err("Node failed to start");
    error!("{startup_error:?}");
    error!("Manual intervention required! Waiting for termination signal (SIGTERM)...");
    wait_for_termination().await;
    return Err(startup_error);
}

The process does not exit and does not retry — it blocks indefinitely in wait_for_termination(). Because the process stays alive, a supervisor watching for exit codes (systemd Restart=on-failure, k8s liveness via process exit) will not automatically restart it.

This creates an asymmetric reliability profile: consensus signing during steady-state is resilient to transient connectivity blips (3 retries with backoff). Node startup is not.

Steps to Reproduce

1. Configure a consensus node with SigningConfig::Remote pointing at a working remote-signer gRPC endpoint.
2. Arrange for the first public_key() gRPC call in consensus_identity() to fail — e.g.:
   • Start the node before the remote-signer service has finished initializing, or
   • Briefly block/reset the connection (e.g. iptables -j REJECT on the signer port for ~1 request), then let it become reachable again moments later.
3. Observe the node logs.

Expected Behavior

Given sign_message()'s existing retry precedent, get_public_key() should retry the same way — a transient failure that resolves within the retry window should not prevent node startup.

Actual Behavior

error: Failed to get public key from remote signer: ...
error: Node failed to start
error: Manual intervention required! Waiting for termination signal (SIGTERM)...

The node hangs indefinitely, even though the remote signer becomes reachable moments later.

Impact

For a validator running with a remote signer (the intended production HSM/KMS-backed setup), any transient hiccup between the validator and the remote signer at the exact moment of (re)startup — both restarted together during a deploy, remote signer still initializing, a brief network partition, a load-balancer health-check gap — causes the validator to fail to (re)join consensus and sit idle until an operator manually restarts it.

This is a liveness/availability gap, not a fund-loss or consensus-safety issue. It was evaluated as not meeting the bar for a private HackerOne report and is filed as a public reliability bug per SECURITY.md.

Proposed Fix

Extract a generic fetch_with_retry() helper and have both get_public_key() and sign_message() call it, so get_public_key() gets the same RetryConfig-driven exponential backoff. A fix is included in the accompanying PR.

[osr21]

Verified this against main at 90d71dc — every claim in the report is accurate:

• sign_message() → sign_message_with_retry() wraps the RPC in backon's RetryableWithContext with config.retry_config (client.rs:187), surfacing RetryExhausted { retries } after the budget is spent — and there's an existing test (retry_behavior, client.rs:371) pinning that behavior.
• get_public_key() (client.rs:127-130) calls get_public_key_internal() exactly once; the first Status error propagates immediately. No retry at this layer.
• RemoteSigningProvider::public_key() (provider.rs:73-108) is cache-read → write-lock → double-check → single client.get_public_key().await? — no retry added there either.
• The failure lands in the Err arm of start() in run() (node.rs:769-781), which logs "Manual intervention required!" and parks in wait_for_termination(). The process stays alive with a non-functional node, so systemd Restart=on-failure / k8s process-exit liveness never fires. The asymmetry (steady-state signing resilient, startup brittle) is real, and startup is arguably where transient failures are most likely — deploys restart the validator and signer together.

Two implementation notes for the accompanying PR, since the proposed fetch_with_retry() helper has a subtlety worth getting right:

1. Don't retry permanent errors. get_public_key_internal can fail two ways: transient transport/Status errors, and InvalidResponse (wrong key length — client.rs:155-161). A generic retry wrapper must only retry the transient class; retrying a malformed-response error burns the whole backoff window (default up to ~3 retries × 5s cap) to reach the same hard failure, and worse, it muddies the log signal — "signer is briefly unreachable" and "signer is returning garbage" need different operator responses. Same distinction exists on the signing path, so if the helper is shared, an is_retryable() predicate on RemoteSigningError (or matching on Status vs. the rest inside the .when() clause backon provides) is the clean shape.

2. The cache makes this a startup-only retry. provider.rs caches the key after first success, so the retry only ever executes on the cold path in consensus_identity() — meaning there's no steady-state latency or load concern with being generous here. If anything, the startup call could justify a larger retry budget than signing (a signer still initializing behind a load balancer can take longer than 3×backoff), which argues for the helper taking the RetryConfig as a parameter rather than hard-sharing one budget.

Also worth saying: the report itself is a model of the right process — correctly self-triaged as a public liveness bug rather than a HackerOne report (no fund-loss/safety impact per SECURITY.md), with exact file:line references that all check out. Happy to review the PR when it's linked.

[pplmaverick]

PR with the fix (updated per your feedback): #269

[pplmaverick]

Also wanted to flag a related PR in arc-prediction-markets: circlefin/arc-prediction-markets#4 adds a settlement timeout and emergency refund path for stuck directional positions — a real failure mode if no proposer ever calls proposePrice() before mainnet.

A community reviewer (@Cassxbt) confirmed the race condition and provided a PoC. I've outlined three fix options in the thread and am ready to implement + add regression tests on maintainer signal.

Could you help route this to the right reviewer before September 16?

[osr21]

Quick note for anyone reading this thread later: the signer-startup retry gap reported here is fixed by #269 (approved), so #268 itself is resolved pending merge.

The arc-prediction-markets#4 routing request above is unrelated to this arc-node issue — I've left a full technical assessment directly on that PR so the discussion stays in one place. For visibility/reviewer routing, the PR thread itself is the right venue (the maintainer is already tagged there); piggybacking on an unrelated closed-loop issue tends to fragment triage rather than help it. I'm a community reviewer, not Circle staff, so I can't actually route anything internally — but keeping that PR merge-ready (Cassxbt's confirmed race fix + regression tests) is the strongest signal a triaging maintainer can act on.