agent: size the signature buffer from the identity

[yosuke-wolfssl]

Problem

PostSignRequest() in src/agent.c signed into a fixed byte sig[256]. wc_RsaSSL_Sign() rejects an output buffer smaller than the modulus, so every RSA identity above 2048 bits — including the 3072-bit ssh-keygen -t rsa default — failed each sign request with WS_RSA_E. Deterministic, gated on WOLFSSH_AGENT, and untested. Not a memory-safety defect; the agent simply cannot sign with a mainstream key size.

Adjacent to it, wolfSSH_AGENT_SignRequest() read agent replies into byte rxBuf[512], which cannot hold an RSA-4096 sign response (521 bytes).

Fix (src/agent.c)

• PostSignRequest() sizes the signature buffer from the identity — id->key.rsa.nSz for RSA, the existing ECDSA_ASN_SIG_SZ for ECDSA — allocates it from agent->heap, and frees it on every path. A modulus longer than RSA_MAX_SIZE returns WS_BUFFER_E.
• wolfSSH_AGENT_SignRequest() reads into a heap buffer of WOLFSSH_AGENT_MAX_RSP_SZ (2048, #ifndef-overridable), freed after the last use of agent->msg, which points into it.

Both functions lose their large stack buffers. SignHashRsa() and SignHashEcc() are unmodified, so their pre-existing stack usage is untouched.

Closes f-10541.

Tests (tests/api.c)

Two tests drive real keys through the fake-agent reply path the existing agent tests already use, with key components as hex constants decoded by ConvertHexToBin():

Test	Key	Proof of signing
..._rsa_3072	RSA-3072	clear the stored d, re-sign, expect WS_RSA_E
..._ecc_p521	NIST P-521	corrupt the stored point, re-sign, expect WS_ECC_E

The trailing step matters because a successful sign and a key-blob miss both surface as WS_AGENT_NO_KEY_E; without it a drifted key blob would pass vacuously.

Verification

• gcc-13 -Werror preflight: lint clean, all 6 configs clean.
• make check: 11 passed, 1 skipped, 0 failed.
• Negative controls: clamping the RSA buffer to 256 gives WS_RSA_E; clamping ECC to 136 gives WS_ECC_E; flipping one bit of the sign-request key blob trips the match proof.
• ASan + UBSan on api.test: clean.

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR fixes wolfSSH agent signing failures with larger RSA identities by sizing signature/reply buffers dynamically (or via a larger configurable cap) and adds regression tests for RSA-3072 and ECDSA P-521 agent signing paths.

Changes:

• Allocate the RSA/ECDSA signature buffer in PostSignRequest() based on identity size instead of a fixed 256-byte stack buffer.
• Increase agent reply read capacity in wolfSSH_AGENT_SignRequest() by reading into a heap buffer sized by WOLFSSH_AGENT_MAX_RSP_SZ.
• Add API tests that drive RSA-3072 and ECDSA P-521 identities through the fake-agent path to validate the fix and prevent vacuous “no key” passes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
src/agent.c	Sizes signature buffer from identity and increases agent reply read buffer via heap allocation.
tests/api.c	Adds regression tests for RSA-3072 and ECDSA P-521 agent sign requests and expands test IO buffers.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1179

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.