internal: gate public-key auth on every compiled signing algorithm

[yosuke-wolfssl]

Problem

Public-key user authentication is gated on compile-time algorithm macros in five places in src/internal.c, but the guards are not the same expression. Two of them list only RSA and ECDSA:

• GetAllowedAuth() — the server's default advertised method list
• DoUserAuthFailure() — the client's ID_USERAUTH_PUBLICKEY case

Ed25519 and ML-DSA are fully supported public-key user-auth algorithms: DoUserAuthRequestPublicKey(), PrepareUserAuthRequestPublicKey() and BuildUserAuthRequestPublicKey() all have real branches for them, and all three are guarded on the correct four-way predicate.

So in a build with WOLFSSH_NO_RSA + WOLFSSH_NO_ECDSA and Ed25519 (or ML-DSA) enabled, the verify and sign paths are compiled in but unreachable: the server never advertises publickey, and the client drops it from authType, so a peer list containing only publickey returns WS_USER_AUTH_E. Public-key auth is unusable in those configurations. Closes f-10542.

Fix (src/internal.c, wolfssh/internal.h)

wolfssh/internal.h derives a shared predicate next to the existing WOLFSSH_NO_RSA / WOLFSSH_NO_ECDSA aggregates:

#if defined(WOLFSSH_NO_RSA) && defined(WOLFSSH_NO_ECDSA) && \
    defined(WOLFSSH_NO_ED25519) && defined(WOLFSSH_NO_MLDSA)
    #undef WOLFSSH_NO_PUBKEY_AUTH
    #define WOLFSSH_NO_PUBKEY_AUTH
#endif

All five sites now use #ifndef WOLFSSH_NO_PUBKEY_AUTH, so the predicate has one definition.

Not a pure rename at two sites — these are the behavioral fixes:

Site	Old guard
GetAllowedAuth()	!NO_RSA || !NO_ECDSA
DoUserAuthFailure()	!NO_RSA || !NO_ECDSA || WOLFSSH_TPM

The WOLFSSH_TPM term is dropped. The TPM user-auth signer exists only for RSA, and with all four algorithm families disabled the "You need at least one signing algorithm" #error fires, so no buildable config relied on it.

Verification

• --enable-all: 11 passed, 1 skipped (external.test), 0 failed.
• gcc-13 -Werror preflight sweep: 6 configurations clean.
• Ed25519-only build (-DWOLFSSH_NO_RSA -DWOLFSSH_NO_ECDSA against wolfSSL with --enable-ed25519-stream): src/internal.c compiles clean, and preprocessing confirms both typeAllowed |= WOLFSSH_USERAUTH_PUBLICKEY and the ID_USERAUTH_PUBLICKEY case now survive. Negative control: both disappear with the change reverted.

Not in this PR

No test is added: the Ed25519-only configuration cannot run make check today because examples/client/common.c and examples/echoserver/echoserver.c fail to compile there — they guard the ECDSA sample keys on WOLFSSH_NO_ECC (curve support) while selecting the key material on WOLFSSH_NO_ECDSA_SHA2_NISTP* (signing). That is pre-existing and tracked separately; fixing it is the prerequisite for adding an Ed25519-only CI job.

[Copilot]

Pull request overview

This pull request fixes a compile-time gating inconsistency for SSH publickey user authentication by introducing a single shared “no public-key auth available” predicate and using it consistently across the internal user-auth flow. This ensures publickey auth remains reachable/advertised when any supported signing algorithm family (RSA/ECDSA/Ed25519/ML-DSA) is compiled in, including Ed25519-only/ML-DSA-only builds.

Changes:

• Add a derived WOLFSSH_NO_PUBKEY_AUTH macro in wolfssh/internal.h when all compiled signing algorithm families are disabled.
• Replace multiple per-site algorithm guard expressions in src/internal.c with #ifndef WOLFSSH_NO_PUBKEY_AUTH at the five affected sites.
• Remove the prior WOLFSSH_TPM exception from the client-side ID_USERAUTH_PUBLICKEY enablement path, aligning behavior with actual available signing algorithms.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
wolfssh/internal.h	Introduces a shared derived macro (WOLFSSH_NO_PUBKEY_AUTH) to centralize the “no publickey auth possible” compile-time predicate.
src/internal.c	Switches all five publickey auth gating sites to use #ifndef WOLFSSH_NO_PUBKEY_AUTH for consistent reachability/advertisement across configurations.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1183

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.