[AZL3] EVP_PKEY_Q_keygen rejects provider-backed ML-KEM

[qmuntal]

Description

Azure Linux 3 exposes ML-KEM through symcryptprovider, but OpenSSL's EVP_PKEY_Q_keygen rejects the algorithm before provider dispatch.

In the same process:

• EVP_KEYMGMT_fetch("ML-KEM-768") succeeds.
• EVP_PKEY_Q_keygen("ML-KEM-768") fails.
• Generic name-based key generation succeeds.

Reproducer

docker run --rm -it \
  mcr.microsoft.com/oss/go/microsoft/golang:1.27-azurelinux3.0 sh
tdnf install -y gcc openssl-devel

Create /tmp/repro.c:

#include <openssl/err.h>
#include <openssl/evp.h>
#include <stdio.h>

int main(void) {
    EVP_KEYMGMT *keymgmt = EVP_KEYMGMT_fetch(NULL, "ML-KEM-768", NULL);
    printf("EVP_KEYMGMT_fetch: %s\n", keymgmt != NULL ? "ok" : "failed");
    EVP_KEYMGMT_free(keymgmt);

    EVP_PKEY *quick = EVP_PKEY_Q_keygen(NULL, NULL, "ML-KEM-768");
    int quick_ok = quick != NULL;
    printf("EVP_PKEY_Q_keygen: %s\n", quick_ok ? "ok" : "failed");
    if (!quick_ok)
        ERR_print_errors_fp(stderr);
    EVP_PKEY_free(quick);

    EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new_from_name(NULL, "ML-KEM-768", NULL);
    EVP_PKEY *generic = NULL;
    int generic_ok =
        ctx != NULL &&
        EVP_PKEY_keygen_init(ctx) > 0 &&
        EVP_PKEY_keygen(ctx, &generic) > 0;

    printf("name-based keygen: %s\n", generic_ok ? "ok" : "failed");

    EVP_PKEY_free(generic);
    EVP_PKEY_CTX_free(ctx);

    return quick_ok ? 0 : 1;
}

Compile and run:

cc -Wall -Wextra /tmp/repro.c -o /tmp/repro -lcrypto
/tmp/repro

Actual result

EVP_KEYMGMT_fetch: ok
EVP_PKEY_Q_keygen: failed
error:03080106:digital envelope routines:EVP_PKEY_Q_keygen:passed invalid argument:crypto/evp/evp_lib.c:1229:
name-based keygen: ok

The program exits with status 1.

Expected result

EVP_PKEY_Q_keygen should generate a key for an algorithm available through the configured provider, consistent with the generic name-based EVP path.

Environment

• Azure Linux 3.0.20260809
• ARM64
• openssl-3.3.7-4.azl3
• openssl-libs-3.3.7-4.azl3
• SymCrypt-103.8.0-1.azl3
• SymCrypt-OpenSSL-1.10.0-1.azl3

Root cause

The OpenSSL 3.3.7 implementation of EVP_PKEY_Q_keygen only recognizes a fixed set of algorithm names. Although ML-KEM-768 is available from symcryptprovider, the helper rejects it with EVP_R_PASSED_INVALID_ARGUMENT before provider key generation is attempted.

The generic provider path through EVP_PKEY_CTX_new_from_name does not have this limitation and succeeds.

Suggested resolution

Backport the relevant OpenSSL support for ML-KEM in EVP_PKEY_Q_keygen.

Related to #17967.

[eric-desrochers]

Thanks. I created an internal ADO ticket to assign to our dev team.
Reference: #63694970

[liram-vardi]

We hit this exact bug independently and wanted to add some corroborating data points that may help with prioritization/verification.

Confirms cross-application impact: we saw the identical EVP_PKEY_Q_keygen:passed invalid argument:crypto/evp/evp_lib.c:1229 failure in two unrelated Kubernetes controller-runtime operators (different codebases, same base image), both built with mcr.microsoft.com/oss/go/microsoft/golang:1.27.0 (CGO_ENABLED=1) on top of mcr.microsoft.com/azurelinux/base/core:3.0.20260820. Both crash-loop at startup because the very first outbound TLS call (to the Kubernetes API server) triggers ML-KEM hybrid key-share generation during the TLS 1.3 handshake.

Operator 1 (controller setup):

unable to create controller {"controller": "node-storage-capacity-controller",
"error": "failed to get API group resources: unable to retrieve the complete
list of server APIs: v1: Get \"https://10.100.0.1:443/api/v1\":
EVP_PKEY_Q_keygen_MLKEM
openssl error(s):
C026B10FED7A0000:error:03080106:digital envelope routines:EVP_PKEY_Q_keygen:passed invalid argument:crypto/evp/evp_lib.c:1229:"}

Operator 2 (client-go leader election, independent codebase):

error retrieving resource lock kuiper-monitoring/...: Get "https://10.100.0.1:443/apis/coordination.k8s.io/...": EVP_PKEY_Q_keygen_MLKEM
error:03080106:digital envelope routines:EVP_PKEY_Q_keygen:passed invalid argument:crypto/evp/evp_lib.c:1229:

Bisection: reverting only the azurelinux/base/core image tag (back to the prior known-good build, 3.0.20260728) while keeping Go at 1.27.0 resolves the crash entirely. We verified via the running binary's embedded build info (go1.27.0-1-microsoft) that Go 1.27.0 was still in use post-fix — confirming the base image/OpenSSL package is the sole variable, not the Go toolchain version.

Blast radius note: since the affected path is the TLS client handshake itself (not a narrow/isolated keygen API call), this breaks any CGO-enabled Microsoft Go binary on this base image that makes outbound HTTPS calls at all — not just ML-KEM-specific code. It fully prevents pod startup for affected workloads, which may warrant higher severity than a typical algorithm-support gap.

Happy to share full repro manifests/logs if useful. We're tracking this via the internal ADO ticket referenced above (#63694970) as well.

[eric-desrochers]

We have discussed it internally, and we will evaluate if a backport of : openssl/openssl@a57c6f8 is feasible. This will enable ML-KEM for any other code out there that uses EVP_PKEY_Q_keygen. Hence preventing issue for instance with Go 1.27.