Lock globalRNGMutex around all shared globalRNG access

[yosuke-wolfssl]

Description

The OpenSSL-compat layer shares a single static WC_RNG globalRNG whose
Hash-DRBG state is not internally thread-safe. The canonical access pattern
(wolfSSL_RAND_bytes) holds globalRNGMutex for the entire duration of any
operation on globalRNG. Several sibling functions mutated the same
globalRNG without taking the lock, so concurrent callers — or one
running alongside a lock-holding RAND_bytes/RAND_add caller — could race
on the shared DRBG V/C/reseed-counter and seed state, corrupting the DRBG and
the randomness it yields to compat consumers (IVs, nonces, keys).

This PR extends the existing globalRNGMutex discipline to every remaining
unprotected access.
Addressed by f_6555 and f_6556.

Changes

src/ssl.c

• wolfSSL_RAND_write_file: acquire globalRNGMutex around
  wc_RNG_GenerateBlock(&globalRNG, ...), unlocking before the file I/O to
  keep the critical section minimal.
• wolfSSL_RAND_egd: lock around the wc_RNG_DRBG_Reseed(&globalRNG, ...)
  of the collected EGD entropy.
• wolfSSL_RAND_poll: hold the lock from before wc_GenerateSeed(&globalRNG.seed, ...)
  (previously an unlocked write to globalRNG.seed) through
  wc_RNG_DRBG_Reseed, unlocking on every path (including the
  HAVE_INTEL_RDRAND and no-HASHDRBG branches). Added a comment noting the
  lock intentionally covers wc_GenerateSeed so the scope is not narrowed
  back later.

src/ssl_p7p12.c

• wolfSSL_SMIME_write_PKCS7: lock around the wc_RNG_GenerateBlock(&globalRNG, ...)
  that generates the detached S/MIME MIME boundary, following the function's
  existing ret > 0 fallthrough style.

Notes

• wolfSSL_RAND_poll is called by wolfSSL_RAND_bytes only after that
  function releases globalRNGMutex (and it re-acquires it afterward), so
  having RAND_poll take the lock itself does not self-deadlock.
• globalRNGMutex is in scope for ssl_p7p12.c because it is #included
  into ssl.c after the mutex's static definition, within the same
  OPENSSL_EXTRA/HAVE_GLOBAL_RNG context.

Testing

• ./configure --enable-all built under -fsanitize=address,undefined -fno-omit-frame-pointer; full ./tests/unit.test passes (exit 0),
  including test_wolfSSL_RAND_poll, the ossl_rand group, and
  test_wolfSSL_SMIME_write_PKCS7.

[Copilot]

Pull request overview

This PR strengthens thread safety in the OpenSSL-compat RNG path by extending the existing globalRNGMutex locking discipline to all remaining operations that touch the shared static WC_RNG globalRNG, preventing races in DRBG/seed state when called concurrently.

Changes:

• Add globalRNGMutex locking around globalRNG operations in wolfSSL_RAND_write_file, wolfSSL_RAND_egd, and wolfSSL_RAND_poll.
• Add globalRNGMutex locking around detached S/MIME boundary generation in wolfSSL_SMIME_write_PKCS7.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/ssl.c	Adds missing globalRNGMutex locking around reseed/seed/generate operations on the shared globalRNG.
src/ssl_p7p12.c	Locks globalRNGMutex while generating the detached S/MIME boundary using globalRNG.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10824

Scan targets checked: wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅