Fix CK_ULONG length truncation in C_GenerateRandom and C_SeedRandom

[mingulov]

Very briefly, it is a fix for the current behaviour on 64-bit systems:

Demo:

  CK_ULONG    size = 0x100000008UL;                 /* 4 GiB + 8 bytes */
  CK_BYTE_PTR buf  = (CK_BYTE_PTR)calloc(size, 1);   /* zero-filled */
  CK_RV       rv   = C_GenerateRandom(s, buf, size);
  /* rv == CKR_OK, yet only buf[0..7] are randomised; buf[8..] stays 0 (4 GiB of zeros) */

Description:

C_GenerateRandom() and C_SeedRandom() cast their CK_ULONG length argument straight to int before calling WP11_Slot_GenerateRandom() / WP11_Slot_SeedRandom(). On LP64 platforms a length above INT_MAX is silently truncated, so e.g. C_GenerateRandom(s, buf, 0x100000008) produced only 8 bytes yet returned CKR_OK, leaving the remainder of the caller's buffer untouched. A caller that trusts the success code then uses uninitialised / predictable memory as key or nonce material. C_GenerateRandom takes no output-length out-parameter, so per PKCS#11 a CKR_OK return must mean the whole buffer was filled.

C_GenerateRandom now generates the buffer in RNG_MAX_BLOCK_LEN-sized chunks using CK_ULONG arithmetic, filling the entire request regardless of size. This also fixes a pre-existing limitation: wc_RNG_GenerateBlock() rejects any single request larger than RNG_MAX_BLOCK_LEN (64 KiB) with BAD_FUNC_ARG, so a single C_GenerateRandom of more than 64 KiB previously failed with CKR_FUNCTION_FAILED.

C_SeedRandom now rejects a seed length that does not fit in the int taken by the lower layer with CKR_ARGUMENTS_BAD (a return value already listed for this function) rather than truncating it; the seed is consumed as a one-shot RNG nonce and cannot be accumulated across calls.

tests/pkcs11test.c: test_random now verifies that an oversized seed length is rejected and that a multi-chunk (>64 KiB) generate request fills the whole buffer.

[wolfSSL-Bot]

Can one of the admins verify this patch?

[dgarske]

Hi @mingulov , thank you for this code contribution. I have asked @LinuxJedi to look it over. Can you tell us more about your project and use of wolfPKCS11? In order to accept this code we need to have a signed contributor agreement. Please email support at wolfssl dot com and reference this pull request.
Thanks, David Garske, wolfSSL

[Copilot]

Pull request overview

Fixes incorrect handling of large CK_ULONG lengths passed to PKCS#11 RNG APIs on platforms where CK_ULONG is wider than int, preventing silent truncation and ensuring CKR_OK implies the full requested buffer was processed.

Changes:

• C_GenerateRandom() now generates output in bounded chunks (capped at RNG_MAX_BLOCK_LEN) using CK_ULONG arithmetic to fill the entire caller buffer.
• C_SeedRandom() now rejects seed lengths that don’t fit in the lower-layer int length parameter (returns CKR_ARGUMENTS_BAD instead of truncating).
• tests/pkcs11test.c:test_random adds regression coverage for oversized seed-length rejection and for multi-chunk random generation filling the tail of a large buffer.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/crypto.c	Prevents CK_ULONG→int truncation by chunking random generation and validating seed length against INT_MAX.
tests/pkcs11test.c	Adds regression tests covering oversized seeding and large (>64 KiB) generation filling the full buffer.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mingulov]

Hi @dgarske,
Thanks for reviewing! I just sent an email to support regarding the contributor agreement.

To answer your question, I use wolfPKCS11 as a reliable test target for pkcs11-check (https://github.com/mingulov/pkcs11-check/), a framework for validating PKCS#11 implementations.

[LinuxJedi]

@dgarske I approve of this once CLA is in place. Actually fixes F-3412 before I could get to it, so I've linked it.

[dgarske]

Hi @mingulov I see your request and we are tracking in zendesk 22014.

[kareem-wolfssl]

OK to review - contributor agreement approved

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

[LinuxJedi]

Many thanks for the contribution. I'll make sure you are credited in the upcoming release.