Skip to content

Fix request_signer body serialization and wildcard double-encoding - #336

Open
xluo-aws wants to merge 1 commit into
opensearch-project:mainfrom
xluo-aws:fix/signer-and-wildcard-encoding
Open

Fix request_signer body serialization and wildcard double-encoding#336
xluo-aws wants to merge 1 commit into
opensearch-project:mainfrom
xluo-aws:fix/signer-and-wildcard-encoding

Conversation

@xluo-aws

Copy link
Copy Markdown
Member

Description

Fixes two bugs that should be resolved before the 4.0.0 release.

Fix 1 — request_signer body serialization mismatch (#311)

The `request_signer` feature (added in 4.0 beta via #281) was passing the raw Ruby `body` object (a Hash) to `sign_request`. However, the Faraday transport then serialized the body separately via `__convert_to_json`. When these two serializations produced different byte strings (different key order, custom serializer, etc.), the resulting signature didn't match the wire body, causing 403 errors on any signed request with a body.

Fix: serialize the body through the transport's own serializer before passing it to `sign_request`, mirroring exactly what the transport sends on the wire. String bodies are passed through unchanged.

Fix 2 — wildcard/special character double-encoding in index params (#319)

`normalize_value` used `CGI.escape` which encodes `*` → `%2A`. The HTTP layer then percent-encodes the `%` → `%25`, so OpenSearch received `test-%2A` literally instead of treating it as a wildcard, causing `index_not_found_exception` errors.

Fix: switch to `URI::DEFAULT_PARSER.escape` with a safe-character set that preserves `*` and other characters meaningful in OpenSearch path patterns. Spaces are encoded as `%20` (correct for URL paths) rather than `+`.

Issues Resolved

Fixes #311
Fixes #319

Testing

  • Updated `request_signer_spec.rb`: existing test updated to reflect correct serialized-string behaviour; 3 new cases covering Hash body, nil body, and pre-serialized String body.
  • Updated `utils_spec.rb`: existing expectation updated for new `%20` encoding; 6 new `#normalize_value` cases covering wildcard preservation, comma-separated patterns, special chars, and arrays.
  • Full unit suite: 22 RSpec examples + 116 minitest runs + 107 transport tests — 0 failures.

Check List

  • New functionality includes testing.
  • New functionality has been documented (CHANGELOG).
  • Commits are signed per the DCO using `--signoff`.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Fix two bugs before 4.0.0 release:

1. request_signer body mismatch (opensearch-project#311)
   The signer was receiving the raw body object (Hash/nil) while the
   Faraday transport serialized it separately via __convert_to_json.
   This meant the signer computed a signature over different bytes than
   what was actually sent, producing 403 errors on signed requests with
   a body. Fix: serialize the body through the transport serializer
   before passing it to sign_request, mirroring what the transport does.
   String bodies are passed through unchanged.

2. Wildcard double-encoding in index params (opensearch-project#319)
   normalize_value used CGI.escape which encodes '*' -> '%2A'. The HTTP
   layer then percent-encodes '%' -> '%25', so OpenSearch received
   'test-%2A' literally instead of the intended wildcard. Fix: switch to
   URI::DEFAULT_PARSER.escape with a safe set that preserves '*' and
   other characters meaningful in OpenSearch path patterns. Spaces are
   now encoded as '%20' rather than '+', which is correct for URL paths.

Both fixes include updated/new tests.

Signed-off-by: Xuesong Luo <lxuesong@amazon.com>
@xluo-aws
xluo-aws force-pushed the fix/signer-and-wildcard-encoding branch from 2aaa1e6 to 4866a6a Compare July 25, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Asynchronous searching does not work with wildcard indexes. (double escape) [BUG] request_signer inconsistent body serialization

1 participant