Skip to content

fix: truncate oversized reranking passages instead of failing the request - #2532

Draft
erichare wants to merge 1 commit into
mainfrom
fix/nvidia-rerank-truncate-end
Draft

fix: truncate oversized reranking passages instead of failing the request#2532
erichare wants to merge 1 commit into
mainfrom
fix/nvidia-rerank-truncate-end

Conversation

@erichare

@erichare erichare commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

Changes the NVIDIA reranking request's truncate option from NONE to END.

With NONE, a single query+passage pair over the model's 8192-token limit fails the whole batch with HTTP 400 (Input length 16384 exceeds maximum allowed token size 8192), and because rerank batches are dispatched fail-fast, the entire findAndRerank command fails — deterministically, for any query whose candidate set contains one oversized document. No hybridLimits setting avoids it when the oversized document ranks highly.

Observed against a production collection containing large OCR'd table chunks (13–15k chars, digit/punctuation-heavy): ~3% of eval queries fail 100% of the time with RERANKING_PROVIDER_CLIENT_ERROR. With END, the NIM truncates the oversized pair and scores the remaining tokens — a strictly better outcome than failing the whole command.

One-line constant change plus javadoc; no config surface added while findAndRerank is still stabilizing. If per-model configurability is wanted later, truncate can move into reranking-providers-config.yaml as a follow-up.

Validation:

  • NvidiaRerankingProviderTest + RerankingProviderTest pass (4 tests)
  • formatting clean (fmt:format)

Which issue(s) this PR fixes:

N/A — follow-up to the GPU-plane concurrency investigation (same series as #2530).

Checklist

  • Changes manually tested after deployment
  • Automated Tests added/updated (no existing test exercises the wire payload; not adding mock-server scaffolding for a constant)
  • Documentation added/updated (javadoc)
  • CLA Signed: DataStax CLA

@erichare
erichare requested a review from a team as a code owner August 7, 2026 16:32
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Unit Test Coverage Report

Overall Project 52.97% 🍏
File Coverage
NvidiaRerankingProvider.java 17.58% 🍏

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (dse69-it)

Overall Project 71.43% 🍏
File Coverage
NvidiaRerankingProvider.java 12.73% 🍏

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (hcd-it)

Overall Project 72.75% 🍏
File Coverage
NvidiaRerankingProvider.java 12.73% 🍏

@erichare
erichare marked this pull request as draft August 7, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant