Repository navigation
Conversation
HuggingFaceInferenceAPI._convertArgs forwarded max_new_tokens, temperature,
top_k and top_p but dropped options.stop. Its sibling HuggingFaceTGI speaks the
same TGI protocol, posts the same { inputs, parameters } payload and does
forward stop, capped by maxStopWords.
Autocomplete templates always set stop, and stopAtStopTokens only trims the
visible text inside the autocomplete pipeline, so chat and edit lost the
sequences entirely while autocomplete kept generating up to max_new_tokens
instead of stopping.
Apply the same capped forwarding the sibling already uses. Add a provider test
in the shape the repository already uses for non-OpenAI providers.
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
HuggingFaceInferenceAPI._convertArgsforwardsmax_new_tokenstemperaturetop_kandtop_pbut dropsoptions.stopIts sibling
HuggingFaceTGIspeaks the same TGI protocol : both POST{ inputs, parameters }and both readvalue.token.textbackThat one forwards
stop, capped bymaxStopWords, and the two providers sit side by side inPARALLEL_PROVIDERSThis applies the sibling's exact line to the Inference API provider
Why it is not caught today : autocomplete templates always set
stop, andstopAtStopTokensis only wired into the autocomplete pipeline, so autocomplete still shows the right text while the server keeps generating up tomax_new_tokensinstead of stoppingChat and edit have no such filter and lose the sequences outright
Checklist
maxStopWordsandstopare both already documentedScreen recording or screenshot
Not applicable, no UI change
Tests
core/llm/llms/HuggingFaceInferenceAPI.vitest.ts, built on the samerunLlmTestshape asCohere.vitest.tsAnthropic.vitest.tsandOpenAI.vitest.tsThose three carry the same local copy of the helper, since this provider is not OpenAI-compatible and cannot use
createOpenAISubclassTeststhe wayGroqandFireworksdoAt 230 lines it is the smallest of the four
The calls mirror
CompletionStreamer.ts:36-39, which is how autocomplete reaches this provider,raw: trueincludedThree cases : sequences are forwarded, they are capped at
maxStopWords, and nothing is sent when none are givenBoth failures are
Anthropic > streamChatandGroq > streamChattiming out at 5000msI reproduced
Anthropic > streamChaton a tree with both of my files removed, so it predates this change on my machineGroqpasses on its own at 4780ms and only tips over under the full suite load