Skip to content

fix(huggingface): pass stop sequences to the Inference API provider - #13349

Open
Ayoubhm07 wants to merge 1 commit into
continuedev:mainfrom
Ayoubhm07:fix/hf-inference-api-stop
Open

Ayoubhm07 wants to merge 1 commit into
continuedev:mainfrom
Ayoubhm07:fix/hf-inference-api-stop

Conversation

@Ayoubhm07

Copy link
Copy Markdown

Description

HuggingFaceInferenceAPI._convertArgs forwards max_new_tokens temperature top_k and top_p but drops options.stop

Its sibling HuggingFaceTGI speaks the same TGI protocol : both POST { inputs, parameters } and both read value.token.text back
That one forwards stop, capped by maxStopWords, and the two providers sit side by side in PARALLEL_PROVIDERS

This applies the sibling's exact line to the Inference API provider

Why it is not caught today : autocomplete templates always set stop, and stopAtStopTokens is only wired into the autocomplete pipeline, so autocomplete still shows the right text while the server keeps generating up to max_new_tokens instead of stopping
Chat and edit have no such filter and lose the sequences outright

Checklist

  • I've read the contributing guide
  • The relevant docs, if any, have been updated or created
    • no user facing option changes, maxStopWords and stop are both already documented
  • The relevant tests, if any, have been updated or created

Screen recording or screenshot

Not applicable, no UI change

Tests

core/llm/llms/HuggingFaceInferenceAPI.vitest.ts, built on the same runLlmTest shape as Cohere.vitest.ts Anthropic.vitest.ts and OpenAI.vitest.ts
Those three carry the same local copy of the helper, since this provider is not OpenAI-compatible and cannot use createOpenAISubclassTests the way Groq and Fireworks do
At 230 lines it is the smallest of the four

The calls mirror CompletionStreamer.ts:36-39, which is how autocomplete reaches this provider, raw: true included

Three cases : sequences are forwarded, they are capped at maxStopWords, and nothing is sent when none are given

with the one line removed : 2 failed, 1 passed
                            the two failures are the missing "stop" key
with the one line present : 3 passed

npx vitest run llm/           466 passed, 2 failed

Both failures are Anthropic > streamChat and Groq > streamChat timing out at 5000ms
I reproduced Anthropic > streamChat on a tree with both of my files removed, so it predates this change on my machine
Groq passes on its own at 4780ms and only tips over under the full suite load

HuggingFaceInferenceAPI._convertArgs forwarded max_new_tokens, temperature,
top_k and top_p but dropped options.stop. Its sibling HuggingFaceTGI speaks the
same TGI protocol, posts the same { inputs, parameters } payload and does
forward stop, capped by maxStopWords.

Autocomplete templates always set stop, and stopAtStopTokens only trims the
visible text inside the autocomplete pipeline, so chat and edit lost the
sequences entirely while autocomplete kept generating up to max_new_tokens
instead of stopping.

Apply the same capped forwarding the sibling already uses. Add a provider test
in the shape the repository already uses for non-OpenAI providers.
@Ayoubhm07
Ayoubhm07 requested a review from a team as a code owner October 5, 2026 23:55
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@Ayoubhm07

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant