feat(api): add /v1/detokenize endpoint#9620
Open
Dennisadira wants to merge 1 commit intomudler:masterfrom
Open
feat(api): add /v1/detokenize endpoint#9620Dennisadira wants to merge 1 commit intomudler:masterfrom
Dennisadira wants to merge 1 commit intomudler:masterfrom
Conversation
Closes mudler#1649. Mirror of the existing /v1/tokenize path, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing. - Add Detokenize gRPC RPC with DetokenizeRequest{tokens} / DetokenizeResponse{content} messages. - Implement in the llama.cpp backend using common_token_to_piece, the same primitive TokenizeString already uses internally. - Other backends inherit the default Unimplemented from base.Base, in line with how Detect, Rerank, etc. are gated per-backend. - Wire up the Go gRPC interface, server, client, and in-process embed wrapper alongside their TokenizeString counterparts. - Add the schema types, ModelDetokenize wrapper, HTTP handler, route registration, RouteFeatureRegistry entry (gated by FeatureTokenize so no new feature flag is needed), and the discovery map entry under ai_functions. - Regenerated swagger reflects the new endpoint and types. - Update authentication.md to list /v1/detokenize alongside /v1/tokenize. Assisted-by: Claude:claude-opus-4-7
mudler
reviewed
Apr 30, 2026
| return grpc::Status::OK; | ||
| } | ||
|
|
||
| grpc::Status Detokenize(ServerContext* context, const backend::DetokenizeRequest* request, backend::DetokenizeResponse* response) override { |
Owner
There was a problem hiding this comment.
this requires a test addition to our e2e-backend test suite where we exercise a mocked backend via api
mudler
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1649. Mirror of
/v1/tokenizefor the inverse direction: take a list of token IDs and return the detokenized text, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing.The proto/handler shape was discussed in #1649 (comment). @benniekiss reacted positively; landing this with the strict-mirror-of-tokenize precedent in mind. Happy to adjust if the proto naming or response shape should differ.
What's added
backend/backend.proto): newDetokenize(DetokenizeRequest) returns (DetokenizeResponse)RPC, withDetokenizeRequest{repeated int32 tokens}andDetokenizeResponse{string content}. The Go bindings are regenerated bymake protogen-go(gitignored as usual).backend/cpp/llama-cpp/grpc-server.cpp): handler that callscommon_token_to_pieceper token and concatenates — the same primitiveTokenizeStringalready uses internally at the same file.Unimplementedfrompkg/grpc/base.Base— same pattern asDetect,Rerank, etc. Backends can opt in later.pkg/grpc/{interface,server,backend,client,embed}.go+pkg/grpc/base/base.goupdated alongside theirTokenizeStringcounterparts.POST /v1/detokenizeincore/http/endpoints/localai/detokenize.goandcore/http/routes/localai.go. Request{"model": "...", "tokens": [...]}, response{"content": "..."}.RouteFeatureRegistrygated by the existingFeatureTokenize— no new feature flag.ai_functionsin the routes index.authentication.mdupdated to list the new endpoint.Test plan
make protogen-goregenerates cleango build ./core/... ./pkg/grpc/...cleango vet ./core/... ./pkg/grpc/...cleango test -c -o /dev/null ./core/services/nodes/...clean (the existing testcontainers-based suite needs Docker; only updated the two interface mocks so the test package still compiles)make swaggerregenerates with the new endpoint visiblePOST /v1/tokenize→POST /v1/detokenizereturns the original text on a llama.cpp modelAssisted-by: Claude:claude-opus-4-7