fix(llama): grow the detokenize buffer when a token piece exceeds 32 bytes

[Anai-Guo]

Fixes #2362.

The bug

LlamaModel.detokenize() gives llama_token_to_piece a fixed 32-byte buffer and ignores the "did not fit" signal:

size = 32
buffer = (ctypes.c_char * size)()
for token in tokens:
    n = llama_cpp.llama_token_to_piece(
        self.vocab, llama_cpp.llama_token(token), buffer, size, 0, special
    )
    assert n <= size
    output += bytes(buffer[:n])

llama_token_to_piece returns negative -(required size) when the buffer is too small, and writes nothing (vendor/llama.cpp/src/llama-vocab.cpp:3596-3597):

if (length < (int32_t)size) {
    return -(int32_t) size;
}
memcpy(buf, token, size);
return (int32_t) size;

So for a piece of 64 bytes, n == -64. assert n <= size still passes (-64 <= 32), and bytes(buffer[:n]) becomes bytes(buffer[:-64]) on a 32-byte buffer — which is b''. The token is dropped, silently.

That is exactly the reported symptom: a run of 64 spaces tokenizes fine, and the whitespace token detokenizes to b'', so detokenize(tokenize(x)) != x. Long whitespace runs are the common trigger, but any piece over 32 bytes is affected, and detokenize() is on the generation path (llama.py calls it ~20 times while streaming completions), so this can corrupt model output, not just round-trips.

The fix

Retry with a big-enough buffer, keeping it for the remaining tokens. This is the same convention tokenize() one method up already handles for llama_tokenize:

if n_tokens < 0:
    n_tokens = abs(n_tokens)
    tokens = (llama_cpp.llama_token * n_tokens)()
    n_tokens = llama_cpp.llama_tokenize(...)

and what llama.cpp itself does in llama_vocab::impl::token_to_piece_for_cache (llama-vocab.cpp:3328-3334):

const int n_chars = vocab.token_to_piece(token, &piece[0], piece.size(), 0, special);
if (n_chars < 0) {
    piece.resize(-n_chars);
    int check = vocab.token_to_piece(token, &piece[0], piece.size(), 0, special);
    GGML_ASSERT(check == -n_chars);
}

Pieces of 32 bytes or fewer take exactly the same code path as before — one call, same buffer.

Verification

Building llama.cpp was not needed: both detokenize() bodies (upstream main and this branch) are sliced verbatim out of llama_cpp/_internals.py by AST line range and run against a Python transcription of llama_token_to_piece following the C contract quoted above, with real ctypes buffers so the buffer[:n] slicing behaviour is the real one. The vocabulary is one token per whitespace-run length plus " X", mirroring the [15270, 1599] pair in the issue.

           input            tokens       upstream main             patched
------------------------------------------------------------------------------
            1 sp      [1000, 1599]                  OK                  OK
            2 sp      [1001, 1599]                  OK                  OK
            4 sp      [1002, 1599]                  OK                  OK
            8 sp      [1003, 1599]                  OK                  OK
           16 sp      [1004, 1599]                  OK                  OK
           32 sp      [1005, 1599]                  OK                  OK
           40 sp      [1006, 1599]   FAIL (lost 40 sp)                  OK
           64 sp      [1007, 1599]   FAIL (lost 64 sp)                  OK
          128 sp      [1008, 1599]  FAIL (lost 128 sp)                  OK
------------------------------------------------------------------------------
round-trips preserved: upstream 6/9, patched 9/9

single token 1007 (piece = 64 spaces):
  upstream main -> b''
  patched       -> b'                    '... (64 bytes)

The 1/2/4/8/16/32-OK, 40/64/128-FAIL boundary and the b'' for the oversized token match the issue report exactly. 32 is the last size that fits, because the C check is length < size.

ruff (>=0.15.7, as lint.yaml installs it, run against the repo pyproject.toml): check passes, format --check reports the file already formatted.

Note on tests

I did not add a regression test: tests/test_llama.py runs against ./vendor/llama.cpp/models/ggml-vocab-llama-spm.gguf, and I could not confirm that vocabulary contains a token whose piece exceeds 32 bytes — a test that silently passes on both branches would be worse than none. Happy to add one if you can point me at a test vocab that has such a token (or if you would prefer a llama_token_to_piece-level unit test along the lines of the harness above).

---

🤖 Generated with Claude Code