[mudler-agent]

Summary

Adds offline transcription for nvidia/nemotron-labs-unified-asr-0.6b: a prompt-conditioned RNNT model with NeMo's chunked_limited_with_rc attention and a 4095-piece vocabulary. Offline inference uses full attention. Buffered streaming is not part of this PR and will follow separately.

What changed

• The converter writes a parakeet.unified.* block for the chunked_limited_with_rc style and no streaming keys. Other attention styles are unchanged.
• The loader reads the block and checks the presets ([L,C,R] triples, C >= 1, default index in range). streaming.present stays false.
• stream_begin and parakeet-cli transcribe --stream refuse unified models with a clear error. Before this change the model was routed into the cache-aware streaming path.
• StreamingEncoder and StreamingMel checked their preconditions with assert, which Release builds compile out. These are now real checks that throw.
• New key parakeet.unified.valid_length (default off). When it is set, the mel frontend keeps only floor(samples / hop) frames, the count NeMo passes as the encoder length. The converter writes it only for unified checkpoints, so every other GGUF takes the same code path as before. The CPU and GPU frontends both apply it.
• parakeet-cli info prints the unified fields.
• Tests: test_unified_offline (mel, encoder, prompt kernel, RNNT token ids, text, and the C-API refusal) and test_unified_load_negative (malformed presets, and the gate defaulting off). The NeMo prompt baseline now also stores the mel.

Verification

Reference: NeMo main cd5e4fbd36ba098d0c78b4552f6f9592ad34de93, CPU, float32, batch 1. The reference decodes the prompt-conditioned encoder output with rnnt_decoder_predictions_tensor. Stages use the production frontend and encoder, f32 weights.

Stage	English clip (7.4 s)	German clip (18 s, public FLEURS utterance)
Encoder max abs diff	2.4e-7	6.0e-7
Prompt kernel max abs diff	7.2e-7	9.5e-7
RNNT token ids	exact	exact
Mel max abs diff	2.0e-4	3.3e-5

WER against the NeMo reference text, offline, for en and de: 0.0000 for f32, f16 and q8_0. The auto prompt and other languages were not measured.

Existing checkpoints convert to the same bytes as before. The sha256 matched before and after this change for nvidia/parakeet-tdt_ctc-110m (f32) and nvidia/parakeet_realtime_eou_120m-v1 (f16). The model-dependent suite for those two checkpoints ran before the valid_length change and matched the pre-change result: 41 passed, 0 failed, the rest skipped for missing fixtures. The valid_length path is off for them.

Commands:

cmake -B build -DPARAKEET_BUILD_TESTS=ON -DGGML_NATIVE=ON && cmake --build build -j
ctest --test-dir build -LE model
scripts/gen_nemo_baseline.py --model nvidia/nemotron-labs-unified-asr-0.6b \
    --audio tests/fixtures/speech.wav --lang en --output unified-en.gguf
PARAKEET_TEST_GGUF_UNIFIED=unified-f32.gguf PARAKEET_TEST_BASELINE_UNIFIED=unified-en.gguf \
    ctest --test-dir build -R test_unified

Model-free suite: 54 tests, all pass (8 skip for missing fixtures).

Known issues

• Mel: the English clip differs by up to 2e-4 (mean 5e-7), in about 20 of 743 frames, mostly in the highest mel bins where filterbank power is small. The frontend computes in double precision and the NeMo reference in float32, so reference rounding is the likely cause. This code is shared with every model, so it was not changed here.
• Streaming is not implemented. The stream_* entry points refuse unified models until the buffered driver lands.
• The f16 and q8_0 GGUFs used for WER were built from the f32 file. The valid_length key was added with a metadata-copy step, and q8_0 with parakeet-cli quantize. Converting a checkpoint with this branch writes the same key.
• Not measured: auto and the other prompt languages. The scene stream runs the unified model offline per segment; it was not tested.

🤖 Generated with Claude Code