feat(endpoints): support explicit empty streaming chunks - #42
Merged
cquil11 merged 1 commit intoAug 20, 2026
Merged
Conversation
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@e8ece5d29bd0285af1ced36fbc4c670328b2d58cRecommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@e8ece5d29bd0285af1ced36fbc4c670328b2d58cLast updated for commit: |
Add an opt-in allow_empty_content endpoint setting for streaming OpenAI Chat Completions. Some compatible servers send an explicit empty content or reasoning delta before generated text; retaining that event gives TTFT and the shared response timeline the server-observed boundary without treating it as output text or tokens. The default remains disabled. The parser accepts only typed Chat Completions chunks, preserves zero-token semantics, keeps TTFO tied to non-empty output, and ignores malformed stream events instead of raising. Focused metric and parser coverage verifies the enabled, disabled, empty-only, reasoning, and usage-only cases. The in-repo mock server now supports --emit-empty-chat-content, which emits one leading empty content delta without token accounting. Usage: aiperf-mock-server --emit-empty-chat-content --ttft 100 aiperf profile --streaming --allow-empty-content --url http://127.0.0.1:8000 --model <model> ... This makes the end-to-end timing distinction reproducible: TTFT observes the empty chunk while TTFO waits for the first non-empty output token. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
ajcasagrande
force-pushed
the
ajc/allow-ttft-empty
branch
from
August 15, 2026 05:05
23e6949 to
e8ece5d
Compare
cquil11
approved these changes
Aug 19, 2026
cquil11
marked this pull request as ready for review
August 20, 2026 20:12
cquil11
merged commit Aug 20, 2026
efc0f66
into
SemiAnalysisAI:release/agentx-v1.1
1 of 4 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
--allow-empty-contentfor streaming OpenAI Chat Completions. Some compatible servers send explicit emptycontentor reasoning deltas before generated text. With this opt-in flag, AIPerf retains those events on the shared response timeline so TTFT reflects the first server stream event.The default remains
false, preserving existing behavior for all current benchmarks. The parser accepts only explicit empty strings from valid streaming Chat Completions chunks; usage-only, finish-only, null, and malformed events remain excluded.Usage
CLI:
YAML:
Metric semantics
Validation
pre-commit run --files ...The in-repo mock server gained
--emit-empty-chat-contentsolely to make this end-to-end timing case reproducible.Known test-suite state
uv run pytest tests/unit/ -n autocurrently reports four failures in health-server fixed-port tests, console ANSI styling, and a records-manager scenario fixture. The focused feature suite is green; these failures were not modified in this change.