Skip to content

Add explicit sampled-output STOP certification - #973

Closed
bradhilton wants to merge 3 commits into
mainfrom
hayek/explicit-sampled-stop-api-20260925
Closed

bradhilton wants to merge 3 commits into
mainfrom
hayek/explicit-sampled-stop-api-20260925

Conversation

@bradhilton

@bradhilton bradhilton commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Histories built from recorded completions can omit STOP flags even when the original token IDs and logprobs are available. This leaves training code with incomplete stop labels.

await art.tokenize_sampled(...) runs ordinary tokenization, then checks the recorded prompts, outputs, logprobs and stop evidence. It adds only missing, verified STOP flags and preserves tokens, logprobs, order and original objects. Missing logprobs cannot be certified through placeholder NaNs; explicitly recorded NaNs remain valid evidence.

Existing art.tokenize behavior is unchanged. Ordinary tokenization errors still propagate; #976 adds a separate native representation for cases that cannot be rendered faithfully.

Focused regressions and full CI pass. Independent Astra and Fable source reviews found no blockers. The API requires resolvable tokenizer authority for the recorded model; a different base_model cannot substitute for it.

Validation details
  • CI for the current head: lint, formatting, type and lock checks pass; 1,264 lightweight tests and 2,081 unit tests pass, with 34 skipped and 2 deselected.
  • 55 focused public cases include six regressions that previously certified missing logprobs or unsupported finish evidence.
  • GPT-6 Astra xhigh and Claude Fable 5.1 Extra High each returned PASS_WITH_NOTES for head d3f325dcc; a separate changed-source peer also passed.
  • The captured-input validation described in Add an explicit native representation for sampled tokenization #976 exercises this shared guard through native mode. It does not validate the rendered route on that input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant