Add GenUI Scorecard, stream robustness, and CI + contract-drift guard#6
Open
Vext-Labs wants to merge 4 commits into
Open
Add GenUI Scorecard, stream robustness, and CI + contract-drift guard#6Vext-Labs wants to merge 4 commits into
Vext-Labs wants to merge 4 commits into
Conversation
…k:drift script entry
…n both directions
…dly error mapping - Flush a trailing unterminated data: event after reader done (finish_reason could be lost when a proxy closes without a final newline) - 20s byte-gap watchdog per read with caller-AbortSignal composition; timers cleared on every settle path, socket cancelled on stall - Error taxonomy: keep 401/403 key rejection, add 429 rate-limit, 5xx provider-unavailable, network-failure messages; sanitize other HTTP error bodies to a 160-char JSON-free single line - 18 new tests: chunk-boundary SSE, manual UTF-8 decoder, [DONE], truncated/ dropped, tool-call index accumulation, MAX_TOOL_ROUNDS, NEEDS_LIVE_DATA, watchdog stall/reset, residual flush, every error branch Exports and signatures unchanged (store.ts contract holds).
- scorecard/corpus: 12 exemplar openui-lang screens covering all 33 contract components (incl. TabItem/SelectItem/Series sub-components) - scorecard/harness.ts: parses with the library JSON schema, validates each element's unwrapped props against its zod schema, lists unknown components, formats the score table - __tests__/scorecard.test.tsx: renders every screen through both cupertino and material libraries, writes scorecard/SCORECARD.md, enforces coverage - npm run scorecard + docs/scorecard.md (methodology, how to add entries)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three additive, test-backed quality improvements from a short engineering audit of the streaming + generation pipeline:
1. GenUI Scorecard (
scorecard/) — a headless eval harness that scores openui-lang screens on contract validity (zod), render success through both the Cupertino and Material renderer sets, and unknown-component usage. Ships with a 12-screen corpus covering all 33 contract components, anpm run scorecardtable reporter, anddocs/scorecard.md. This is meant as a small, repo-local complement to the validation-playground discussion in thesysdev/openui#786 — a way to measure generation quality before changing prompts, contracts, or models.2. Stream robustness (
src/genos/stream.ts) — three fixes, all covered by the new__tests__/stream.test.ts(18 tests; this file previously had no direct coverage):data:event at end-of-stream was silently discarded. Latent with Cerebras, a real risk for the advertised "any OpenAI-compatible endpoint".AbortSignal; no dangling timers.3. CI + contract-drift guard — a GitHub Actions workflow (typecheck, jest, drift check) and
scripts/check-contract-drift.mjs, which fails CI ifui/contract.tsxand the embedded system prompt drift apart (allowlist documents the intentional exceptions: Stack, CheckBoxGroup, RadioGroup).Notes
stream.tsare unchanged;store.tsand the tool layer are unaffected.Test plan
npm run typecheck✓npm test— 54 passed (22 existing + 32 new) ✓npm run check:drift✓ (33 contract components, 3 documented exceptions)npm run scorecard— 12/12 corpus screens pass both renderer sets ✓Contributed by Vext Labs.