Skip to content

Add robust external response.schema_ref support (JSON + Zod modules) with metadata, validation, and tests#19

Merged
PredictabilityAtScale merged 1 commit into
mainfrom
codex/add-response-json-schema-validation
May 18, 2026
Merged

Add robust external response.schema_ref support (JSON + Zod modules) with metadata, validation, and tests#19
PredictabilityAtScale merged 1 commit into
mainfrom
codex/add-response-json-schema-validation

Conversation

@PredictabilityAtScale
Copy link
Copy Markdown
Owner

Motivation

  • Let prompts reference a single source-of-truth schema file so teams can author schemas outside YAML and reuse them in prompts and runtime code.
  • Support both simple JSON Schema files and code-first Zod schemas while keeping a safe, discoverable, and debuggable load/compile/validate UX.
  • Harden error handling and DX (actionable error codes, inspectable provenance, and provider-specific warnings) to avoid confusing filesystem or import failures.

Description

  • Added resolver src/parser/response-schema-ref.ts and wired it into the prompt loader so response.schema_ref is resolved relative to the prompt and normalized into response.schema with response.schema_ref removed.
  • Supported file types: .json (parsed as JSON Schema) and .js/.mjs/.cjs (imported modules exporting a Zod schema converted to JSON Schema via zod-to-json-schema), and enforced a whitelist of allowed extensions; .ts/.tsx are intentionally not supported by default.
  • Emit and populate response.schema_source metadata (mode, ref, resolved_path, hash) so inspect and compiled artifacts can show schema provenance.
  • Enforced mutual exclusion of response.schema and response.schema_ref in runtime schema definitions and validation (ResponseSchemaWithValidation).
  • Improved error handling with clear, feature-specific error codes (POK050 for JSON ref errors, POK051 for module/ref import/export/extension issues) and added provider-sensitive schema keyword warnings (POK052/POK053/POK054).
  • Updated docs (docs/schema-dx.md, docs/schema.md, README.md, docs/index.md, docs/overrides.md) to document schema_ref usage, constraints, and the executable-module trust boundary; fixed a promptopspromptopskit wording drift.
  • Added zod-to-json-schema dependency and comprehensive tests covering happy paths and negative cases (invalid JSON, missing module file, module without Zod export, unsupported extensions), and fixed related test organization/regressions.

Testing

  • Ran unit test suite with npm test (Vitest); all tests passed: 13 test files, 221 tests (all green).
  • Ran static typecheck/lint via npm run lint (tsc --noEmit); the command completed successfully.
  • Added/updated integration tests that assert: inspect exposes response.schema_source, validate/compile resolve prompt-relative JSON refs, compile inlines resolved response.schema and removes schema_ref, and failure modes for invalid JSON, missing Zod modules, modules without a Zod export, and unsupported extensions cause the expected CLI validation errors.

Codex Task

@PredictabilityAtScale PredictabilityAtScale merged commit 43ace8f into main May 18, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant