Skip to content

Add reusable dataset generation API - #16

Draft
owenqwenstarsky wants to merge 1 commit into
mainfrom
codex/add-library-import-functionality
Draft

Add reusable dataset generation API#16
owenqwenstarsky wants to merge 1 commit into
mainfrom
codex/add-library-import-functionality

Conversation

@owenqwenstarsky

Copy link
Copy Markdown
Member

Motivation

  • Provide a programmatic API so the tool can be imported as a library and used to generate datasets without the CLI while keeping the existing CLI behavior intact.

Description

  • Added exported types GenerateDatasetOptions and GenerateDatasetResult and a new generateDataset(options) function that encapsulates prompt reading, concurrency, OpenRouter calls, JSONL writing, spend tracking, and key cleanup.
  • Refactored the existing generation flow out of main() into generateDataset() and updated main() to parse CLI args/env then call the new function so CLI behavior is preserved.
  • Made stderr injectable via the options to allow library consumers to control output and progress rendering, and added a small refactor to use ensureReadableFile() inside the new function.
  • Added an end-to-end test that mocks fetch and validates JSONL output and counts, and bumped lockfile root version to match package.json (0.1.12).

Testing

  • Attempted npm test with the default test runner invocation (failed in this environment due to how the test glob is invoked), and then ran a targeted run: npm run build:test && node --test dist-test/test/datagen.test.js which executed the test suite and all tests passed (18 tests, 0 failures).
  • Ran npm run build which succeeded and produced compiled outputs.
  • Added and ran a new test generateDataset writes JSONL output for prompts that validates output structure and counts and it passed in the test run.

Codex Task

@owenqwenstarsky
owenqwenstarsky marked this pull request as draft February 26, 2026 21:51
@CompactAIOfficial CompactAIOfficial added enhancement New feature or request triaged labels Aug 30, 2026
@CompactAIOfficial CompactAIOfficial changed the title feat: add reusable dataset generation library API while preserving CLI Add reusable dataset generation API, preserve CLI Aug 30, 2026
@CompactAIOfficial CompactAIOfficial changed the title Add reusable dataset generation API, preserve CLI Add reusable dataset generation API Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex enhancement New feature or request triaged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants