diff --git a/.changeset/api-sync-docs-rename.md b/.changeset/api-sync-docs-rename.md new file mode 100644 index 00000000..f1b19d4e --- /dev/null +++ b/.changeset/api-sync-docs-rename.md @@ -0,0 +1,6 @@ +--- +"chkit": patch +"@chkit/plugin-ingest": patch +--- + +Point README documentation links at the renamed API Sync section (`/api-sync/`). diff --git a/README.md b/README.md index a8a799bc..0d478093 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,9 @@ chkit is an open-source CLI for ClickHouse. Review migration SQL before applying it. Keep table definitions and API readers in your repository, alongside the code that uses them. Run the CLI from the terminal or CI. -**TypeScript:** schemas, migrations, and ingestion. **Python:** schemas and migrations through [chkit-py](https://chkit.obsessiondb.com/python/overview/). +**TypeScript:** schemas, migrations, and API sync. **Python:** schemas and migrations through [chkit-py](https://chkit.obsessiondb.com/python/overview/). -[Get started](https://chkit.obsessiondb.com/getting-started/) · [Build a data source](https://chkit.obsessiondb.com/ingestion/quickstart/) · [Documentation](https://chkit.obsessiondb.com) +[Get started](https://chkit.obsessiondb.com/getting-started/) · [Build a data source](https://chkit.obsessiondb.com/api-sync/quickstart/) · [Documentation](https://chkit.obsessiondb.com) > **Beta:** the public API is still evolving. Keep the CLI, core, and plugins on matching versions. @@ -117,7 +117,7 @@ export const content = definePipeline({ id: 'content', streams: [postStream] }) bunx chkit ingest run --tag pipeline:content ``` -The default loader writes the rows. For production sources, add [pagination](https://chkit.obsessiondb.com/ingestion/readers/) and [incremental reads](https://chkit.obsessiondb.com/ingestion/incremental-syncs/) when the provider supports them. +The default loader writes the rows. For production sources, add [pagination](https://chkit.obsessiondb.com/api-sync/readers/) and [incremental reads](https://chkit.obsessiondb.com/api-sync/incremental-syncs/) when the provider supports them. ### 4. Query through a view @@ -149,7 +149,7 @@ Expected result for the demo dataset: | 2 | 10 | | 3 | 10 | -If the final shape may change, [retain raw records and transform in ClickHouse](https://chkit.obsessiondb.com/ingestion/destinations/) instead of mapping every field up front. +If the final shape may change, [retain raw records and transform in ClickHouse](https://chkit.obsessiondb.com/api-sync/destinations/) instead of mapping every field up front. ### 5. Evolve the model @@ -177,7 +177,7 @@ bunx chkit ingest list bunx chkit ingest run --tag pipeline:content ``` -Use `check` in CI for migration state and schema drift. Schedule ingestion through cron, CI, or another job runner, with one ingestion process per target. Incremental sources resume from committed state; this full-sync demo reads the dataset again. See [scheduling and recovery](https://chkit.obsessiondb.com/ingestion/operations/). +Use `check` in CI for migration state and schema drift. Schedule ingestion through cron, CI, or another job runner, with one ingestion process per target. Incremental sources resume from committed state; this full-sync demo reads the dataset again. See [scheduling and recovery](https://chkit.obsessiondb.com/api-sync/operations/). ## Set up chkit for your project @@ -185,7 +185,7 @@ Use `check` in CI for migration state and schema drift. Schedule ingestion throu |---|---| | Manage a new schema | [Getting started](https://chkit.obsessiondb.com/getting-started/) | | Adopt an existing database | [Pull a live schema](https://chkit.obsessiondb.com/plugins/pull/) | -| Implement an API source | [Ingestion quickstart](https://chkit.obsessiondb.com/ingestion/quickstart/) | +| Implement an API source | [API sync quickstart](https://chkit.obsessiondb.com/api-sync/quickstart/) | | Generate application types | [TypeScript codegen](https://chkit.obsessiondb.com/plugins/codegen/) | | Recompute stored data | [SQL backfills](https://chkit.obsessiondb.com/plugins/backfill/) | | Work with a coding agent | [Agent skills](https://chkit.obsessiondb.com/ai-agents/) | @@ -208,7 +208,7 @@ See the [CLI reference](https://chkit.obsessiondb.com/cli/overview/) for command ## Python -Install [`chkit-py`](https://pypi.org/project/chkit-py/) (`pip install chkit-py`) to define schemas and run migrations, drift detection, and CI checks with Python config and schema files. API ingestion requires TypeScript. The Python source is in [`chkit_python/`](chkit_python). +Install [`chkit-py`](https://pypi.org/project/chkit-py/) (`pip install chkit-py`) to define schemas and run migrations, drift detection, and CI checks with Python config and schema files. API sync requires TypeScript. The Python source is in [`chkit_python/`](chkit_python). ## Documentation diff --git a/apps/docs/astro.config.mjs b/apps/docs/astro.config.mjs index 65e718b8..9b4e5b6d 100644 --- a/apps/docs/astro.config.mjs +++ b/apps/docs/astro.config.mjs @@ -75,27 +75,27 @@ export default defineConfig({ autogenerate: { directory: 'schema' }, }, { - label: 'Ingestion', + label: 'API Sync', items: [ - { label: 'Overview', slug: 'ingestion' }, - { label: 'Quickstart', slug: 'ingestion/quickstart' }, - { label: 'Authoring skill', slug: 'ingestion/agent-skill' }, + { label: 'Overview', slug: 'api-sync' }, + { label: 'Quickstart', slug: 'api-sync/quickstart' }, + { label: 'Authoring skill', slug: 'api-sync/agent-skill' }, { label: 'Build a source', collapsed: true, items: [ - { slug: 'ingestion/readers' }, - { slug: 'ingestion/destinations' }, - { slug: 'ingestion/incremental-syncs' }, - { slug: 'ingestion/loading' }, + { slug: 'api-sync/readers' }, + { slug: 'api-sync/destinations' }, + { slug: 'api-sync/incremental-syncs' }, + { slug: 'api-sync/loading' }, ], }, { label: 'Run and verify', collapsed: true, items: [ - { slug: 'ingestion/operations' }, - { slug: 'ingestion/testing' }, + { slug: 'api-sync/operations' }, + { slug: 'api-sync/testing' }, ], }, ], diff --git a/apps/docs/functions/_middleware.ts b/apps/docs/functions/_middleware.ts index b1a15972..b79319dc 100644 --- a/apps/docs/functions/_middleware.ts +++ b/apps/docs/functions/_middleware.ts @@ -1,6 +1,14 @@ export const onRequest: PagesFunction<{ ASSETS: Fetcher }> = async (context) => { const url = new URL(context.request.url); + // The Ingestion section moved to /api-sync/. Keep old links (READMEs, + // installed agent skills, search results) working, including .md URLs. + const moved = url.pathname.match(/^\/ingestion(\/.*|\.md)?$/); + if (moved) { + url.pathname = `/api-sync${moved[1] ?? '/'}`; + return Response.redirect(url.toString(), 301); + } + // Serve the raw Markdown asset behind /_raw/ as text/markdown. async function serveRaw(slug: string): Promise { const assetUrl = new URL(`/_raw/${slug}.md`, url.origin); diff --git a/apps/docs/src/components/Footer.astro b/apps/docs/src/components/Footer.astro index 31694afc..ca40d82a 100644 --- a/apps/docs/src/components/Footer.astro +++ b/apps/docs/src/components/Footer.astro @@ -41,7 +41,7 @@ const repo = 'https://github.com/obsessiondb/chkit';
  • CLI reference
  • Configuration
  • Schema
    • -
  • Ingestion
    • +
  • API Sync
    • diff --git a/apps/docs/src/components/Hero.astro b/apps/docs/src/components/Hero.astro index de33c0da..e333aa9b 100644 --- a/apps/docs/src/components/Hero.astro +++ b/apps/docs/src/components/Hero.astro @@ -22,7 +22,7 @@ const isHome = Astro.locals.starlightRoute.slug === ''; ))}
    See the six-step example -

    TypeScript schema + ingestion Python schema + migrations

    +

    TypeScript schema + API sync Python schema + migrations

    Built by ObsessionDBOpen source · MIT

    diff --git a/apps/docs/src/content/docs/ai-agents.md b/apps/docs/src/content/docs/ai-agents.md index a42ea630..4317a93f 100644 --- a/apps/docs/src/content/docs/ai-agents.md +++ b/apps/docs/src/content/docs/ai-agents.md @@ -23,7 +23,7 @@ Append `.md` to a documentation URL to read Markdown, such as [`/ai-agents.md`]( ## What chkit is -Use chkit to define ClickHouse schemas in TypeScript or Python, generate migration SQL, and check the live database for drift. Use the ingestion plugin to load API data with TypeScript readers. +Use chkit to define ClickHouse schemas in TypeScript or Python, generate migration SQL, and check the live database for drift. Use the ingest plugin to sync API data with TypeScript readers. Run chkit through shell commands. Install the agent skills for command and authoring guidance. @@ -56,15 +56,15 @@ chkit skills add obsessiondb/chkit --skill chkit The skill installs into the project's agent directory (for example `.claude/skills/chkit/` or `.agents/skills/chkit/`). On an interactive `chkit init`, chkit also detects the active agent and offers to install the skill automatically. -### Authoring ingestion sources +### Authoring API sync sources -For TypeScript API ingestion, install the focused authoring skill: +For TypeScript API sync, install the focused authoring skill: ```sh chkit skills add obsessiondb/chkit --skill chkit-ingestion ``` -It guides decisions about raw versus shaped data, transformations, pagination, incremental state, and loaders, then links to the relevant docs. Start with the [ingestion quickstart](/ingestion/quickstart/); each guide explains when to use its alternatives. Ingestion requires a direct `clickhouse` connection; the workbench executor alone is insufficient. See [skill installation and usage](/ingestion/agent-skill/). +It guides decisions about raw versus shaped data, transformations, pagination, incremental state, and loaders, then links to the relevant docs. Start with the [API sync quickstart](/api-sync/quickstart/); each guide explains when to use its alternatives. API sync requires a direct `clickhouse` connection; the workbench executor alone is insufficient. See [skill installation and usage](/api-sync/agent-skill/). ## Step 3: Scaffold based on the answers @@ -148,7 +148,7 @@ In TypeScript, plugins are npm packages registered in the `plugins` array of `cl | Adopt chkit on an **existing** ClickHouse database | [`@chkit/plugin-pull`](/plugins/pull/) | Introspects the live database into local schema files so the user starts from real tables, not a blank example. | | Generate **typed row models**: TypeScript types (and optional Zod schemas), or Pydantic models in Python: from the schema | [`@chkit/plugin-codegen`](/plugins/codegen/) | Keeps application row types in sync with the schema definitions. | | **Backfill** historical data into materialized views | [`@chkit/plugin-backfill`](/plugins/backfill/) | Time-windowed loads with checkpoints, for large or resumable backfills. | -| **Ingest application API data** into ClickHouse | [`@chkit/plugin-ingest`](/ingestion/) | TypeScript only; finite pulls with journaled checkpoints and an external scheduler. | +| **Ingest application API data** into ClickHouse | [`@chkit/plugin-ingest`](/api-sync/) | TypeScript only; finite pulls with journaled checkpoints and an external scheduler. | | Deploy to **ObsessionDB** | [`@chkit/plugin-obsessiondb`](/obsessiondb/overview/) | ObsessionDB connection and engine configuration; rewrites `Shared` engines when targeting non-ObsessionDB ClickHouse. | Install plugins for the project's stated requirements. diff --git a/apps/docs/src/content/docs/ingestion/agent-skill.md b/apps/docs/src/content/docs/api-sync/agent-skill.md similarity index 79% rename from apps/docs/src/content/docs/ingestion/agent-skill.md rename to apps/docs/src/content/docs/api-sync/agent-skill.md index 504135e2..6509ff83 100644 --- a/apps/docs/src/content/docs/ingestion/agent-skill.md +++ b/apps/docs/src/content/docs/api-sync/agent-skill.md @@ -1,6 +1,6 @@ --- -title: Ingestion authoring skill -description: Install a concise agent skill for implementing new chkit ingestion sources. +title: API sync authoring skill +description: Install a concise agent skill for implementing new chkit API sync sources. --- Install `chkit-ingestion` to give your coding agent source-authoring instructions and links to the relevant guides. @@ -19,7 +19,7 @@ The TypeScript CLI also provides a pass-through: chkit skills add obsessiondb/chkit --skill chkit-ingestion ``` -Choose the agent in the installer. Install `@chkit/plugin-ingest` and configure credentials through the [quickstart](/ingestion/quickstart/) before running a sync. The separate `chkit` skill covers schema and migration workflows. +Choose the agent in the installer. Install `@chkit/plugin-ingest` and configure credentials through the [quickstart](/api-sync/quickstart/) before running a sync. The separate `chkit` skill covers schema and migration workflows. For a local checkout containing the skill, use: @@ -48,6 +48,6 @@ The skill covers the authoring steps and requirements for bounded readers, check ## Related pages -- [Destinations and transformations](/ingestion/destinations/): choose storage and mapping for the source. -- [Quickstart](/ingestion/quickstart/): a working first source. +- [Destinations and transformations](/api-sync/destinations/): choose storage and mapping for the source. +- [Quickstart](/api-sync/quickstart/): a working first source. - [For AI agents](/ai-agents/): general chkit setup and schema workflows. diff --git a/apps/docs/src/content/docs/ingestion/destinations.md b/apps/docs/src/content/docs/api-sync/destinations.md similarity index 96% rename from apps/docs/src/content/docs/ingestion/destinations.md rename to apps/docs/src/content/docs/api-sync/destinations.md index 1ad0a111..d14bce30 100644 --- a/apps/docs/src/content/docs/ingestion/destinations.md +++ b/apps/docs/src/content/docs/api-sync/destinations.md @@ -108,7 +108,7 @@ Changing the mapping affects future writes. Updating older rows requires replayi Decide what one row represents alongside its stored shape. A reader can fetch a root object and its children, but chkit does not discover relationships or load child objects automatically. -**Embed children when the application consumes a complete object.** For example, retain a ticket and its comments together for document retrieval. This denormalized shape keeps its context in one record, at the cost of rebuilding that record when a child changes. Fetch all required child pages before yielding the root; see [nested loading](/ingestion/readers/#parent-records-and-child-collections). +**Embed children when the application consumes a complete object.** For example, retain a ticket and its comments together for document retrieval. This denormalized shape keeps its context in one record, at the cost of rebuilding that record when a child changes. Fetch all required child pages before yielding the root; see [nested loading](/api-sync/readers/#parent-records-and-child-collections). **Use separate entity tables when children need independent queries or updates.** For a warehouse, `tickets`, `ticket_comments`, and `customers` can have their own columns and stable keys, with `ticket_id` and `customer_id` linking them. Each stream has one destination; define separate streams for separately loaded tables, or derive them in ClickHouse from retained raw objects. @@ -200,7 +200,7 @@ A separate stream can read retained raw tables, assemble documents in `read`, an ## Related pages -- [Readers and pagination](/ingestion/readers/): fetch root objects and their children. -- [Loading and batching](/ingestion/loading/): start with the default loader. +- [Readers and pagination](/api-sync/readers/): fetch root objects and their children. +- [Loading and batching](/api-sync/loading/): start with the default loader. - [Schema DSL](/schema/dsl-reference/): table and view definitions. -- [Scheduling and recovery](/ingestion/operations/#backfill-source-data): source rereads versus SQL backfills. +- [Scheduling and recovery](/api-sync/operations/#backfill-source-data): source rereads versus SQL backfills. diff --git a/apps/docs/src/content/docs/ingestion/incremental-syncs.md b/apps/docs/src/content/docs/api-sync/incremental-syncs.md similarity index 88% rename from apps/docs/src/content/docs/ingestion/incremental-syncs.md rename to apps/docs/src/content/docs/api-sync/incremental-syncs.md index 5efe089b..4d692050 100644 --- a/apps/docs/src/content/docs/ingestion/incremental-syncs.md +++ b/apps/docs/src/content/docs/api-sync/incremental-syncs.md @@ -7,9 +7,9 @@ Use an incremental strategy to select the records to fetch and the state to comm ## Start with a full sync -For a small, bounded dataset, omit `incremental` as in the [quickstart](/ingestion/quickstart/). Each run reads the source again with no bookmark. Keep this approach when the provider has no reliable change filter and the cost of a complete read is acceptable. +For a small, bounded dataset, omit `incremental` as in the [quickstart](/api-sync/quickstart/). Each run reads the source again with no bookmark. Keep this approach when the provider has no reliable change filter and the cost of a complete read is acceptable. -A full sync does not replace the destination snapshot or detect missing records. See [deletion handling](/ingestion/destinations/#handle-deleted-records) before treating the destination as a complete current-state mirror. +A full sync does not replace the destination snapshot or detect missing records. See [deletion handling](/api-sync/destinations/#handle-deleted-records) before treating the destination as a complete current-state mirror. For expensive full reads, use the provider's change mechanism: a timestamp window for time filters, or durable provider state for a resumable change feed. Pagination tokens alone do not establish either contract. @@ -17,7 +17,7 @@ For expensive full reads, use the provider's change mechanism: a timestamp windo `start` is the first run's lower bound. Later runs start at the last watermark minus `overlapMs` (default `0`). The upper bound is a fixed execution cutoff, or the explicit backfill upper bound. Choose overlap based on provider indexing delays and late updates; reconcile repeated records in the destination. -This complete source module uses the client from [Readers and pagination](/ingestion/readers/#a-reusable-page-client). Save it as `src/sources/helpdesk.ts` and re-export its table and pipeline from the project entry: +This complete source module uses the client from [Readers and pagination](/api-sync/readers/#a-reusable-page-client). Save it as `src/sources/helpdesk.ts` and re-export its table and pipeline from the project entry: ```ts import { definePipeline, defineStream, paginate, rawRows, rawTable, timestampWindow } from '@chkit/plugin-ingest' @@ -126,6 +126,6 @@ The built-in full-sync and cursor strategies do not interpret `--from` / `--to`. ## Related pages -- [Readers and pagination](/ingestion/readers/): provider contracts and bounded fetching. -- [Loading and batching](/ingestion/loading/#batch-identity): checkpoint state versus batch identity. -- [Scheduling and recovery](/ingestion/operations/#backfill-source-data): isolated historical runs. +- [Readers and pagination](/api-sync/readers/): provider contracts and bounded fetching. +- [Loading and batching](/api-sync/loading/#batch-identity): checkpoint state versus batch identity. +- [Scheduling and recovery](/api-sync/operations/#backfill-source-data): isolated historical runs. diff --git a/apps/docs/src/content/docs/api-sync/index.md b/apps/docs/src/content/docs/api-sync/index.md new file mode 100644 index 00000000..009eb6ab --- /dev/null +++ b/apps/docs/src/content/docs/api-sync/index.md @@ -0,0 +1,34 @@ +--- +title: API Sync +description: Read application APIs into ClickHouse with TypeScript streams and durable checkpoints. +--- + +chkit API sync runs finite pulls from application APIs, loads rows into schema-managed ClickHouse tables, and records progress after writes succeed. + +## How it fits together + +Start with **API → reader → raw table → SQL view** when the final shape may change. For an established schema, map records in the reader and load typed tables. A loader writes the rows; a checkpoint records where the next execution should resume. + +- A **stream** owns a stable ID, destination, reader, and incremental strategy. +- A **pipeline** groups streams, tags, concurrency limits, and retry defaults. It has no durable state and does not order dependent streams. +- A **run** executes a selection of streams once. Cron, CI, or another scheduler starts the next run. + +Use TypeScript and a direct `clickhouse` connection, including for ObsessionDB databases. The workbench executor does not support API sync. Create destination tables through schema migrations; ingestion creates its journal and writes data. + +## Start + +- [Quickstart](/api-sync/quickstart/): ingest a small public API and query the result. +- [Install the authoring skill](/api-sync/agent-skill/): give a coding agent the authoring workflow and relevant documentation. + +## Build + +- [Readers and pagination](/api-sync/readers/): provider requests, credentials, SDKs, and bounded pages. +- [Destinations and transformations](/api-sync/destinations/): raw or shaped storage, related objects, current state, and history. +- [Incremental syncs](/api-sync/incremental-syncs/): full syncs, timestamp windows, and provider state. +- [Loading and batching](/api-sync/loading/): use the default loader and tune it when needed. + +## Operate + +- [Scheduling and recovery](/api-sync/operations/): tags, retries, budgets, backfills, and monitoring. +- [Test a source](/api-sync/testing/): exercise checkpoints and recovery without a live database. +- [Plugin reference](/plugins/ingest/): configuration and command summary. diff --git a/apps/docs/src/content/docs/ingestion/loading.md b/apps/docs/src/content/docs/api-sync/loading.md similarity index 92% rename from apps/docs/src/content/docs/ingestion/loading.md rename to apps/docs/src/content/docs/api-sync/loading.md index 929089d3..3d3e6150 100644 --- a/apps/docs/src/content/docs/ingestion/loading.md +++ b/apps/docs/src/content/docs/api-sync/loading.md @@ -7,7 +7,7 @@ A loader writes the rows yielded by the reader, whether they contain raw objects ## Start with the default loader -Omit `loader` from the stream definition, as in the [quickstart](/ingestion/quickstart/). chkit uses `simpleLoader()` to insert rows and attach ingestion metadata. Use this for both raw and shaped storage, including rows with embedded children. +Omit `loader` from the stream definition, as in the [quickstart](/api-sync/quickstart/). chkit uses `simpleLoader()` to insert rows and attach ingestion metadata. Use this for both raw and shaped storage, including rows with embedded children. `simpleLoader` is the only bundled loader. It makes direct, at-least-once inserts, adds ingestion metadata, and splits a load batch into deterministic write units with stable deduplication tokens. It does not stage data or replace a destination snapshot atomically. @@ -76,6 +76,6 @@ The stream's `retry` policy controls source attempts and reader recovery. Load a ## Related pages -- [Destinations and transformations](/ingestion/destinations/): map rows without replacing the loader. -- [Incremental syncs](/ingestion/incremental-syncs/): when committed writes advance progress. -- [Test a source](/ingestion/testing/): simulate a failed write and replay. +- [Destinations and transformations](/api-sync/destinations/): map rows without replacing the loader. +- [Incremental syncs](/api-sync/incremental-syncs/): when committed writes advance progress. +- [Test a source](/api-sync/testing/): simulate a failed write and replay. diff --git a/apps/docs/src/content/docs/ingestion/operations.md b/apps/docs/src/content/docs/api-sync/operations.md similarity index 96% rename from apps/docs/src/content/docs/ingestion/operations.md rename to apps/docs/src/content/docs/api-sync/operations.md index 74c66cb8..04e0ca83 100644 --- a/apps/docs/src/content/docs/ingestion/operations.md +++ b/apps/docs/src/content/docs/api-sync/operations.md @@ -116,12 +116,12 @@ Retry with `ingest run` after fixing the error; the journal supplies committed s For dependent stages, such as raw ingestion followed by a derived-document stream, invoke the stages sequentially and check the first exit code before starting the second. An unfiltered run may execute both at once; exported pipelines do not form a dependency graph. -Use programmatic `runIngestion` when embedding execution in an existing service or supplying runtime adapters. The host then owns configuration, cancellation, and outcome handling. See [Testing](/ingestion/testing/) for a complete runtime example. +Use programmatic `runIngestion` when embedding execution in an existing service or supplying runtime adapters. The host then owns configuration, cancellation, and outcome handling. See [Testing](/api-sync/testing/) for a complete runtime example. ## Related pages -- [Incremental syncs](/ingestion/incremental-syncs/): strategy-dependent resumption. -- [Loading and batching](/ingestion/loading/): concurrency, buffers, and write identity. -- [Test a source](/ingestion/testing/): verify recovery before scheduling it. +- [Incremental syncs](/api-sync/incremental-syncs/): strategy-dependent resumption. +- [Loading and batching](/api-sync/loading/): concurrency, buffers, and write identity. +- [Test a source](/api-sync/testing/): verify recovery before scheduling it. diff --git a/apps/docs/src/content/docs/ingestion/quickstart.md b/apps/docs/src/content/docs/api-sync/quickstart.md similarity index 90% rename from apps/docs/src/content/docs/ingestion/quickstart.md rename to apps/docs/src/content/docs/api-sync/quickstart.md index 8fc01487..5fdc69cc 100644 --- a/apps/docs/src/content/docs/ingestion/quickstart.md +++ b/apps/docs/src/content/docs/api-sync/quickstart.md @@ -1,5 +1,5 @@ --- -title: Ingestion quickstart +title: API sync quickstart description: Read a public API into a raw ClickHouse table and query it through a SQL view. --- @@ -9,7 +9,7 @@ Ingest the small [JSONPlaceholder posts dataset](https://jsonplaceholder.typicod Use a TypeScript project, Bun, a direct ClickHouse connection, and a chkit release that includes `@chkit/plugin-ingest`. This example uses native JSON; use ClickHouse 25.3+ for production support of that type. See [compatibility](/guides/clickhouse-compatibility/). -JSONPlaceholder is a public demonstration API with a small dataset, so use a full sync. For larger sources, add [pagination](/ingestion/readers/) and [incremental reads](/ingestion/incremental-syncs/) based on the provider API. +JSONPlaceholder is a public demonstration API with a small dataset, so use a full sync. For larger sources, add [pagination](/api-sync/readers/) and [incremental reads](/api-sync/incremental-syncs/) based on the provider API. ## Install and configure @@ -106,6 +106,6 @@ Keep a small first source in one entry file. As sources grow, move each provider ## Related pages -- [Destinations and transformations](/ingestion/destinations/): choose the stored shape and where to map fields. -- [Readers and pagination](/ingestion/readers/): fetch bounded pages with retries. -- [Incremental syncs](/ingestion/incremental-syncs/): avoid rereading the whole source. +- [Destinations and transformations](/api-sync/destinations/): choose the stored shape and where to map fields. +- [Readers and pagination](/api-sync/readers/): fetch bounded pages with retries. +- [Incremental syncs](/api-sync/incremental-syncs/): avoid rereading the whole source. diff --git a/apps/docs/src/content/docs/ingestion/readers.md b/apps/docs/src/content/docs/api-sync/readers.md similarity index 89% rename from apps/docs/src/content/docs/ingestion/readers.md rename to apps/docs/src/content/docs/api-sync/readers.md index 46faee9c..8175988e 100644 --- a/apps/docs/src/content/docs/ingestion/readers.md +++ b/apps/docs/src/content/docs/api-sync/readers.md @@ -44,7 +44,7 @@ export async function fetchTicketPage( } ``` -This helper performs one request. Call it through `context.attempt` or from `paginate`'s `fetchPage` callback, as shown in [Incremental syncs](/ingestion/incremental-syncs/#timestamp-windows). Validate untrusted response shapes at the source boundary in production; a TypeScript assertion does not validate JSON. +This helper performs one request. Call it through `context.attempt` or from `paginate`'s `fetchPage` callback, as shown in [Incremental syncs](/api-sync/incremental-syncs/#timestamp-windows). Validate untrusted response shapes at the source boundary in production; a TypeScript assertion does not validate JSON. ## Pagination is not a checkpoint @@ -61,7 +61,7 @@ Use a manual loop when page metadata must become a chunk `id` or durable `state` ## Bound work and separate accounts -Yield pages as they arrive; do not collect a large source into one array. Source page size controls response memory; `batchSize` controls loading and is not a hard limit on a yielded chunk. See [Loading and batching](/ingestion/loading/). +Yield pages as they arrive; do not collect a large source into one array. Source page size controls response memory; `batchSize` controls loading and is not a hard limit on a yielded chunk. See [Loading and batching](/api-sync/loading/). Use stable stream IDs for independently resumable accounts or resources, for example `helpdesk.account-42.tickets`. The stream ID owns the checkpoint. If several accounts share a destination table, include account identity in the record key too; a provider-local ticket ID alone may collide. @@ -75,10 +75,10 @@ For a document that needs a root object and its children, assemble one complete The default loader writes the assembled row. There is no automatic nested loader or parent/child registration. If a required child request fails, let the reader fail before yielding that root; do not publish a partial object as complete. Bound the assembled object size and report any provider-imposed truncation. -For separately queried children, load a `ticket_comments` table with its own stream, stable comment IDs, and a `ticket_id` column. A child stream can discover its own roots when the provider only exposes parent-scoped endpoints. It must not assume another stream has already loaded the parents: pipeline execution does not order dependencies. Choose the row model alongside [stored shape](/ingestion/destinations/#root-objects-and-related-tables). +For separately queried children, load a `ticket_comments` table with its own stream, stable comment IDs, and a `ticket_id` column. A child stream can discover its own roots when the provider only exposes parent-scoped endpoints. It must not assume another stream has already loaded the parents: pipeline execution does not order dependencies. Choose the row model alongside [stored shape](/api-sync/destinations/#root-objects-and-related-tables). :::caution[Child changes need a discovery path] -If editing or deleting a comment does not update the ticket's timestamp, an incremental ticket scan may never revisit it. Use a child change feed or an application-defined reconciliation scan. Handle [deleted records](/ingestion/destinations/#handle-deleted-records) explicitly. +If editing or deleting a comment does not update the ticket's timestamp, an incremental ticket scan may never revisit it. Use a child change feed or an application-defined reconciliation scan. Handle [deleted records](/api-sync/destinations/#handle-deleted-records) explicitly. ::: ## Use an SDK when it helps @@ -94,6 +94,6 @@ Pass `FetchContext` to reusable clients that need only `attempt` and `signal`; t ## Related pages -- [Incremental syncs](/ingestion/incremental-syncs/): choose the durable resume boundary. -- [Scheduling and recovery](/ingestion/operations/#retries-and-provider-errors): retry policies and provider error classification. -- [Test a source](/ingestion/testing/): check pagination and failure behavior with fixtures. +- [Incremental syncs](/api-sync/incremental-syncs/): choose the durable resume boundary. +- [Scheduling and recovery](/api-sync/operations/#retries-and-provider-errors): retry policies and provider error classification. +- [Test a source](/api-sync/testing/): check pagination and failure behavior with fixtures. diff --git a/apps/docs/src/content/docs/ingestion/testing.md b/apps/docs/src/content/docs/api-sync/testing.md similarity index 95% rename from apps/docs/src/content/docs/ingestion/testing.md rename to apps/docs/src/content/docs/api-sync/testing.md index ddf685f9..5109de3e 100644 --- a/apps/docs/src/content/docs/ingestion/testing.md +++ b/apps/docs/src/content/docs/api-sync/testing.md @@ -87,6 +87,6 @@ Start with `chkit ingest list` and `chkit check --offline` to verify exports and ## Related pages -- [Readers and pagination](/ingestion/readers/): request and source contracts. -- [Incremental syncs](/ingestion/incremental-syncs/): durable state boundaries. -- [Scheduling and recovery](/ingestion/operations/): interpret production outcomes. +- [Readers and pagination](/api-sync/readers/): request and source contracts. +- [Incremental syncs](/api-sync/incremental-syncs/): durable state boundaries. +- [Scheduling and recovery](/api-sync/operations/): interpret production outcomes. diff --git a/apps/docs/src/content/docs/cli/overview.mdx b/apps/docs/src/content/docs/cli/overview.mdx index c043387d..6cd8b857 100644 --- a/apps/docs/src/content/docs/cli/overview.mdx +++ b/apps/docs/src/content/docs/cli/overview.mdx @@ -24,7 +24,7 @@ Use the `chkit` CLI to define ClickHouse schemas in TypeScript or Python, review | [`chkit pull`](/cli/pull/) | Introspect live ClickHouse and generate a schema file | | [`chkit codegen`](/cli/codegen/) | Generate typed row models from schema definitions | | [`chkit plugin`](/cli/plugin/) | List or run plugin commands | -| [`chkit ingest`](/plugins/ingest/#commands) | Run, list, or inspect API ingestion streams (`@chkit/plugin-ingest`, TypeScript only) | +| [`chkit ingest`](/plugins/ingest/#commands) | Run, list, or inspect API sync streams (`@chkit/plugin-ingest`, TypeScript only) | ## Connection requirements diff --git a/apps/docs/src/content/docs/getting-started/index.mdx b/apps/docs/src/content/docs/getting-started/index.mdx index d2b47b96..87e2481c 100644 --- a/apps/docs/src/content/docs/getting-started/index.mdx +++ b/apps/docs/src/content/docs/getting-started/index.mdx @@ -11,7 +11,7 @@ import CopyPromptButton from '../../../components/CopyPromptButton.astro'; chkit manages ClickHouse schemas and migrations in TypeScript and Python, and ingests application APIs through TypeScript streams. Pick the path that matches what you're working on. -For API data, start with the [ingestion quickstart](/ingestion/quickstart/) or [install the ingestion skill](/ingestion/agent-skill/). +For API data, start with the [API sync quickstart](/api-sync/quickstart/) or [install the authoring skill](/api-sync/agent-skill/). :::note[Working in Python?] Install [chkit-py](/python/overview/) with `pip install chkit-py`. Use the Python tabs in the schema and CLI reference pages. diff --git a/apps/docs/src/content/docs/guides/clickhouse-compatibility.md b/apps/docs/src/content/docs/guides/clickhouse-compatibility.md index 86cb4106..8571cbab 100644 --- a/apps/docs/src/content/docs/guides/clickhouse-compatibility.md +++ b/apps/docs/src/content/docs/guides/clickhouse-compatibility.md @@ -17,7 +17,7 @@ A few schema features depend on the ClickHouse version of your target: | Feature | Requirement | |---------|-------------| -| Ingestion `rawTable` (native `JSON`) | Native JSON is production-ready on **25.3+**. For older targets, use a [custom destination](/ingestion/destinations/) with supported column types. See the [ClickHouse JSON reference](https://clickhouse.com/docs/reference/data-types/newjson). | +| API sync `rawTable` (native `JSON`) | Native JSON is production-ready on **25.3+**. For older targets, use a [custom destination](/api-sync/destinations/) with supported column types. See the [ClickHouse JSON reference](https://clickhouse.com/docs/reference/data-types/newjson). | | [Refreshable materialized views](/schema/refreshable-views/) | Production-ready on **24.10+** (no flag). Experimental and flag-gated on 23.12–24.9. chkit targets 24.10+. | | `set` data-skipping index | **ClickHouse 26+** requires the `set(0)` form rather than a bare `set`; chkit emits `set(maxRows)` accordingly. See the [DSL reference](/schema/dsl-reference/). | | `uniqueKey` | Renders `UNIQUE KEY` DDL, which is supported on ObsessionDB / ClickHouse Cloud engines but rejected by vanilla `MergeTree`. | diff --git a/apps/docs/src/content/docs/index.mdx b/apps/docs/src/content/docs/index.mdx index cde18bae..31931f27 100644 --- a/apps/docs/src/content/docs/index.mdx +++ b/apps/docs/src/content/docs/index.mdx @@ -1,5 +1,5 @@ --- -title: "chkit: ClickHouse schemas and API ingestion" +title: "chkit: ClickHouse schemas and API sync" description: Define ClickHouse schemas, review migrations, and sync API data into your tables with chkit. template: splash prev: false @@ -12,7 +12,7 @@ hero: link: /getting-started/ icon: right-arrow - text: Sync API data - link: /ingestion/quickstart/ + link: /api-sync/quickstart/ variant: secondary --- @@ -32,22 +32,22 @@ import { Tabs, TabItem } from '@astrojs/starlight/components'; Explore backfills
    -

    03 / Ingestion

    +

    03 / API Sync

    Sync data from any source.

    Build reliable syncs from HTTP APIs, databases, and other sources into ClickHouse.

    - Build a data source + Build a data source
    -

    Schema and ingestion example

    +

    Schema and API sync example

    ## From a schema to a working data sync Start with a table, load 100 demo API records, then query posts per author. Change the view later using the data already stored in ClickHouse. -

    Install the packages and configure a ClickHouse connection before running the example. Add the TypeScript snippets to src/chkit.ts. For schema management without ingestion, use the schema tutorial.

    +

    Install the packages and configure a ClickHouse connection before running the example. Add the TypeScript snippets to src/chkit.ts. For schema management without API sync, use the schema tutorial.

    @@ -73,14 +73,14 @@ export const posts = table({ }) ``` -[Explore the schema DSL](/schema/dsl-reference/). Schema management is available in TypeScript and Python; this walkthrough uses TypeScript for ingestion. +[Explore the schema DSL](/schema/dsl-reference/). Schema management is available in TypeScript and Python; this walkthrough uses TypeScript for API sync. ### Generate and review a migration -Point the project config at `src/chkit.ts` and register `ingest()` when adding ingestion. With a ClickHouse connection configured, generate the migration and inspect it before applying. +Point the project config at `src/chkit.ts` and register `ingest()` when adding API sync. With a ClickHouse connection configured, generate the migration and inspect it before applying. ```sh title="Terminal" bunx chkit generate --name create-posts @@ -90,7 +90,7 @@ bunx chkit migrate bunx chkit migrate --apply ``` -Commit the schema, migration SQL, and snapshot together. The [setup guide](/ingestion/quickstart/#install-and-configure) includes the complete config and package installation. +Commit the schema, migration SQL, and snapshot together. The [setup guide](/api-sync/quickstart/#install-and-configure) includes the complete config and package installation.
    See the generated table SQL @@ -140,7 +140,7 @@ const postStream = defineStream({ export const content = definePipeline({ id: 'content', streams: [postStream] }) ``` -Run `bunx chkit ingest run --tag pipeline:content`. This bounded demo reads all posts; add [pagination](/ingestion/readers/) and [incremental state](/ingestion/incremental-syncs/) when the source needs them. +Run `bunx chkit ingest run --tag pipeline:content`. This bounded demo reads all posts; add [pagination](/api-sync/readers/) and [incremental state](/api-sync/incremental-syncs/) when the source needs them.

    Result: 100 posts in default.posts, ready to query. This reader supplies the data; the default loader supplies batching and ingestion metadata.

    @@ -181,7 +181,7 @@ Expected result for the demo dataset:
    -If the final schema is still evolving, [retain raw objects and transform them in ClickHouse](/ingestion/destinations/) instead of mapping every field up front. +If the final schema is still evolving, [retain raw objects and transform them in ClickHouse](/api-sync/destinations/) instead of mapping every field up front. @@ -202,7 +202,7 @@ export const postsByAuthor = view({ }) ``` -Generate another migration, review it, and apply it. chkit recreates this ordinary view with the new SQL. [Learn about transformations](/ingestion/destinations/). +Generate another migration, review it, and apply it. chkit recreates this ordinary view with the new SQL. [Learn about transformations](/api-sync/destinations/).

    Query the new matching_posts measure over the records loaded in step 3. This view change requires no API re-fetch or table backfill.

    @@ -224,7 +224,7 @@ bunx chkit ingest list bunx chkit ingest run --tag pipeline:content ``` -Serialize ingestion processes per target. When using an incremental source, the next run starts from its committed state. See [scheduling and recovery](/ingestion/operations/) and [source testing](/ingestion/testing/). +Serialize ingestion processes per target. When using an incremental source, the next run starts from its committed state. See [scheduling and recovery](/api-sync/operations/) and [source testing](/api-sync/testing/).     @@ -283,5 +283,5 @@ Serialize ingestion processes per target. When using an incremental source, the

    Set up chkit.

    -

    Manage a schema →Sync your first source →

    +

    Manage a schema →Sync your first source →

    diff --git a/apps/docs/src/content/docs/ingestion/index.md b/apps/docs/src/content/docs/ingestion/index.md deleted file mode 100644 index 17c4f2c7..00000000 --- a/apps/docs/src/content/docs/ingestion/index.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Ingestion -description: Read application APIs into ClickHouse with TypeScript streams and durable checkpoints. ---- - -chkit ingestion runs finite pulls from application APIs, loads rows into schema-managed ClickHouse tables, and records progress after writes succeed. - -## How it fits together - -Start with **API → reader → raw table → SQL view** when the final shape may change. For an established schema, map records in the reader and load typed tables. A loader writes the rows; a checkpoint records where the next execution should resume. - -- A **stream** owns a stable ID, destination, reader, and incremental strategy. -- A **pipeline** groups streams, tags, concurrency limits, and retry defaults. It has no durable state and does not order dependent streams. -- A **run** executes a selection of streams once. Cron, CI, or another scheduler starts the next run. - -Use TypeScript and a direct `clickhouse` connection, including for ObsessionDB databases. The workbench executor does not support ingestion. Create destination tables through schema migrations; ingestion creates its journal and writes data. - -## Start - -- [Quickstart](/ingestion/quickstart/): ingest a small public API and query the result. -- [Install the authoring skill](/ingestion/agent-skill/): give a coding agent the authoring workflow and relevant documentation. - -## Build - -- [Readers and pagination](/ingestion/readers/): provider requests, credentials, SDKs, and bounded pages. -- [Destinations and transformations](/ingestion/destinations/): raw or shaped storage, related objects, current state, and history. -- [Incremental syncs](/ingestion/incremental-syncs/): full syncs, timestamp windows, and provider state. -- [Loading and batching](/ingestion/loading/): use the default loader and tune it when needed. - -## Operate - -- [Scheduling and recovery](/ingestion/operations/): tags, retries, budgets, backfills, and monitoring. -- [Test a source](/ingestion/testing/): exercise checkpoints and recovery without a live database. -- [Plugin reference](/plugins/ingest/): configuration and command summary. diff --git a/apps/docs/src/content/docs/plugins/ingest.md b/apps/docs/src/content/docs/plugins/ingest.md index 0637f891..7e15d042 100644 --- a/apps/docs/src/content/docs/plugins/ingest.md +++ b/apps/docs/src/content/docs/plugins/ingest.md @@ -7,7 +7,7 @@ sidebar: Use `@chkit/plugin-ingest` to load application API data into tables defined in your chkit schema. -Start with the [ingestion quickstart](/ingestion/quickstart/) for a complete first sync. Read the [ingestion guides](/ingestion/) for reader, storage, and checkpoint choices, or install the [authoring skill](/ingestion/agent-skill/) for a coding agent. +Start with the [API sync quickstart](/api-sync/quickstart/) for a complete first sync. Read the [API sync guides](/api-sync/) for reader, storage, and checkpoint choices, or install the [authoring skill](/api-sync/agent-skill/) for a coding agent. ## Capabilities @@ -129,7 +129,7 @@ const ticketStream = defineStream({ }) ``` -The raw table uses `ReplacingMergeTree` to retain the latest ingested version per `id`. Query with `FINAL` to resolve repeats before background merges finish. Changes to an ordinary view can use retained fields without re-fetching the source. A materialized view needs a backfill to update stored results; fields you did not retain require a source re-fetch. See [Destinations and transformations](/ingestion/destinations/). +The raw table uses `ReplacingMergeTree` to retain the latest ingested version per `id`. Query with `FINAL` to resolve repeats before background merges finish. Changes to an ordinary view can use retained fields without re-fetching the source. A materialized view needs a backfill to update stored results; fields you did not retain require a source re-fetch. See [Destinations and transformations](/api-sync/destinations/). ## Progress and checkpoints @@ -168,7 +168,7 @@ chkit ingest run --backfill jan --from 2026-01-01 --to 2026-02-01 Every stream also carries the derived tags `pipeline:` and `stream:`. `schedule:` is a convention only: chkit never interprets it. A `--tag` filter that matches nothing fails before any work runs. -A backfill uses its own checkpoint namespace, so it never moves the scheduled bookmark. Reusing its ID reuses that state, but resumption depends on the strategy: explicit timestamp bounds take precedence over the watermark and reread that range. Full-sync and cursor strategies do not interpret date bounds. See [Backfill source data](/ingestion/operations/#backfill-source-data). +A backfill uses its own checkpoint namespace, so it never moves the scheduled bookmark. Reusing its ID reuses that state, but resumption depends on the strategy: explicit timestamp bounds take precedence over the watermark and reread that range. Full-sync and cursor strategies do not interpret date bounds. See [Backfill source data](/api-sync/operations/#backfill-source-data). `chkit check` verifies that every stream destination carries the ingestion metadata columns. @@ -192,6 +192,6 @@ Run at most one ingestion process per project and target at a time. Use your sch ## Related pages -- [Destinations and transformations](/ingestion/destinations/): raw or shaped storage and where to map fields. -- [Loading and batching](/ingestion/loading/): loader choices, insert sizing, and concurrency defaults. -- [Scheduling and recovery](/ingestion/operations/): retry defaults, execution limits, and troubleshooting. +- [Destinations and transformations](/api-sync/destinations/): raw or shaped storage and where to map fields. +- [Loading and batching](/api-sync/loading/): loader choices, insert sizing, and concurrency defaults. +- [Scheduling and recovery](/api-sync/operations/): retry defaults, execution limits, and troubleshooting. diff --git a/apps/docs/src/content/docs/plugins/overview.mdx b/apps/docs/src/content/docs/plugins/overview.mdx index 57f927b3..107db4a4 100644 --- a/apps/docs/src/content/docs/plugins/overview.mdx +++ b/apps/docs/src/content/docs/plugins/overview.mdx @@ -7,7 +7,7 @@ sidebar: import { Tabs, TabItem } from '@astrojs/starlight/components'; -Add plugins for code generation, schema imports, API ingestion, or SQL backfills. Install TypeScript plugins as npm packages. Python plugins ship inside `chkit-py`; ingestion requires TypeScript. Register plugins in your config: +Add plugins for code generation, schema imports, API sync, or SQL backfills. Install TypeScript plugins as npm packages. Python plugins ship inside `chkit-py`; API sync requires TypeScript. Register plugins in your config: @@ -62,4 +62,4 @@ For `@chkit/plugin-obsessiondb` (Python: `chkit_plugin_obsessiondb`), see the [O - [`@chkit/plugin-codegen`](/plugins/codegen/): TypeScript row types and optional Zod schemas (Python: Pydantic models), generated from your schema files. - [`@chkit/plugin-pull`](/plugins/pull/): introspect a live ClickHouse database into local schema files. Useful for adopting chkit on an existing database. Built into the Python CLI as `chkit pull`. - [`@chkit/plugin-backfill`](/plugins/backfill/): time-windowed data backfill with checkpoints, for materialized views and historical data loads. -- [`@chkit/plugin-ingest`](/ingestion/): API ingestion with an external scheduler with journaled checkpoints (TypeScript only). The dedicated guides cover source readers, storage, checkpoints, loading, and an [installable skill](/ingestion/agent-skill/). +- [`@chkit/plugin-ingest`](/api-sync/): API sync with an external scheduler with journaled checkpoints (TypeScript only). The dedicated guides cover source readers, storage, checkpoints, loading, and an [installable skill](/api-sync/agent-skill/). diff --git a/apps/docs/src/content/docs/python/overview.md b/apps/docs/src/content/docs/python/overview.md index e8e87188..4278a565 100644 --- a/apps/docs/src/content/docs/python/overview.md +++ b/apps/docs/src/content/docs/python/overview.md @@ -49,7 +49,7 @@ The schema/migration CLI and backfill engine share the TypeScript workflow. The - `chkit skills` proxy and the `create-chkit` scaffolder: use `chkit init` instead. - `deps.ts`-style dependency auto-install: install packages explicitly with `pip`. -- [`@chkit/plugin-ingest`](/ingestion/) and the project `entry` module: ingestion source authoring requires TypeScript. +- [`@chkit/plugin-ingest`](/api-sync/) and the project `entry` module: API sync source authoring requires TypeScript. ## These pages diff --git a/apps/docs/src/integrations/raw-markdown.ts b/apps/docs/src/integrations/raw-markdown.ts index 2b35fad7..d32cd7dc 100644 --- a/apps/docs/src/integrations/raw-markdown.ts +++ b/apps/docs/src/integrations/raw-markdown.ts @@ -4,7 +4,7 @@ import { fileURLToPath } from 'node:url'; import type { AstroIntegration } from 'astro'; const BASE_URL = 'https://chkit.obsessiondb.com'; -const SITE_TAGLINE = 'ClickHouse schemas, migrations, and API ingestion in code. Schema workflows in TypeScript and Python; ingestion in TypeScript.'; +const SITE_TAGLINE = 'ClickHouse schemas, migrations, and API sync in code. Schema workflows in TypeScript and Python; API sync in TypeScript.'; interface DocEntry { slug: string; diff --git a/packages/cli/README.md b/packages/cli/README.md index ea6e0d49..b5c2599c 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -4,9 +4,9 @@ chkit is an open-source CLI for ClickHouse. Review migration SQL before applying it. Keep table definitions and TypeScript API readers in your repository, alongside the code that uses them. Run the CLI from the terminal or CI. -Use [chkit-py](https://pypi.org/project/chkit-py/) for schema and migration workflows in Python. API ingestion requires TypeScript. +Use [chkit-py](https://pypi.org/project/chkit-py/) for schema and migration workflows in Python. API sync requires TypeScript. -[See the six-step walkthrough](https://chkit.obsessiondb.com/#from-a-schema-to-a-working-data-sync) · [Get started](https://chkit.obsessiondb.com/getting-started/) · [Build a source](https://chkit.obsessiondb.com/ingestion/quickstart/) +[See the six-step walkthrough](https://chkit.obsessiondb.com/#from-a-schema-to-a-working-data-sync) · [Get started](https://chkit.obsessiondb.com/getting-started/) · [Build a source](https://chkit.obsessiondb.com/api-sync/quickstart/) ## Features @@ -17,7 +17,7 @@ Use [chkit-py](https://pypi.org/project/chkit-py/) for schema and migration work - **CI gate** -- Run `chkit check` to fail builds on pending migrations, checksum mismatches, or schema drift - **TypeScript codegen** -- Generate row types and optional Zod schemas from your schema definitions (`@chkit/plugin-codegen`) - **Schema pulling** -- Introspect an existing ClickHouse database into local schema files (`@chkit/plugin-pull`) -- **API ingestion** -- Define source readers against schema-managed tables, with batching, request retries, and journaled incremental checkpoints (`@chkit/plugin-ingest`) +- **API sync** -- Define source readers against schema-managed tables, with batching, request retries, and journaled incremental checkpoints (`@chkit/plugin-ingest`) - **Data backfill** -- Time-windowed, checkpointed backfill operations with retry logic (`@chkit/plugin-backfill`) - **JSON output** -- Every command supports `--json` for scripting and automation @@ -53,7 +53,7 @@ bunx chkit check All commands support `--json` for machine-readable output and `--config ` to specify a custom config file. -## Add data ingestion +## Add API sync Install `@chkit/plugin-ingest` at the same version as the CLI and core, register `ingest()` in the config, and export tables and pipelines from a TypeScript entry file. A stream pairs a destination table with an async reader. Keep raw objects for flexible SQL transformations, or map records into known columns. @@ -63,7 +63,7 @@ bunx chkit ingest run --tag pipeline:content bunx chkit ingest status ``` -Runs are finite; use an external scheduler and serialize processes per target. The [ingestion quickstart](https://chkit.obsessiondb.com/ingestion/quickstart/) includes a complete public-API example and config. +Runs are finite; use an external scheduler and serialize processes per target. The [API sync quickstart](https://chkit.obsessiondb.com/api-sync/quickstart/) includes a complete public-API example and config. ## Requirements @@ -76,7 +76,7 @@ Runs are finite; use an external scheduler and serialize processes per target. T |--------|-------------| | [`@chkit/plugin-codegen`](https://www.npmjs.com/package/@chkit/plugin-codegen) | Generate TypeScript row types and Zod schemas | | [`@chkit/plugin-pull`](https://www.npmjs.com/package/@chkit/plugin-pull) | Pull schemas from a live ClickHouse instance | -| [`@chkit/plugin-ingest`](https://www.npmjs.com/package/@chkit/plugin-ingest) | API ingestion with batching, retries, and journaled checkpoints | +| [`@chkit/plugin-ingest`](https://www.npmjs.com/package/@chkit/plugin-ingest) | API sync with batching, retries, and journaled checkpoints | | [`@chkit/plugin-backfill`](https://www.npmjs.com/package/@chkit/plugin-backfill) | Time-windowed data backfill with checkpoints | | [`@chkit/plugin-obsessiondb`](https://www.npmjs.com/package/@chkit/plugin-obsessiondb) | Auto-rewrite Shared engines for ObsessionDB compatibility | diff --git a/packages/plugin-ingest/README.md b/packages/plugin-ingest/README.md index fe3b92ab..702bbb81 100644 --- a/packages/plugin-ingest/README.md +++ b/packages/plugin-ingest/README.md @@ -19,9 +19,9 @@ export default defineConfig({ chkit ingest run --tag schedule:1h ``` -Documentation: https://chkit.obsessiondb.com/ingestion/ +Documentation: https://chkit.obsessiondb.com/api-sync/ -For raw storage, typed rows, and where to map fields, see [Destinations and transformations](https://chkit.obsessiondb.com/ingestion/destinations/). +For raw storage, typed rows, and where to map fields, see [Destinations and transformations](https://chkit.obsessiondb.com/api-sync/destinations/). Install the source-authoring skill for a coding agent: diff --git a/skills/chkit-ingestion/SKILL.md b/skills/chkit-ingestion/SKILL.md index dae4c6d6..5c000f7a 100644 --- a/skills/chkit-ingestion/SKILL.md +++ b/skills/chkit-ingestion/SKILL.md @@ -17,15 +17,15 @@ Start with the default loader and a full sync for small datasets; add provider-s ## Read only the relevant docs -Use local `apps/docs/src/content/docs/ingestion/` or these URLs. Match the installed version; inspect exported types if docs are unavailable. - -- [Quickstart](https://chkit.obsessiondb.com/ingestion/quickstart.md): complete config and first sync. -- [Readers](https://chkit.obsessiondb.com/ingestion/readers.md): fetching, auth, pagination, SDKs. -- [Destinations](https://chkit.obsessiondb.com/ingestion/destinations.md): raw or shaped rows, relationships, current state/history, tombstones. -- [Incremental syncs](https://chkit.obsessiondb.com/ingestion/incremental-syncs.md): full, timestamp, cursor, custom strategies. -- [Loading](https://chkit.obsessiondb.com/ingestion/loading.md): batching and custom loader contracts. -- [Operations](https://chkit.obsessiondb.com/ingestion/operations.md): retries, scheduling, backfills, limits. -- [Testing](https://chkit.obsessiondb.com/ingestion/testing.md): offline failure and replay checks. +Use local `apps/docs/src/content/docs/api-sync/` or these URLs. Match the installed version; inspect exported types if docs are unavailable. + +- [Quickstart](https://chkit.obsessiondb.com/api-sync/quickstart.md): complete config and first sync. +- [Readers](https://chkit.obsessiondb.com/api-sync/readers.md): fetching, auth, pagination, SDKs. +- [Destinations](https://chkit.obsessiondb.com/api-sync/destinations.md): raw or shaped rows, relationships, current state/history, tombstones. +- [Incremental syncs](https://chkit.obsessiondb.com/api-sync/incremental-syncs.md): full, timestamp, cursor, custom strategies. +- [Loading](https://chkit.obsessiondb.com/api-sync/loading.md): batching and custom loader contracts. +- [Operations](https://chkit.obsessiondb.com/api-sync/operations.md): retries, scheduling, backfills, limits. +- [Testing](https://chkit.obsessiondb.com/api-sync/testing.md): offline failure and replay checks. ## Preserve these contracts diff --git a/skills/chkit/SKILL.md b/skills/chkit/SKILL.md index 1b8c0b34..32610279 100644 --- a/skills/chkit/SKILL.md +++ b/skills/chkit/SKILL.md @@ -10,7 +10,7 @@ chkit lets you define ClickHouse schemas in TypeScript, generate migration SQL, Docs: https://chkit.obsessiondb.com -For application API source authoring with `@chkit/plugin-ingest`, use the separate `chkit-ingestion` skill (`npx skills add obsessiondb/chkit --skill chkit-ingestion`) and the [ingestion guides](https://chkit.obsessiondb.com/ingestion.md). Its streams and checkpoints are separate from plugin-codegen's generated insert helpers. +For application API source authoring with `@chkit/plugin-ingest`, use the separate `chkit-ingestion` skill (`npx skills add obsessiondb/chkit --skill chkit-ingestion`) and the [API sync guides](https://chkit.obsessiondb.com/api-sync.md). Its streams and checkpoints are separate from plugin-codegen's generated insert helpers. ## Configuration