From d4e351ac189f5855bc7eefbbd6f666aa886186b0 Mon Sep 17 00:00:00 2001 From: KeKs0r Date: Thu, 24 Sep 2026 12:50:31 -0700 Subject: [PATCH] =?UTF-8?q?=F0=9F=93=9D=20Add=20chkit=20ingest=20to=20the?= =?UTF-8?q?=20CLI=20reference?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vex-Session: session-ad7ddb262a78682dcf732e38 --- apps/docs/src/content/docs/cli/ingest.md | 200 +++++++++++++++++++ apps/docs/src/content/docs/cli/overview.mdx | 2 +- apps/docs/src/content/docs/cli/plugin.mdx | 2 +- apps/docs/src/content/docs/plugins/ingest.md | 3 +- 4 files changed, 204 insertions(+), 3 deletions(-) create mode 100644 apps/docs/src/content/docs/cli/ingest.md diff --git a/apps/docs/src/content/docs/cli/ingest.md b/apps/docs/src/content/docs/cli/ingest.md new file mode 100644 index 00000000..3b6f4145 --- /dev/null +++ b/apps/docs/src/content/docs/cli/ingest.md @@ -0,0 +1,200 @@ +--- +title: "chkit ingest" +description: "Run, list, or inspect the ingestion streams exported by your project entry." +sidebar: + order: 11 +--- + +Runs the TypeScript ingestion streams provided by [`@chkit/plugin-ingest`](/plugins/ingest/), lists them, or shows their committed checkpoints. + +## Synopsis + +``` +chkit ingest run [flags] +chkit ingest list [flags] +chkit ingest status [flags] +``` + +`chkit ingest ` is equivalent to `chkit plugin ingest `. + +## Flags + +### Selection (all subcommands) + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--tag ` | string[] | — | Exact tag every selected stream must carry. Repeat for AND matching | + +### `run` only + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--backfill ` | string | — | Stable backfill ID. Runs in an isolated checkpoint namespace | +| `--from ` | string | — | Backfill range lower bound (ISO timestamp). Requires `--backfill` | +| `--to ` | string | — | Backfill range upper bound (ISO timestamp). Requires `--backfill` | +| `--max-duration ` | string | `3600` | Execution budget. Overrides the plugin's `maxDurationSeconds` option | + +Global flags documented on [CLI Overview](/cli/overview/#global-flags). + +## Behavior + +### Prerequisites + +The ingest plugin must be registered in `plugins` and the config must set `entry` to a module that exports at least one `definePipeline(...)` value. Without an exported pipeline, every subcommand fails with exit code 2. + +`run` and `status` require a direct `clickhouse` connection in the config. Host-provided executors, including the ObsessionDB workbench executor, are rejected. `list` reads only the definition graph and needs no connection. + +### Stream selection + +Without `--tag`, a subcommand selects every stream in the loaded graph. Each `--tag` narrows the selection: a stream is selected only when it carries every requested tag. Every stream carries its declared pipeline and stream tags plus the derived tags `pipeline:` and `stream:`. + +A `--tag` filter that matches no stream fails with exit code 2 before any work runs. + +### `run` + +Executes the selected streams from their committed checkpoints and records progress in the ingestion journal (`_chkit_ingestion_journal` by default). Progress advances only after writes succeed; a later run resumes from the last committed checkpoint. + +The run ends when every selected stream finishes, the execution budget is exhausted, or the process receives `SIGINT` or `SIGTERM`. A budget-exhausted or interrupted run keeps committed progress and exits with code 1. Each stream reports one outcome: `succeeded`, `failed`, `budget_exhausted`, or `cancelled`. + +Run at most one ingestion process per project and target at a time. See [Scheduling and recovery](/ingestion/operations/). + +### Backfills + +`--backfill ` runs the selected streams under the checkpoint namespace `#backfill:`, so it never moves the scheduled checkpoint. Reusing an ID resumes that backfill's state. The ID must start with a letter or digit and contain only letters, digits, `_`, `.`, and `-`. + +`--from` and `--to` set explicit bounds for `timestampWindow` streams and take precedence over the backfill's watermark. Full-sync and cursor strategies ignore them. Passing `--from` or `--to` without `--backfill` fails with exit code 2. + +### `list` + +Prints each selected stream with its destination table and effective tags. + +### `status` + +Reads the journal and prints the committed checkpoint of each selected stream's scheduled namespace. Streams that have never committed progress show `(no checkpoint)`. + +## Examples + +**Run everything on an hourly schedule:** + +```sh +chkit ingest run --tag schedule:1h +``` + +**Run a single stream:** + +```sh +chkit ingest run --tag stream:helpdesk.tickets +``` + +**Backfill January without moving the scheduled checkpoint:** + +```sh +chkit ingest run --backfill jan --from 2026-01-01 --to 2026-02-01 +``` + +**Cap a CI run at ten minutes:** + +```sh +chkit ingest run --tag pipeline:helpdesk --max-duration 600 +``` + +**Inspect progress as JSON:** + +```sh +chkit ingest status --tag pipeline:helpdesk --json +``` + +## Exit codes + +| Code | Meaning | +|------|---------| +| 0 | Success. For `run`, every selected stream succeeded | +| 1 | Error, or a `run` with any stream that did not succeed | +| 2 | Configuration error: no exported pipeline, no matching stream, invalid flags, or missing direct connection | + +## JSON output + +**`run`:** + +```json +{ + "command": "run", + "runId": "3f0c9a7e-2b1d-4c55-9e0a-6d8b1f2a4c11", + "cutoff": "2026-09-24T10:00:00.000Z", + "ok": true, + "streams": [ + { + "streamId": "helpdesk.tickets", + "pipelineId": "helpdesk", + "namespaceId": "helpdesk.tickets", + "outcome": "succeeded", + "rows": 1240, + "batches": 3, + "chunks": 13, + "checkpointVersion": 42 + } + ] +} +``` + +A failed stream sets `outcome` to `failed` and includes an `error` message; the top-level `ok` is then `false`. + +**`list`:** + +```json +{ + "ok": true, + "command": "list", + "streams": [ + { + "streamId": "helpdesk.tickets", + "pipelineId": "helpdesk", + "destination": "crm.tickets", + "strategy": "chkit.timestamp_window@1", + "tags": ["schedule:1h", "pipeline:helpdesk", "stream:helpdesk.tickets"] + } + ] +} +``` + +**`status`:** + +```json +{ + "ok": true, + "command": "status", + "targetId": "clickhouse.example.com:8443/crm", + "streams": [ + { + "streamId": "helpdesk.tickets", + "pipelineId": "helpdesk", + "destination": "crm.tickets", + "strategy": "chkit.timestamp_window@1", + "tags": ["schedule:1h", "pipeline:helpdesk", "stream:helpdesk.tickets"], + "checkpointVersion": 42, + "checkpoint": { + "strategy": "chkit.timestamp_window", + "version": 1, + "state": { "watermark": "2026-09-24T09:00:00.000Z" } + } + } + ] +} +``` + +**Error:** + +```json +{ + "ok": false, + "command": "run", + "error": "No stream matches every requested tag: --tag schedule:15m. Nothing was executed." +} +``` + +## Related + +- [Ingest plugin reference](/plugins/ingest/) — plugin setup, options, and stream definitions +- [Ingestion quickstart](/ingestion/quickstart/) — define a first stream and run it +- [Scheduling and recovery](/ingestion/operations/) — tags, retries, budgets, and backfills +- [`chkit check`](/cli/check/) — verifies that stream destinations carry the ingestion metadata columns diff --git a/apps/docs/src/content/docs/cli/overview.mdx b/apps/docs/src/content/docs/cli/overview.mdx index c043387d..1fb756c3 100644 --- a/apps/docs/src/content/docs/cli/overview.mdx +++ b/apps/docs/src/content/docs/cli/overview.mdx @@ -24,7 +24,7 @@ Use the `chkit` CLI to define ClickHouse schemas in TypeScript or Python, review | [`chkit pull`](/cli/pull/) | Introspect live ClickHouse and generate a schema file | | [`chkit codegen`](/cli/codegen/) | Generate typed row models from schema definitions | | [`chkit plugin`](/cli/plugin/) | List or run plugin commands | -| [`chkit ingest`](/plugins/ingest/#commands) | Run, list, or inspect API ingestion streams (`@chkit/plugin-ingest`, TypeScript only) | +| [`chkit ingest`](/cli/ingest/) | Run, list, or inspect API ingestion streams (`@chkit/plugin-ingest`, TypeScript only) | ## Connection requirements diff --git a/apps/docs/src/content/docs/cli/plugin.mdx b/apps/docs/src/content/docs/cli/plugin.mdx index 435b8882..b5ed53ac 100644 --- a/apps/docs/src/content/docs/cli/plugin.mdx +++ b/apps/docs/src/content/docs/cli/plugin.mdx @@ -2,7 +2,7 @@ title: "chkit plugin" description: "List installed plugins or run plugin commands." sidebar: - order: 11 + order: 12 --- import { Tabs, TabItem } from '@astrojs/starlight/components'; diff --git a/apps/docs/src/content/docs/plugins/ingest.md b/apps/docs/src/content/docs/plugins/ingest.md index 0637f891..ebfad8ae 100644 --- a/apps/docs/src/content/docs/plugins/ingest.md +++ b/apps/docs/src/content/docs/plugins/ingest.md @@ -170,7 +170,7 @@ Every stream also carries the derived tags `pipeline:` and `stream:`. `s A backfill uses its own checkpoint namespace, so it never moves the scheduled bookmark. Reusing its ID reuses that state, but resumption depends on the strategy: explicit timestamp bounds take precedence over the watermark and reread that range. Full-sync and cursor strategies do not interpret date bounds. See [Backfill source data](/ingestion/operations/#backfill-source-data). -`chkit check` verifies that every stream destination carries the ingestion metadata columns. +`chkit check` verifies that every stream destination carries the ingestion metadata columns. See [`chkit ingest`](/cli/ingest/) for the full flag reference, exit codes, and JSON output. ## Delivery guarantee @@ -192,6 +192,7 @@ Run at most one ingestion process per project and target at a time. Use your sch ## Related pages +- [`chkit ingest`](/cli/ingest/): command flags, exit codes, and JSON output. - [Destinations and transformations](/ingestion/destinations/): raw or shaped storage and where to map fields. - [Loading and batching](/ingestion/loading/): loader choices, insert sizing, and concurrency defaults. - [Scheduling and recovery](/ingestion/operations/): retry defaults, execution limits, and troubleshooting.