🐛 Journal run facts per run instead of in one shared @run namespace

.changeset/ingest-run-namespace.md

@@ -0,0 +1,5 @@
+---
+"@chkit/plugin-ingest": patch
+---
+
+Each ingestion run now journals its `run_started` and `run_finished` facts under its own `@run:<run id>` namespace. Previously every run shared one `@run` namespace, so two executor processes that overlapped once (for example a local run during a scheduled one) left conflicting facts at the same sequence number and every later run refused to start. Stream checkpoints were never affected; per-stream namespaces still detect overlap.

README.md

@@ -1,107 +1,196 @@
-<p align="center">
-  <img src="./assets/hero.png" alt="chkit — ClickHouse schema and migrations, as code" width="100%">
-</p>
-
 # chkit
 
-**ClickHouse schema and migration toolkit for TypeScript and Python.**
+**Manage ClickHouse schemas and sync API data.**
 
 [![npm version](https://img.shields.io/npm/v/chkit?label=npm)](https://www.npmjs.com/package/chkit)
 [![CI](https://github.com/obsessiondb/chkit/actions/workflows/ci.yml/badge.svg)](https://github.com/obsessiondb/chkit/actions/workflows/ci.yml)
 [![Docs](https://img.shields.io/badge/docs-chkit.obsessiondb.com-blue)](https://chkit.obsessiondb.com)
 
-Define your ClickHouse tables, views, materialized views, and dictionaries in TypeScript or Python. chkit diffs your schema, generates migration SQL, applies it safely, and keeps your dev and production databases in sync -- all from the command line.
+chkit is an open-source CLI for ClickHouse. Review migration SQL before applying it. Keep table definitions and API readers in your repository, alongside the code that uses them. Run the CLI from the terminal or CI.
+
+**TypeScript:** schemas, migrations, and ingestion. **Python:** schemas and migrations through [chkit-py](https://chkit.obsessiondb.com/python/overview/).
+
+[Get started](https://chkit.obsessiondb.com/getting-started/) · [Build a data source](https://chkit.obsessiondb.com/ingestion/quickstart/) · [Documentation](https://chkit.obsessiondb.com)
+
+> **Beta:** the public API is still evolving. Keep the CLI, core, and plugins on matching versions.
 
-> **Status: beta.** chkit powers production workloads and the CLI surface and schema DSL are stable. We may still make small breaking changes to UX and internal APIs before 1.0.
+## Why chkit
 
-## Key Features
+- **Define tables and views.** Keep ClickHouse tables, views, materialized views, and dictionaries in code. Import the schema from an existing database.
+- **Migrate and backfill data.** Handle complex schema changes and backfill data across materialized views. Track progress and resume interrupted runs.
+- **Sync data from any source.** Build reliable syncs from HTTP APIs, databases, and other sources into ClickHouse.
+- **Store raw or mapped records.** Retain raw objects for SQL transformations, or map records into established entity tables before loading.
+- **Check schema drift in CI.** Detect pending migrations, checksum mismatches, and live schema differences with `chkit check`.
+- **Generate types and backfill data.** Plugins generate TypeScript types and Zod schemas or run SQL backfills. Agent skills guide schema and source authoring.
+
+## From a schema to a working sync
+
+Start with a table, load 100 demo API records, then query posts per author. Change the view later using the data already stored in ClickHouse. This walkthrough uses TypeScript and the ingestion plugin. For a project using only `chkit` and `@chkit/core`, follow the [schema tutorial](https://chkit.obsessiondb.com/tutorials/first-schema/).
+
+```sh
+bun add -d chkit@beta @chkit/core@beta @chkit/plugin-ingest@beta
+```
 
-- **TypeScript-native schema definitions** -- tables, views, materialized views
-- **Automatic migration generation** -- diff-based SQL from your schema changes
-- **Safe migration execution** -- preview first, destructive-operation blocking
-- **Schema drift detection** -- compare live database to expected state
-- **CI gate command** -- `chkit check` fails your build on pending migrations or drift
-- **TypeScript codegen** -- row types and optional Zod schemas from your schema
-- **Plugin system** -- pull, codegen, backfill, or write your own
-- **JSON output mode** -- `--json` on every command for scripting
+<details>
+<summary>Project configuration</summary>
 
-## Quick Start
+Create `clickhouse.config.ts` and set the connection environment variables for the intended development database. Export the definitions below from `src/chkit.ts`.
 
-```bash
-bun add -d chkit @chkit/core
-bunx chkit init
+```ts
+import { defineConfig } from '@chkit/core'
+import { ingest } from '@chkit/plugin-ingest'
+
+export default defineConfig({
+  entry: './src/chkit.ts',
+  plugins: [ingest()],
+  clickhouse: {
+    url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123',
+    username: process.env.CLICKHOUSE_USER ?? 'default',
+    password: process.env.CLICKHOUSE_PASSWORD ?? '',
+    database: 'default',
+  },
+})
 ```
 
-Define a table in `src/db/schema/example.ts`:
+The [configuration guide](https://chkit.obsessiondb.com/configuration/overview/) covers existing projects and additional plugins.
+
+</details>
+
+### 1. Define a table
+
+Create `src/chkit.ts`. Explicit columns describe the query shape; `ingestionColumns` adds the metadata used by the reader in step 3.
 
 ```ts
-import { schema, table } from '@chkit/core'
+import { table } from '@chkit/core'
+import { ingestionColumns } from '@chkit/plugin-ingest'
 
-const events = table({
-  database: 'default',
-  name: 'events',
-  engine: 'MergeTree',
+export const posts = table({
+  database: 'default', name: 'posts',
   columns: [
-    { name: 'id', type: 'UInt64' },
-    { name: 'source', type: 'String' },
-    { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' },
+    { name: 'id', type: 'String' },
+    { name: 'title', type: 'String' },
+    { name: 'user_id', type: 'UInt64' },
+    ...ingestionColumns,
   ],
-  primaryKey: ['id'],
-  orderBy: ['id'],
-  partitionBy: 'toYYYYMM(ingested_at)',
+  engine: 'ReplacingMergeTree(_chkit_ingested_at)',
+  primaryKey: ['id'], orderBy: ['id'],
 })
-
-export default schema(events)
 ```
 
-Generate and apply your first migration:
+### 2. Review and apply a migration
+
+```sh
+bunx chkit generate --name create-posts
+bunx chkit migrate
 
-```bash
-bunx chkit generate --name init
+# After reviewing the generated SQL and preview:
 bunx chkit migrate --apply
-bunx chkit status
 ```
 
-## Commands
+### 3. Add a source reader
 
-| Command | Description |
-|---------|-------------|
-| `chkit init` | Scaffold config and example schema |
-| `chkit generate` | Diff schema and generate migration SQL |
-| `chkit migrate` | Preview and apply pending migrations |
-| `chkit status` | Show migration counts and checksum status |
-| `chkit drift` | Compare live database to expected schema |
-| `chkit check` | CI gate: fail on pending/drift/mismatch |
-| `chkit codegen` | Generate TypeScript types from schema |
-| `chkit pull` | Pull existing ClickHouse schema to local files |
+Add this to `src/chkit.ts`. The public demonstration API returns a bounded dataset, so this reader performs a full sync and maps each record into the table.
+
+```ts
+import { definePipeline, defineStream, HttpError } from '@chkit/plugin-ingest'
+
+type Post = { id: number; title: string; userId: number }
+
+const postStream = defineStream({
+  id: 'demo.posts', destination: posts,
+  async *read(context) {
+    const page = await context.attempt(async (signal) => {
+      const response = await fetch('https://jsonplaceholder.typicode.com/posts', { signal })
+      if (!response.ok) throw await HttpError.fromResponse(response)
+      return await response.json() as Post[]
+    })
+    yield { rows: page.map((post) => ({
+      id: String(post.id), title: post.title, user_id: post.userId,
+    })) }
+  },
+})
+
+export const content = definePipeline({ id: 'content', streams: [postStream] })
+```
 
-All commands support `--json` for machine-readable output. See the [full CLI reference](https://chkit.obsessiondb.com/cli/overview/) for details.
+```sh
+bunx chkit ingest run --tag pipeline:content
+```
+
+The default loader writes the rows. For production sources, add [pagination](https://chkit.obsessiondb.com/ingestion/readers/) and [incremental reads](https://chkit.obsessiondb.com/ingestion/incremental-syncs/) when the provider supports them.
 
-## Configuration
+### 4. Query through a view
+
+Add this view to the same entry. `FINAL` reconciles repeated object versions before aggregation.
 
 ```ts
-// clickhouse.config.ts
-import { defineConfig } from '@chkit/core'
-import { pull } from '@chkit/plugin-pull'
-import { codegen } from '@chkit/plugin-codegen'
+import { view } from '@chkit/core'
 
-export default defineConfig({
-  schema: './src/db/schema/**/*.ts',
-  outDir: './chkit',
-  plugins: [
-    pull({ outFile: './src/db/schema/pulled.ts' }),
-    codegen({ outFile: './src/generated/chkit-types.ts' }),
-  ],
-  clickhouse: {
-    url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123',
-    username: process.env.CLICKHOUSE_USER ?? 'default',
-    password: process.env.CLICKHOUSE_PASSWORD ?? '',
-    database: process.env.CLICKHOUSE_DB ?? 'default',
-  },
+export const postsByAuthor = view({
+  database: 'default', name: 'posts_by_author',
+  as: `SELECT user_id, count() AS posts
+       FROM default.posts FINAL
+       GROUP BY user_id`,
+})
+```
+
+```sh
+bunx chkit generate --name posts-by-author
+bunx chkit migrate
+bunx chkit migrate --apply
+bunx chkit query "SELECT * FROM default.posts_by_author ORDER BY user_id LIMIT 3"
+```
+
+Expected result for the demo dataset:
+
+| user_id | posts |
+| --- | --- |
+| 1 | 10 |
+| 2 | 10 |
+| 3 | 10 |
+
+If the final shape may change, [retain raw records and transform in ClickHouse](https://chkit.obsessiondb.com/ingestion/destinations/) instead of mapping every field up front.
+
+### 5. Evolve the model
+
+Replace the previous view definition to add a measure:
+
+```ts
+import { view } from '@chkit/core'
+
+export const postsByAuthor = view({
+  database: 'default', name: 'posts_by_author',
+  as: `SELECT user_id, count() AS posts,
+         countIf(positionCaseInsensitive(title, 'qui') > 0) AS matching_posts
+       FROM default.posts FINAL
+       GROUP BY user_id`,
 })
 ```
 
-See the [configuration docs](https://chkit.obsessiondb.com/configuration/overview/) for all options.
+Generate, preview, and apply another migration. chkit recreates the ordinary view with the new SQL. The `matching_posts` measure uses already-stored rows: no API re-fetch or table backfill for this change.
+
+### 6. Verify and repeat
+
+```sh
+bunx chkit check
+bunx chkit ingest list
+bunx chkit ingest run --tag pipeline:content
+```
+
+Use `check` in CI for migration state and schema drift. Schedule ingestion through cron, CI, or another job runner, with one ingestion process per target. Incremental sources resume from committed state; this full-sync demo reads the dataset again. See [scheduling and recovery](https://chkit.obsessiondb.com/ingestion/operations/).
+
+## Set up chkit for your project
+
+| Goal | Start here |
+|---|---|
+| Manage a new schema | [Getting started](https://chkit.obsessiondb.com/getting-started/) |
+| Adopt an existing database | [Pull a live schema](https://chkit.obsessiondb.com/plugins/pull/) |
+| Implement an API source | [Ingestion quickstart](https://chkit.obsessiondb.com/ingestion/quickstart/) |
+| Generate application types | [TypeScript codegen](https://chkit.obsessiondb.com/plugins/codegen/) |
+| Recompute stored data | [SQL backfills](https://chkit.obsessiondb.com/plugins/backfill/) |
+| Work with a coding agent | [Agent skills](https://chkit.obsessiondb.com/ai-agents/) |
+
+See the [CLI reference](https://chkit.obsessiondb.com/cli/overview/) for commands, flags, and JSON output.
 
 ## Packages
 
@@ -114,31 +203,31 @@ See the [configuration docs](https://chkit.obsessiondb.com/configuration/overvie
 | [`@chkit/plugin-pull`](packages/plugin-pull) | Pull live schema into local files |
 | [`@chkit/plugin-codegen`](packages/plugin-codegen) | Codegen plugin for the CLI |
 | [`@chkit/plugin-backfill`](packages/plugin-backfill) | Backfill plugin for data migrations |
-| [`@chkit/plugin-ingest`](packages/plugin-ingest) | Ingestion plugin: scheduled API pulls with journaled checkpoints |
+| [`@chkit/plugin-ingest`](packages/plugin-ingest) | API source readers, batching, retries, and journaled checkpoints |
 | [`@chkit/plugin-obsessiondb`](packages/plugin-obsessiondb) | ObsessionDB integration: auto-rewrite `Shared` engines for ClickHouse targets |
 
 ## Python
 
-chkit is also available for Python as [`chkit-py`](https://pypi.org/project/chkit-py/) (`pip install chkit-py`) — same CLI, same schema semantics, with config and schema files written as `.py`. The port lives in [`chkit_python/`](chkit_python).
+Install [`chkit-py`](https://pypi.org/project/chkit-py/) (`pip install chkit-py`) to define schemas and run migrations, drift detection, and CI checks with Python config and schema files. API ingestion requires TypeScript. The Python source is in [`chkit_python/`](chkit_python).
 
 ## Documentation
 
-Full documentation is available at **[chkit.obsessiondb.com](https://chkit.obsessiondb.com)**.
+Read the documentation at **[chkit.obsessiondb.com](https://chkit.obsessiondb.com)**.
 
 ## ObsessionDB
 
-chkit is built by the team behind [**ObsessionDB**](https://obsessiondb.com), a fully-managed ClickHouse database. ObsessionDB is the recommended way to run chkit in production:
+The [**ObsessionDB**](https://obsessiondb.com) team builds chkit and provides a managed ClickHouse service:
 
-- **First-party integration.** The [`@chkit/plugin-obsessiondb`](packages/plugin-obsessiondb) plugin auto-detects ObsessionDB and keeps your schema on `SharedReplacingMergeTree` / `SharedMergeTree` for managed replication — same TypeScript schema as your local ClickHouse, no manual switching.
-- **No ops.** No Keeper, no replica tuning, no manual scaling.
-- **Tested alongside chkit.** Every chkit release runs its E2E suite against ObsessionDB.
+- **Engine configuration.** Use the [`@chkit/plugin-obsessiondb`](packages/plugin-obsessiondb) plugin to select `SharedReplacingMergeTree` / `SharedMergeTree` for managed replication while keeping the same TypeScript schema for your development database.
+- **Managed infrastructure.** ObsessionDB manages Keeper, replicas, and scaling.
+- **Release tests.** The chkit release pipeline runs its E2E suite against ObsessionDB.
 
 [Try ObsessionDB →](https://obsessiondb.com)
 
 ## Community
 
-- [@ObsessionDB on X](https://x.com/ObsessionDB) — release notes and updates
-- [GitHub Issues](https://github.com/obsessiondb/chkit/issues) — bugs and feature requests
+- [@ObsessionDB on X](https://x.com/ObsessionDB): release notes and updates
+- [GitHub Issues](https://github.com/obsessiondb/chkit/issues): bugs and feature requests
 
 ## Contributing
 

apps/docs/astro.config.mjs

@@ -17,7 +17,7 @@ export default defineConfig({
 	integrations: [
 		starlight({
 			title: 'chkit Docs',
-			description: 'Public documentation for chkit, the ClickHouse schema and migration CLI.',
+			description: 'Define ClickHouse schemas, review migrations, and sync API data into your tables with chkit.',
 			customCss: ['./src/styles/custom.css'],
 			// Blog lives at /blog. `navigation: 'none'` so the plugin doesn't
 			// override SiteTitle/ThemeSelect (we already override both) — we add
@@ -74,6 +74,32 @@ export default defineConfig({
 					label: 'Schema',
 					autogenerate: { directory: 'schema' },
 				},
+				{
+					label: 'Ingestion',
+					items: [
+						{ label: 'Overview', slug: 'ingestion' },
+						{ label: 'Quickstart', slug: 'ingestion/quickstart' },
+						{ label: 'Authoring skill', slug: 'ingestion/agent-skill' },
+						{
+							label: 'Build a source',
+							collapsed: true,
+							items: [
+								{ slug: 'ingestion/readers' },
+								{ slug: 'ingestion/destinations' },
+								{ slug: 'ingestion/incremental-syncs' },
+								{ slug: 'ingestion/loading' },
+							],
+						},
+						{
+							label: 'Run and verify',
+							collapsed: true,
+							items: [
+								{ slug: 'ingestion/operations' },
+								{ slug: 'ingestion/testing' },
+							],
+						},
+					],
+				},
 				{
 					label: 'ObsessionDB',
 					autogenerate: { directory: 'obsessiondb' },

apps/docs/src/components/Footer.astro

@@ -18,7 +18,7 @@ const repo = 'https://github.com/obsessiondb/chkit';
 	<div class="chk-site-footer-inner">
 		<div class="chk-footer-brand">
 			<a href="/" class="chk-wordmark chk-footer-wordmark"><span translate="no">ch-kit</span></a>
-			<p class="chk-footer-tagline">Headless O(A)RM for ClickHouse — TypeScript & Python!</p>
+			<p class="chk-footer-tagline">Manage ClickHouse schemas and sync API data.</p>
 			<div class="chk-footer-social">
 				<a href={repo} target="_blank" rel="noopener noreferrer" aria-label="GitHub">
 					<svg viewBox="0 0 16 16" fill="currentColor" aria-hidden="true">
@@ -41,6 +41,7 @@ const repo = 'https://github.com/obsessiondb/chkit';
 					<li><a href="/cli/overview/">CLI reference</a></li>
 					<li><a href="/configuration/overview/">Configuration</a></li>
 					<li><a href="/schema/dsl-reference/">Schema</a></li>
+					<li><a href="/ingestion/">Ingestion</a></li>
 				</ul>
 			</div>
 			<div class="chk-footer-col">