-
Notifications
You must be signed in to change notification settings - Fork 33
feat: add ClickHouse warehouse driver #574
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e63d6c6
2faf365
6c48729
46ff94c
1aa0ece
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| --- | ||
| description: Add a new database driver to Altimate Code. Scaffolds the driver, registers it across all 23 integration points, writes E2E tests, and updates docs. Usage - /add-database-driver <database-name> | ||
| --- | ||
|
|
||
| # Add Database Driver | ||
|
|
||
| Scaffold and fully integrate a new database/warehouse driver into Altimate Code. This command handles all 23 integration points — driver code, registry, discovery, finops, tests, and documentation. | ||
|
|
||
| ## Input | ||
|
|
||
| `$ARGUMENTS` = the database name (e.g., `cockroachdb`, `timescaledb`, `cassandra`, `neo4j`). | ||
|
|
||
| If empty, ask: "Which database should I add support for?" | ||
|
|
||
| ## Step 0: Research | ||
|
|
||
| Before writing any code, research the database: | ||
|
|
||
| 1. **Find the official Node.js/TypeScript client package** on npm. Search for `@{database}/client`, `{database}-js`, or similar. | ||
| 2. **Check supported server versions** — which versions are not EOL? | ||
| 3. **Identify auth methods** — password, token, TLS/certificate, connection string, cloud-specific? | ||
| 4. **Check SQL dialect** — standard SQL? Custom syntax? LIMIT vs TOP vs FETCH FIRST? System tables for schemas/tables/columns? | ||
| 5. **Find Docker image** — official image on Docker Hub for E2E testing? | ||
| 6. **Check if dbt adapter exists** — search for `dbt-{database}` on PyPI. | ||
|
|
||
| Present findings to the user before proceeding: | ||
| ``` | ||
| ## Research: {Database} | ||
|
|
||
| - **npm package**: `{package}` (v{version}) | ||
| - **Server versions**: {non-EOL versions} | ||
| - **Auth methods**: {list} | ||
| - **SQL dialect**: {notes on LIMIT, system tables, parameterized queries} | ||
| - **Docker image**: `{image}:{tag}` | ||
| - **dbt adapter**: {exists/not found} | ||
|
|
||
| Proceed with implementation? | ||
| ``` | ||
|
|
||
| ## Step 1: Read Reference Document | ||
|
|
||
| Read the comprehensive checklist: | ||
| ```bash | ||
| cat packages/drivers/ADDING_A_DRIVER.md | ||
| ``` | ||
|
|
||
| This document has all 23 integration points with exact file paths and code patterns. | ||
|
|
||
| ## Step 2: Read Existing Driver for Pattern | ||
|
|
||
| Read a similar existing driver as a template. Choose based on database type: | ||
|
|
||
| - **SQL database with password auth** → read `packages/drivers/src/mysql.ts` | ||
| - **Cloud warehouse with token auth** → read `packages/drivers/src/databricks.ts` | ||
| - **Database with connection string support** → read `packages/drivers/src/postgres.ts` | ||
| - **HTTP-based client** → read `packages/drivers/src/clickhouse.ts` | ||
| - **Document database (non-SQL)** → read `packages/drivers/src/mongodb.ts` | ||
|
|
||
| Also read: | ||
| - `packages/drivers/src/normalize.ts` — for alias pattern | ||
| - `packages/opencode/src/altimate/native/connections/registry.ts` — for registration pattern | ||
| - `packages/opencode/test/altimate/drivers-docker-e2e.test.ts` — for E2E test pattern | ||
|
|
||
| ## Step 3: Implement (23 integration points) | ||
|
|
||
| Work through all 9 phases from the checklist. Use parallel edits where possible. | ||
|
|
||
| ### Phase 1: Core Driver (4 files) | ||
|
|
||
| 1. **Create `packages/drivers/src/{database}.ts`** | ||
| - Follow the Connector interface: `connect()`, `execute()`, `listSchemas()`, `listTables()`, `describeTable()`, `close()` | ||
| - Lazy-import the npm package | ||
| - Use parameterized queries for schema introspection | ||
| - Handle LIMIT injection with DML guard: `!hasDML` check before appending LIMIT | ||
| - Handle TLS detection from connection strings | ||
|
|
||
| 2. **Add export to `packages/drivers/src/index.ts`** | ||
|
|
||
| 3. **Add optionalDependency to `packages/drivers/package.json`** | ||
|
|
||
| 4. **Add aliases to `packages/drivers/src/normalize.ts`** | ||
|
|
||
| ### Phase 2: Registry (4 files in registry.ts) | ||
|
|
||
| 5. Add to `DRIVER_MAP` | ||
| 6. Add to import switch statement | ||
| 7. Add to `PASSWORD_DRIVERS` (if applicable) | ||
| 8. Remove from `KNOWN_UNSUPPORTED` (if listed) | ||
|
|
||
| ### Phase 3: Discovery (4 files) | ||
|
|
||
| 9. Docker discovery — `docker-discovery.ts` (IMAGE_MAP, ENV_MAP, DEFAULT_PORTS, DEFAULT_USERS) | ||
| 10. Env var detection — `project-scan.ts` (detectEnvVars warehouses array) | ||
| 11. dbt adapter — `dbt-profiles.ts` (ADAPTER_TYPE_MAP) | ||
| 12. dbt lineage — `dbt/lineage.ts` (detectDialect dialectMap) | ||
|
|
||
| ### Phase 4: FinOps (1 file) | ||
|
|
||
| 13. Query history — `finops/query-history.ts` (SQL template + handler if database has system query log) | ||
|
|
||
| ### Phase 5: Build (1 file) | ||
|
|
||
| 14. Peer deps — `script/publish.ts` (driverPeerDependencies) | ||
|
|
||
| ### Phase 6: Tool Descriptions (1 file) | ||
|
|
||
| 15. warehouse_add — `tools/warehouse-add.ts` (config description + error message) | ||
|
|
||
| ### Phase 7: Tests (2 new files + 1 edit) | ||
|
|
||
| 16. E2E tests — `test/altimate/drivers-{database}-e2e.test.ts` | ||
| 17. Normalization tests — add to `test/altimate/driver-normalize.test.ts` | ||
| 18. Verify existing tests pass | ||
|
|
||
| ### Phase 8: Documentation (5 files) | ||
|
|
||
| 19. `docs/docs/configure/warehouses.md` — config section + update count | ||
| 20. `docs/docs/drivers.md` — support matrix + installation + auth + update count | ||
| 21. `docs/docs/data-engineering/tools/warehouse-tools.md` — env vars + Docker | ||
| 22. `README.md` — warehouse list | ||
| 23. `docs/docs/getting-started/index.md` — homepage list | ||
|
|
||
| ### Phase 9: Optional | ||
|
|
||
| - Guide page at `docs/docs/data-engineering/guides/{database}.md` | ||
| - Update `mkdocs.yml` nav and `guides/index.md` | ||
| - Check fingerprint regex in `fingerprint/index.ts` | ||
|
|
||
| ## Step 4: Run Quality Gates | ||
|
|
||
| ```bash | ||
| # Tests (from packages/opencode/) | ||
| cd packages/opencode && bun test test/altimate/driver-normalize.test.ts test/altimate/connections.test.ts | ||
|
|
||
| # Typecheck | ||
| bun turbo typecheck | ||
|
|
||
| # Marker check | ||
| bun run script/upstream/analyze.ts --markers --base main --strict | ||
| ``` | ||
|
|
||
| All three must pass before proceeding. | ||
|
|
||
| ## Step 5: Run Code Review | ||
|
|
||
| Run `/consensus:code-review` to get the implementation reviewed by multiple models before committing. | ||
|
|
||
| ## Step 6: Summary | ||
|
|
||
| Present final summary: | ||
| ``` | ||
| ## {Database} Driver Added | ||
|
|
||
| ### Files Created | ||
| - packages/drivers/src/{database}.ts | ||
| - packages/opencode/test/altimate/drivers-{database}-e2e.test.ts | ||
| - docs/docs/data-engineering/guides/{database}.md (if created) | ||
|
|
||
| ### Files Modified | ||
| - {list all modified files} | ||
|
|
||
| ### Test Results | ||
| - {N} normalization tests pass | ||
| - {N} connection tests pass | ||
| - Typecheck: pass | ||
| - Marker check: pass | ||
|
|
||
| ### E2E Test Coverage | ||
| - {list of test suites and server versions} | ||
|
|
||
| Ready to commit. | ||
| ``` | ||
|
|
||
| ## Rules | ||
|
|
||
| 1. **Read before writing.** Always read existing drivers and the reference doc before creating new code. | ||
| 2. **Don't skip integration points.** All 23 points exist for a reason — missing one causes inconsistencies users will hit. | ||
| 3. **Use parameterized queries** for `listTables` and `describeTable` — never interpolate user input into SQL. | ||
| 4. **Test multiple server versions** — at minimum: latest stable + oldest non-EOL LTS. | ||
| 5. **Run all quality gates** before presenting the summary. | ||
| 6. **Don't modify finops tools** (credit-analyzer, warehouse-advisor, unused-resources) unless the database has equivalent cost/credit APIs. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,29 +1 @@ | ||
| fix: tool reliability improvements for sql-classify, edit, and webfetch (#581) | ||
|
|
||
| **sql-classify.ts:** | ||
| - Fix `computeSqlFingerprint` referencing undefined `core` variable after | ||
| safe-import refactor — extract `extractMetadata` as module-level guard | ||
| - Invert fallback classifier to whitelist reads (`READ_PATTERN`) instead of | ||
| blacklisting writes — treats unknown statements as "write" for safety | ||
| - Handle multi-statement SQL in fallback by splitting on semicolons | ||
| - Strip `--` line comments in fallback (block comments already stripped) | ||
| - Fix `HARD_DENY_PATTERN` trailing `\s` → `\b` to match `TRUNCATE;` | ||
|
|
||
| **edit.ts:** | ||
| - Add `buildNotFoundMessage` with Levenshtein nearest-match snippets for | ||
| LLM self-correction when `oldString` not found | ||
| - Fix substring matching to prefer exact equality over short-line matches | ||
|
|
||
| **webfetch.ts:** | ||
| - Add session-level URL failure cache (404/410/451) with 5-min TTL | ||
| - Add `buildFetchError` with actionable status-specific error messages | ||
| - Add `sanitizeUrl` to strip query strings from error messages | ||
| - Add URL validation via `new URL()` constructor | ||
| - Add `MAX_CACHED_URLS = 500` size cap with oldest-entry eviction | ||
|
|
||
| **Tests:** 12 new tests for `buildNotFoundMessage`, `replace` error | ||
| messages, `computeSqlFingerprint`, and updated webfetch assertions. | ||
|
|
||
| Closes #581 | ||
|
|
||
| Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> | ||
| feat: add ClickHouse warehouse driver |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| # Warehouses | ||
|
|
||
| Altimate Code connects to 9 warehouse types. Configure them in `.altimate-code/connections.json` (project-local) or `~/.altimate-code/connections.json` (global). | ||
| Altimate Code connects to 10 warehouse types. Configure them in `.altimate-code/connections.json` (project-local) or `~/.altimate-code/connections.json` (global). | ||
|
|
||
| ## Configuration | ||
|
|
||
|
|
@@ -288,6 +288,66 @@ If you're already authenticated via `gcloud`, omit `credentials_path`: | |
| !!! info "Server compatibility" | ||
| The MongoDB driver (v6.x) supports MongoDB server versions 3.6 through 8.0, covering all releases from the last 3+ years. | ||
|
|
||
| ## ClickHouse | ||
|
|
||
| ```json | ||
| { | ||
| "clickhouse-prod": { | ||
| "type": "clickhouse", | ||
| "host": "localhost", | ||
| "port": 8123, | ||
| "database": "analytics", | ||
| "user": "default", | ||
| "password": "{env:CLICKHOUSE_PASSWORD}" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| | Field | Required | Description | | ||
| |-------|----------|-------------| | ||
| | `connection_string` | No | Full URL (alternative to individual fields, e.g. `http://user:pass@host:8123`) | | ||
| | `host` | No | Hostname (default: `localhost`) | | ||
| | `port` | No | HTTP port (default: `8123`) | | ||
| | `database` | No | Database name (default: `default`) | | ||
| | `user` | No | Username (default: `default`) | | ||
| | `password` | No | Password | | ||
| | `protocol` | No | `http` or `https` (default: `http`) | | ||
| | `request_timeout` | No | Request timeout in ms (default: `30000`) | | ||
| | `tls_ca_cert` | No | Path to CA certificate for TLS | | ||
| | `tls_cert` | No | Path to client certificate for mutual TLS | | ||
| | `tls_key` | No | Path to client key for mutual TLS | | ||
| | `clickhouse_settings` | No | Object of ClickHouse server settings | | ||
|
|
||
| ### ClickHouse Cloud | ||
|
|
||
| ```json | ||
| { | ||
| "clickhouse-cloud": { | ||
| "type": "clickhouse", | ||
| "host": "abc123.us-east-1.aws.clickhouse.cloud", | ||
| "port": 8443, | ||
| "protocol": "https", | ||
| "user": "default", | ||
| "password": "{env:CLICKHOUSE_CLOUD_PASSWORD}", | ||
| "database": "default" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Using a connection string | ||
|
|
||
| ```json | ||
| { | ||
| "clickhouse-prod": { | ||
| "type": "clickhouse", | ||
| "connection_string": "https://default:secret@my-ch.cloud:8443" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| !!! info "Server compatibility" | ||
| The ClickHouse driver supports ClickHouse server versions 23.3 and later, covering all non-EOL releases. This includes LTS releases 23.8, 24.3, 24.8, and all stable releases through the current version. | ||
|
|
||
|
Comment on lines
+291
to
+350
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove stale “unsupported” references now that ClickHouse is documented as supported. The new ClickHouse section conflicts with two older lines in the same doc: ClickHouse is still listed as unsupported, and Docker auto-discovery text omits ClickHouse. 📝 Suggested doc patch ## Unsupported Databases
@@
-| ClickHouse | Use the bash tool with `clickhouse-client` or `curl` to query directly |
| Cassandra | Use the bash tool with `cqlsh` to query directly |
| CockroachDB | PostgreSQL-compatible — use `type: postgres` |
| TimescaleDB | PostgreSQL extension — use `type: postgres` |
@@
-| Docker containers | Finds running PostgreSQL, MySQL, and SQL Server containers |
+| Docker containers | Finds running PostgreSQL, MySQL, SQL Server, and ClickHouse containers |Also applies to: 383-383, 425-425 🧰 Tools🪛 LanguageTool[style] ~347-~347: Using many exclamation marks might seem excessive (in this case: 18 exclamation marks for a text that’s 9075 characters long) (EN_EXCESSIVE_EXCLAMATION) 🤖 Prompt for AI Agents |
||
| ## SQL Server | ||
|
|
||
| ```json | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix the quality-gate recipe's working-directory and coverage gap.
After the first line, the shell stays in
packages/opencode, sobun turbo typecheckand the marker check run from the wrong directory. This block also never executestest/altimate/drivers-{database}-e2e.test.ts, so the workflow can declare the new driver green without running its Docker suite.🛠️ Proposed fix
🤖 Prompt for AI Agents