Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
181 changes: 181 additions & 0 deletions .claude/commands/add-database-driver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
---
description: Add a new database driver to Altimate Code. Scaffolds the driver, registers it across all 23 integration points, writes E2E tests, and updates docs. Usage - /add-database-driver <database-name>
---

# Add Database Driver

Scaffold and fully integrate a new database/warehouse driver into Altimate Code. This command handles all 23 integration points — driver code, registry, discovery, finops, tests, and documentation.

## Input

`$ARGUMENTS` = the database name (e.g., `cockroachdb`, `timescaledb`, `cassandra`, `neo4j`).

If empty, ask: "Which database should I add support for?"

## Step 0: Research

Before writing any code, research the database:

1. **Find the official Node.js/TypeScript client package** on npm. Search for `@{database}/client`, `{database}-js`, or similar.
2. **Check supported server versions** — which versions are not EOL?
3. **Identify auth methods** — password, token, TLS/certificate, connection string, cloud-specific?
4. **Check SQL dialect** — standard SQL? Custom syntax? LIMIT vs TOP vs FETCH FIRST? System tables for schemas/tables/columns?
5. **Find Docker image** — official image on Docker Hub for E2E testing?
6. **Check if dbt adapter exists** — search for `dbt-{database}` on PyPI.

Present findings to the user before proceeding:
```
## Research: {Database}

- **npm package**: `{package}` (v{version})
- **Server versions**: {non-EOL versions}
- **Auth methods**: {list}
- **SQL dialect**: {notes on LIMIT, system tables, parameterized queries}
- **Docker image**: `{image}:{tag}`
- **dbt adapter**: {exists/not found}

Proceed with implementation?
```

## Step 1: Read Reference Document

Read the comprehensive checklist:
```bash
cat packages/drivers/ADDING_A_DRIVER.md
```

This document has all 23 integration points with exact file paths and code patterns.

## Step 2: Read Existing Driver for Pattern

Read a similar existing driver as a template. Choose based on database type:

- **SQL database with password auth** → read `packages/drivers/src/mysql.ts`
- **Cloud warehouse with token auth** → read `packages/drivers/src/databricks.ts`
- **Database with connection string support** → read `packages/drivers/src/postgres.ts`
- **HTTP-based client** → read `packages/drivers/src/clickhouse.ts`
- **Document database (non-SQL)** → read `packages/drivers/src/mongodb.ts`

Also read:
- `packages/drivers/src/normalize.ts` — for alias pattern
- `packages/opencode/src/altimate/native/connections/registry.ts` — for registration pattern
- `packages/opencode/test/altimate/drivers-docker-e2e.test.ts` — for E2E test pattern

## Step 3: Implement (23 integration points)

Work through all 9 phases from the checklist. Use parallel edits where possible.

### Phase 1: Core Driver (4 files)

1. **Create `packages/drivers/src/{database}.ts`**
- Follow the Connector interface: `connect()`, `execute()`, `listSchemas()`, `listTables()`, `describeTable()`, `close()`
- Lazy-import the npm package
- Use parameterized queries for schema introspection
- Handle LIMIT injection with DML guard: `!hasDML` check before appending LIMIT
- Handle TLS detection from connection strings

2. **Add export to `packages/drivers/src/index.ts`**

3. **Add optionalDependency to `packages/drivers/package.json`**

4. **Add aliases to `packages/drivers/src/normalize.ts`**

### Phase 2: Registry (4 files in registry.ts)

5. Add to `DRIVER_MAP`
6. Add to import switch statement
7. Add to `PASSWORD_DRIVERS` (if applicable)
8. Remove from `KNOWN_UNSUPPORTED` (if listed)

### Phase 3: Discovery (4 files)

9. Docker discovery — `docker-discovery.ts` (IMAGE_MAP, ENV_MAP, DEFAULT_PORTS, DEFAULT_USERS)
10. Env var detection — `project-scan.ts` (detectEnvVars warehouses array)
11. dbt adapter — `dbt-profiles.ts` (ADAPTER_TYPE_MAP)
12. dbt lineage — `dbt/lineage.ts` (detectDialect dialectMap)

### Phase 4: FinOps (1 file)

13. Query history — `finops/query-history.ts` (SQL template + handler if database has system query log)

### Phase 5: Build (1 file)

14. Peer deps — `script/publish.ts` (driverPeerDependencies)

### Phase 6: Tool Descriptions (1 file)

15. warehouse_add — `tools/warehouse-add.ts` (config description + error message)

### Phase 7: Tests (2 new files + 1 edit)

16. E2E tests — `test/altimate/drivers-{database}-e2e.test.ts`
17. Normalization tests — add to `test/altimate/driver-normalize.test.ts`
18. Verify existing tests pass

### Phase 8: Documentation (5 files)

19. `docs/docs/configure/warehouses.md` — config section + update count
20. `docs/docs/drivers.md` — support matrix + installation + auth + update count
21. `docs/docs/data-engineering/tools/warehouse-tools.md` — env vars + Docker
22. `README.md` — warehouse list
23. `docs/docs/getting-started/index.md` — homepage list

### Phase 9: Optional

- Guide page at `docs/docs/data-engineering/guides/{database}.md`
- Update `mkdocs.yml` nav and `guides/index.md`
- Check fingerprint regex in `fingerprint/index.ts`

## Step 4: Run Quality Gates

```bash
# Tests (from packages/opencode/)
cd packages/opencode && bun test test/altimate/driver-normalize.test.ts test/altimate/connections.test.ts

# Typecheck
bun turbo typecheck

# Marker check
bun run script/upstream/analyze.ts --markers --base main --strict
```

All three must pass before proceeding.
Comment on lines +129 to +142
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fix the quality-gate recipe's working-directory and coverage gap.

After the first line, the shell stays in packages/opencode, so bun turbo typecheck and the marker check run from the wrong directory. This block also never executes test/altimate/drivers-{database}-e2e.test.ts, so the workflow can declare the new driver green without running its Docker suite.

🛠️ Proposed fix
 ```bash
 # Tests (from packages/opencode/)
-cd packages/opencode && bun test test/altimate/driver-normalize.test.ts test/altimate/connections.test.ts
+(cd packages/opencode && bun test \
+  test/altimate/driver-normalize.test.ts \
+  test/altimate/connections.test.ts \
+  test/altimate/drivers-{database}-e2e.test.ts)
 
-# Typecheck
+# Typecheck (from repo root)
 bun turbo typecheck
 
-# Marker check
+# Marker check (from repo root)
 bun run script/upstream/analyze.ts --markers --base main --strict
</details>

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
## Step 4: Run Quality Gates

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.claude/commands/add-database-driver.md around lines 129 - 142, The test
block currently runs "cd packages/opencode && bun test
test/altimate/driver-normalize.test.ts test/altimate/connections.test.ts" which
leaves the shell in packages/opencode so the subsequent "bun turbo typecheck"
and "bun run script/upstream/analyze.ts --markers --base main --strict" run from
the wrong directory and the E2E suite isn't executed; change the test invocation
to run in a subshell (e.g., wrap the bun test invocation in parentheses) and
include "test/altimate/drivers-{database}-e2e.test.ts" in the bun test arguments
so the new driver’s Docker tests execute, and ensure "bun turbo typecheck" and
"bun run script/upstream/analyze.ts --markers --base main --strict" are executed
from the repo root (i.e., not affected by the cd).


## Step 5: Run Code Review

Run `/consensus:code-review` to get the implementation reviewed by multiple models before committing.

## Step 6: Summary

Present final summary:
```
## {Database} Driver Added

### Files Created
- packages/drivers/src/{database}.ts
- packages/opencode/test/altimate/drivers-{database}-e2e.test.ts
- docs/docs/data-engineering/guides/{database}.md (if created)

### Files Modified
- {list all modified files}

### Test Results
- {N} normalization tests pass
- {N} connection tests pass
- Typecheck: pass
- Marker check: pass

### E2E Test Coverage
- {list of test suites and server versions}

Ready to commit.
```

## Rules

1. **Read before writing.** Always read existing drivers and the reference doc before creating new code.
2. **Don't skip integration points.** All 23 points exist for a reason — missing one causes inconsistencies users will hit.
3. **Use parameterized queries** for `listTables` and `describeTable` — never interpolate user input into SQL.
4. **Test multiple server versions** — at minimum: latest stable + oldest non-EOL LTS.
5. **Run all quality gates** before presenting the summary.
6. **Don't modify finops tools** (credit-analyzer, warehouse-advisor, unused-resources) unless the database has equivalent cost/credit APIs.
30 changes: 1 addition & 29 deletions .github/meta/commit.txt
Original file line number Diff line number Diff line change
@@ -1,29 +1 @@
fix: tool reliability improvements for sql-classify, edit, and webfetch (#581)

**sql-classify.ts:**
- Fix `computeSqlFingerprint` referencing undefined `core` variable after
safe-import refactor — extract `extractMetadata` as module-level guard
- Invert fallback classifier to whitelist reads (`READ_PATTERN`) instead of
blacklisting writes — treats unknown statements as "write" for safety
- Handle multi-statement SQL in fallback by splitting on semicolons
- Strip `--` line comments in fallback (block comments already stripped)
- Fix `HARD_DENY_PATTERN` trailing `\s` → `\b` to match `TRUNCATE;`

**edit.ts:**
- Add `buildNotFoundMessage` with Levenshtein nearest-match snippets for
LLM self-correction when `oldString` not found
- Fix substring matching to prefer exact equality over short-line matches

**webfetch.ts:**
- Add session-level URL failure cache (404/410/451) with 5-min TTL
- Add `buildFetchError` with actionable status-specific error messages
- Add `sanitizeUrl` to strip query strings from error messages
- Add URL validation via `new URL()` constructor
- Add `MAX_CACHED_URLS = 500` size cap with oldest-entry eviction

**Tests:** 12 new tests for `buildNotFoundMessage`, `replace` error
messages, `computeSqlFingerprint`, and updated webfetch assertions.

Closes #581

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add ClickHouse warehouse driver
21 changes: 21 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ jobs:
- 'packages/opencode/test/altimate/drivers-e2e.test.ts'
- 'packages/opencode/test/altimate/drivers-docker-e2e.test.ts'
- 'packages/opencode/test/altimate/drivers-mongodb-e2e.test.ts'
- 'packages/opencode/test/altimate/drivers-clickhouse-e2e.test.ts'
- 'packages/opencode/test/altimate/connections.test.ts'
dbt-tools:
- 'packages/dbt-tools/**'
Expand Down Expand Up @@ -198,6 +199,19 @@ jobs:
--health-timeout 5s
--health-retries 10

clickhouse:
image: clickhouse/clickhouse-server:latest
env:
CLICKHOUSE_DB: testdb
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
ports:
- 18123:8123
options: >-
--health-cmd "wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1"
--health-interval 5s
--health-timeout 5s
--health-retries 15

steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

Expand Down Expand Up @@ -245,6 +259,13 @@ jobs:
TEST_MONGODB_HOST: 127.0.0.1
TEST_MONGODB_PORT: "27017"

- name: Run ClickHouse driver E2E
run: bun test test/altimate/drivers-clickhouse-e2e.test.ts
working-directory: packages/opencode
env:
TEST_CLICKHOUSE_HOST: 127.0.0.1
TEST_CLICKHOUSE_PORT: "18123"

# Cloud tests NOT included — they require real credentials
# Run locally with:
# ALTIMATE_CODE_CONN_SNOWFLAKE_TEST='...' bun test test/altimate/drivers-snowflake-e2e.test.ts
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Each mode has scoped permissions, tool access, and SQL write-access control.

## Supported Warehouses

Snowflake · BigQuery · Databricks · PostgreSQL · Redshift · DuckDB · MySQL · SQL Server · Oracle · SQLite
Snowflake · BigQuery · Databricks · PostgreSQL · Redshift · ClickHouse · DuckDB · MySQL · SQL Server · Oracle · SQLite · MongoDB

First-class support with schema indexing, query execution, and metadata introspection. SSH tunneling available for secure connections.

Expand Down
5 changes: 5 additions & 0 deletions bun.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

62 changes: 61 additions & 1 deletion docs/docs/configure/warehouses.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Warehouses

Altimate Code connects to 9 warehouse types. Configure them in `.altimate-code/connections.json` (project-local) or `~/.altimate-code/connections.json` (global).
Altimate Code connects to 10 warehouse types. Configure them in `.altimate-code/connections.json` (project-local) or `~/.altimate-code/connections.json` (global).

## Configuration

Expand Down Expand Up @@ -288,6 +288,66 @@ If you're already authenticated via `gcloud`, omit `credentials_path`:
!!! info "Server compatibility"
The MongoDB driver (v6.x) supports MongoDB server versions 3.6 through 8.0, covering all releases from the last 3+ years.

## ClickHouse

```json
{
"clickhouse-prod": {
"type": "clickhouse",
"host": "localhost",
"port": 8123,
"database": "analytics",
"user": "default",
"password": "{env:CLICKHOUSE_PASSWORD}"
}
}
```

| Field | Required | Description |
|-------|----------|-------------|
| `connection_string` | No | Full URL (alternative to individual fields, e.g. `http://user:pass@host:8123`) |
| `host` | No | Hostname (default: `localhost`) |
| `port` | No | HTTP port (default: `8123`) |
| `database` | No | Database name (default: `default`) |
| `user` | No | Username (default: `default`) |
| `password` | No | Password |
| `protocol` | No | `http` or `https` (default: `http`) |
| `request_timeout` | No | Request timeout in ms (default: `30000`) |
| `tls_ca_cert` | No | Path to CA certificate for TLS |
| `tls_cert` | No | Path to client certificate for mutual TLS |
| `tls_key` | No | Path to client key for mutual TLS |
| `clickhouse_settings` | No | Object of ClickHouse server settings |

### ClickHouse Cloud

```json
{
"clickhouse-cloud": {
"type": "clickhouse",
"host": "abc123.us-east-1.aws.clickhouse.cloud",
"port": 8443,
"protocol": "https",
"user": "default",
"password": "{env:CLICKHOUSE_CLOUD_PASSWORD}",
"database": "default"
}
}
```

### Using a connection string

```json
{
"clickhouse-prod": {
"type": "clickhouse",
"connection_string": "https://default:secret@my-ch.cloud:8443"
}
}
```

!!! info "Server compatibility"
The ClickHouse driver supports ClickHouse server versions 23.3 and later, covering all non-EOL releases. This includes LTS releases 23.8, 24.3, 24.8, and all stable releases through the current version.

Comment on lines +291 to +350
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Remove stale “unsupported” references now that ClickHouse is documented as supported.

The new ClickHouse section conflicts with two older lines in the same doc: ClickHouse is still listed as unsupported, and Docker auto-discovery text omits ClickHouse.

📝 Suggested doc patch
 ## Unsupported Databases
@@
-| ClickHouse | Use the bash tool with `clickhouse-client` or `curl` to query directly |
 | Cassandra | Use the bash tool with `cqlsh` to query directly |
 | CockroachDB | PostgreSQL-compatible — use `type: postgres` |
 | TimescaleDB | PostgreSQL extension — use `type: postgres` |
@@
-| Docker containers | Finds running PostgreSQL, MySQL, and SQL Server containers |
+| Docker containers | Finds running PostgreSQL, MySQL, SQL Server, and ClickHouse containers |

Also applies to: 383-383, 425-425

🧰 Tools
🪛 LanguageTool

[style] ~347-~347: Using many exclamation marks might seem excessive (in this case: 18 exclamation marks for a text that’s 9075 characters long)
Context: ...ult:secret@my-ch.cloud:8443" } } ``` !!! info "Server compatibility" The Cli...

(EN_EXCESSIVE_EXCLAMATION)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/docs/configure/warehouses.md` around lines 291 - 350, The docs still
contain stale references that mark ClickHouse as unsupported and omit it from
Docker auto-discovery; locate the ClickHouse section and any mentions of
“unsupported” or “not supported” referring to ClickHouse (search for the
"ClickHouse" heading and the Docker auto-discovery paragraph) and remove or
update those lines to reflect ClickHouse is supported, and add ClickHouse to the
Docker auto-discovery list/description where other supported warehouses are
enumerated (also update the other two occurrences mentioned in the review).
Ensure the wording matches the new supported phrasing used in the ClickHouse
section (e.g., reference supported versions and cloud options).

## SQL Server

```json
Expand Down
Loading
Loading