Skip to content

docs: refresh FEATURES and COMPARISON against main and the other tools #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
debabsah merged 1 commit into from
Oct 3, 2026
Merged

docs: refresh FEATURES and COMPARISON against main and the other tools #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions docs/COMPARISON.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
# How Chartwright Compares

Chartwright is one of several command-line tools for managing Apache Superset assets. They overlap less than the category name suggests: each automates a different job, and most shops could reasonably run more than one. This page states what each tool does so you can pick by the job in front of you.
Chartwright is one of several tools for managing Apache Superset assets from outside the UI. They overlap less than the category name suggests: each automates a different job, and most shops could reasonably run more than one. This page states what each tool does so you can pick by the job in front of you.

Facts checked 2026-07-24 against each project's public README and docs. If something here has gone stale, open an issue and it will be corrected.
Facts checked 2026-10-03 against each project's public README and docs, and Superset's MCP service against its source at the 6.1.0 tag. If something here has gone stale, open an issue and it will be corrected.

## The short version

| Tool | Where a dashboard starts | What gets checked | Built for |
|---|---|---|---|
| [Chartwright](https://github.com/debabsah/chartwright) | a spec you (or an AI) write: a small JSON description of the dashboard you want | every dataset, column, and metric confirmed to exist before anything is built; every chart compared to the spec and its query run once after; drift diff on demand | authoring dashboards as reviewable code on open-source Superset |
| [preset-cli / superset-cli](https://github.com/preset-io/backend-sdk) | the Superset UI; you export the built dashboard as YAML and optionally add Jinja templates | warns when target files already exist (pass `--overwrite` to replace); `delete-assets` dry-runs by default | syncing exported assets and dbt projects into Preset workspaces or standalone Superset |
| [Chartwright](https://github.com/debabsah/chartwright) | a spec you (or an AI) write: a small JSON description of the dashboard you want | every dataset, column, and metric confirmed to exist before anything is built (a wrong column comes back with the closest real names); every chart compared to the spec and its query run once after; drift diff on demand | authoring dashboards as reviewable code on open-source Superset |
| [preset-cli / superset-cli](https://github.com/preset-io/backend-sdk) | the Superset UI; you export the built dashboard as YAML and optionally add Jinja templates | warns when target files already exist (pass `--overwrite` to replace); `delete-assets` dry-runs by default, with optional best-effort rollback | syncing exported assets and dbt projects into Preset workspaces or standalone Superset |
| [sup](https://github.com/preset-io/superset-sup) | the Superset UI; sync configs point at source and target workspaces | `sync run --dry-run` previews the sync operations | power-user and agent workflows: SQL from the terminal, asset search, backup and restore, cross-workspace sync (beta, per its README) |
| [terraform-provider-superset](https://github.com/platacard/terraform-provider-superset) | Terraform HCL | Terraform's own plan and apply lifecycle | databases, datasets, roles, and users; dashboards and charts are not among its resources |
| [Superset's MCP service](https://github.com/apache/superset/tree/6.1.0/superset/mcp_service) (built into Superset 6.1.0) | an AI agent's tool calls: charts first, then a dashboard made from them | `generate_chart` checks a chart's columns against its dataset as it creates that chart | letting an AI agent create charts and dashboards from inside Superset; in 6.1.0, building again creates a new dashboard, and no tool changes an existing dashboard's layout, filters or title |

## The deepest difference: which way the dashboard flows

Expand All @@ -25,18 +26,20 @@ Neither direction is better in general. If your dashboards are built by analysts

The export-and-sync tools check transport concerns: existing targets, previewable operations. Chartwright's job is compilation, so what it verifies is content, at three points:

- **Before writing:** `chartwright check` signs in read-only and confirms every dataset, column, and metric the spec names actually exists. A misspelled column is a clear error before anything is created, which matters most when an AI wrote the spec.
- **After writing:** `chartwright apply` compares the finished dashboard chart-by-chart against the spec and runs each chart's query once to prove it loads data.
- **On demand:** `chartwright plan` diffs the spec against the live dashboard, field by field: charts, filters, scopes, title, layout. It passes when they match and fails when they drifted, ready as a CI gate.
- **Before writing:** `chartwright check` signs in read-only and confirms every dataset, column, and metric the spec names actually exists. A misspelled column is a clear error before anything is created, with the closest real column names suggested, which matters most when an AI wrote the spec.
- **After writing:** `chartwright apply` compares the finished dashboard chart-by-chart against the spec and runs each chart's query once: an error fails the build, and a chart with no rows is named.
- **On demand:** `chartwright plan` diffs the spec against the live dashboard: the charts and filters added, changed or removed, plus title, layout, cross-filter and series-colour changes. It passes when they match and fails when they drifted, ready as a CI gate.

`chartwright decompile` closes the loop from the other side: it turns a dashboard built in the UI into a spec, listing anything it could not carry over.
`chartwright decompile` closes the loop from the other side: it turns a dashboard built in the UI into a spec, listing most of what it couldn't carry over, and you build that spec as a copy at a new slug.

## Tool notes

**preset-cli / superset-cli** is maintained by Preset, the company behind much of Superset's development. Beyond native asset sync it exports RLS rules, roles, users, and ownership, and can build datasets straight from a dbt Core or dbt Cloud project. It runs against Preset workspaces (`preset-cli`) or any standalone Superset (`superset-cli`).

**sup** is Preset's newer CLI, self-described as beta, with a strong terminal experience: run SQL against any workspace database, search charts and datasets server-side, export chart data, back up and restore assets with dependency tracking, and machine-readable output modes aimed at scripts and AI agents.

**Superset's MCP service** ships with Superset 6.1.0 and gives an AI agent tools to create charts, create a dashboard from them, add charts to an existing dashboard, and update one chart. Superset's master branch adds tools that change existing dashboards, not yet in a release. Chartwright's own MCP server works the other way: the agent writes a spec, and every build updates the same dashboard in place.

**terraform-provider-superset** brings the databases-datasets-roles-users layer of a Superset instance under Terraform. If your platform team already lives in Terraform, it can own that layer while a dashboard tool owns the dashboards.

## Using them together
Expand All @@ -46,4 +49,4 @@ The tools compose because they hold different ends of the stack:
- Manage connections, roles, and users with Terraform; author the dashboards on top of them as Chartwright specs.
- Sync dataset definitions from your dbt project with preset-cli; point Chartwright specs at those datasets.
- Keep ad-hoc SQL, asset search, and scheduled backups on sup; keep the dashboards you want under PR review on Chartwright specs, with `plan` as the merge gate.
- Migrating instances? `superset-cli export-assets` moves everything wholesale; `chartwright decompile` adopts the dashboards you want to manage as code from then on.
- Migrating instances? `superset-cli export-assets` moves everything wholesale; `chartwright decompile` turns the dashboards you want to manage as code into specs from then on.
30 changes: 15 additions & 15 deletions docs/FEATURES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ into one. Everything below works from that one file.
- A screenshot of a dashboard in another BI tool, pointed at the same underlying data.
- **Reviewable Checkpoint**: The AI's output is a small spec file you can read, edit, and version like code.
- **Open AI Contract**: `chartwright schema` prints the full JSON Schema so any LLM or tool can generate valid specs.
- **MCP Server**: Ten tools covering the whole lifecycle, usable from any MCP client.
- **Guardrails**: The dashboard is new; the data behind it must be real. Every dataset, column, and metric the AI references is confirmed to exist before anything is built, so a made-up column becomes a clear error message, never a broken chart.
- **MCP Server**: Ten tools covering the whole lifecycle, usable from any MCP client. Failures come back as the same typed JSON errors the CLI prints.
- **Guardrails**: The dashboard is new; the data behind it must be real. Every dataset, column, and metric the AI references is confirmed to exist before anything is built, so a made-up column becomes a clear error message, never a broken chart. The error suggests the closest real column names and lists the dataset's columns, so the AI can correct itself in one round.

## Dashboard Design
- **Deterministic Dashboards**: The same spec always produces the identical dashboard. Diff it in git, review it in a PR.
Expand Down Expand Up @@ -49,30 +49,30 @@ into one. Everything below works from that one file.

## Dashboards as Code
- **Drift Detection**: `chartwright plan` diffs the spec against the live dashboard: charts, filters, scopes, title, layout.
- **Adopt Existing Dashboards**: Turn any dashboard built in the UI into a spec with `chartwright decompile`; anything it cannot carry over is listed, so you know exactly what stayed in the UI.
- **Lossless Round-Trips**: Tool-built dashboards decompile back to their exact spec.
- **Start From Existing Dashboards**: Turn any dashboard built in the UI into a spec with `chartwright decompile`, then build it as a copy at a new slug. Most of what it can't carry over is listed; a few settings (such as dashboard CSS, annotations and tab-scoped filters) are dropped without a note, so compare before you retire the original.
- **Lossless Round-Trips**: Tool-built dashboards with `rows` or `tabs` layouts decompile back with nothing lost; a `sketch` comes back as rows.
- **Targeted Edits**: Replace, rename, resize, or remove one chart and re-apply; old charts are cleaned up, never orphaned.
- **Stable Identity**: Chart ids never change across re-applies, so links, scopes, and open browser tabs stay valid.

## CI/CD and Promotion
- **Environment Promotion**: Specs name their data (connection, schema, table), so the same file applies to dev, staging, and production unchanged; nothing in it is tied to one instance.
- **Environment Promotion**: Specs name their data (connection, schema, table), never instance ids, so the same file applies to dev, staging, and production when they share connection names; where names differ, generate one copy per instance.
- **PR-Gated Dashboard Changes**: Specs live in git; `chartwright plan` passes when the live dashboard matches the spec and fails when it drifted, ready as a merge gate; read-only `chartwright check` runs safely on any schedule.
- **Offline Compilation**: Build the import bundle with `chartwright compile`, no server needed; the output is reproducible byte-for-byte.
- **Instance Migration**: Decompile from one Superset, apply to another.
- **Instance Migration**: Decompile from one Superset, apply the spec to another; it applies to 4.1.4 and 5.0.0 even when it came from 6.1.0, whose exports those releases reject.
- **Git as the Source of Truth**: A lost or mangled dashboard is one re-apply away from its spec.

## Safety and Recovery
- **Automatic Backups**: Every apply saves the previous state first, no flag needed.
- **Automatic Backups**: Every apply to an existing dashboard saves the live state first, no flag needed; backups are named to the microsecond and never overwritten.
- **Complete Restore**: `chartwright restore` brings back the dashboard, chart settings, and filter scopes.
- **Self-Healing Applies**: A failed apply restores the previous state automatically.
- **Self-Healing Applies**: When an apply fails while preparing, importing or updating charts, the previous state is restored automatically, whatever the error. A failure after that (linkage, filter scopes, chart queries) leaves the new version live, and the report gives the backup to restore.
- **Stale-Tab Protection**: An old browser tab writing back stale state is detected by `plan` and repaired by `apply`.
- **Verified at Every Step**: References are checked before anything is written, the finished dashboard is compared chart-by-chart against the spec, and every chart's query is run once to prove it shows data; any failure says what went wrong and where.
- **Ownership Guard**: The tool only ever overwrites dashboards it created. To edit a hand-built dashboard, adopt it first (decompile it into a spec, then apply); the original stays untouched.
- **Verified at Every Step**: References are checked before anything is written, the finished dashboard is compared chart-by-chart against the spec, and every chart's query is run once: an error fails the apply, and a chart that returns no rows is named. Any failure says what went wrong and where.
- **Ownership Guard**: The tool only ever overwrites dashboards it created. To manage a hand-built dashboard, decompile it into a spec and build it at a new slug; the original stays untouched.

## Enterprise Ready
- **Multiple Instances**: Sandbox, staging, and production as profiles in one file.
- **Secrets Stay Out of Your Profiles**: Point each profile at where its secret already lives.
- Self-hosted Superset signs in with a username and password (database or LDAP provider): take the password from an env var (any name) or your credential manager (1Password, macOS Keychain, sops). On SSO-only Superset, ask your admin for a service account with password login enabled.
- Self-hosted Superset signs in with a username and password (database or LDAP provider): take the password from an env var (any name) or your credential manager (1Password, macOS Keychain, sops). Chartwright never signs in through SSO or OAuth; on an instance where people use SSO, ask your admin for an account that has a Superset password.
- Preset-hosted workspaces (preset.io) sign in with an API token and secret: take them from env vars, or reuse the credentials preset-cli already stored.
- **What the AI Can See**: The AI proposes; the tool verifies, using your own Superset login. Verification reads names (datasets, columns, metrics), not rows; the AI never queries your warehouse.
- The post-apply data check keeps a row count and discards the rows.
Expand All @@ -81,10 +81,10 @@ into one. Everything below works from that one file.

## More Ways to Use It
- **Dashboard Documentation**: Decompile any dashboard into a readable inventory of its charts, metrics, and filters.
- **Cloning and Templating**: Copy a spec, change the name and the data it points at, apply. A proven dashboard becomes a starting point.
- **Cloning**: Copy a spec, change the slug and the data it points at, apply. A proven dashboard becomes a starting point; a short script makes one copy per team or tenant (specs hold literal values, no template variables).
- **Programmatic Dashboards**: Specs are JSON; generate or edit them with scripts, one dashboard per file.
- **Drift Audits**: Run `plan` on a schedule to catch UI edits that diverged from the reviewed spec.
- **Training and Demo Environments**: Boot a sandbox, apply a spec, and every student or demo gets the identical dashboard.
- **Training and Demo Environments**: Apply the same spec to each training or demo instance, and every student or demo gets the identical dashboard.
- **Safe Experiments**: Try a redesign under a new name, compare side by side with the original, delete nothing.

## CLI Verbs
Expand All @@ -105,8 +105,8 @@ into one. Everything below works from that one file.
| `chartwright restore` | Bring back a backed-up dashboard, completely |

## Testing and Evidence
- **The full offline suite** on Linux and Windows on every push, and on every pull request ([exact counts](VERIFICATION.md)).
- **Live CI against real Superset 4.1.4, 5.0.0, and 6.1.0** on every push: full apply, lifecycle soak, stale-tab adversary, fault injection.
- **The full offline suite** on Linux and Windows on every pull request and every push to main ([exact counts](VERIFICATION.md)).
- **Live CI against real Superset 4.1.4, 5.0.0, and 6.1.0** on every pull request and every push to main: full apply, lifecycle soak, stale-tab adversary, fault injection.
- **500-cycle soak** passed on the oldest and newest supported versions.
- **Chart options verified against Superset's own source code** for every supported version, so a Superset change surfaces here before it reaches your dashboards.
- Full evidence: `docs/VERIFICATION.md`; source citations: `docs/CONTRACTS.md`.
Expand Down
Loading