diff --git a/README.md b/README.md index f79756b..c7a8834 100644 --- a/README.md +++ b/README.md @@ -1,205 +1,91 @@ # Chartwright -A dashboard compiler for Apache Superset. The `chartwright` command line builds -a dashboard from a small deterministic file, updates it from that file later, -and decompiles an existing dashboard back into one. Every dataset, column, and -metric is checked against your own Superset first. An AI can write the file for -you, or revise it, from a request in plain words. - -![An operations dashboard over NYC yellow-taxi data: KPI cards with unit subtitles, daily trend charts, an hour-by-weekday demand heatmap, a payment donut, borough and zone rankings, and a trip-distance histogram, behind a three-filter bar](https://raw.githubusercontent.com/debabsah/chartwright/main/docs/images/nyc-taxi-operations.png) - -*3.9 million yellow-taxi trips (NYC TLC trip records, May 2026), 11 verified -charts, three filters, one `chartwright apply`. An AI wrote -[the spec](https://github.com/debabsah/chartwright/blob/main/examples/nyc_taxi_operations.json) from a one-paragraph request: -[the request and rebuild steps](https://github.com/debabsah/chartwright/blob/main/examples/README.md).* - -A dashboard that lives in a file gets the workflow code already has: review it -in a pull request, rebuild it identically, and bring it back after a bad -change. `chartwright plan` diffs the file against the live dashboard and exits -non-zero when they differ, so drift fails a CI check. `chartwright decompile` -covers the other direction, and lists anything it could not carry over. - -Write the file yourself, or describe the dashboard you want and have an AI -write it. The same checks run either way. After the build, each chart's query -runs once (an error fails the build, an empty chart is named), and -`chartwright advise` reviews the layout for readability. - -## What a spec looks like - -The file is called a spec: - -```json -{ - "spec_version": "1", - "dashboard": {"title": "Sales Overview", "slug": "sales-overview"}, - "charts": [ - { - "name": "Total Sales", - "type": "big_number_total", - "dataset": {"database": "examples", "table": "cleaned_sales_data"}, - "metric": "SUM(sales)" - }, - { - "name": "Sales Over Time", - "type": "timeseries_line", - "dataset": {"database": "examples", "table": "cleaned_sales_data"}, - "metrics": ["SUM(sales)"], - "time_column": "order_date" - } - ], - "layout": {"rows": [["Total Sales"], ["Sales Over Time"]]} -} -``` +[![PyPI](https://img.shields.io/pypi/v/chartwright)](https://pypi.org/project/chartwright/) [![Python](https://img.shields.io/pypi/pyversions/chartwright)](https://pypi.org/project/chartwright/) [![CI](https://github.com/debabsah/chartwright/actions/workflows/ci.yml/badge.svg)](https://github.com/debabsah/chartwright/actions/workflows/ci.yml) [![Superset](https://img.shields.io/badge/Superset-4.1.4%20%7C%205.0.0%20%7C%206.1.0-1f6fb2)](docs/VERIFICATION.md) [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) -`chartwright apply` on this file signs in to your Superset, confirms the -`cleaned_sales_data` table and the `sales` and `order_date` columns exist, -builds a KPI card above a line chart, and finishes by running each chart's -query once. A misspelled column or a missing table is a clear error naming the -problem, with the closest real names suggested, before anything is created. +Chartwright is a command-line tool that manages Apache Superset dashboards as code: one JSON spec file per dashboard, covering its charts and how each is drawn, its filters, layout, text, colours and styling, and naming the datasets it reads. Chartwright creates or updates the dashboard on your Superset instances from the spec, reports where the live dashboard differs from it, and can generate a spec from a dashboard that already exists. Specs can be written by hand or by an AI agent, and both go through identical validation before anything changes in Superset. -## Creating dashboards with AI +![The Sundown dashboard's Overview in five palettes, then the dashboard scrolling through its Overview, Evening ramp, Storage and Supply tabs, then About](docs/images/sundown.webp) -The spec is typed, and every reference in it is checked before anything is -created. Whatever a model proposes has to survive the same verification your -own specs do. +*Sundown: example dashboard built using Chartwright.* -- **Claude Code skill**: from a clone of this repo, run - `python install-skill.py`; the model writes the spec from your request, and - the tool verifies and builds it. -- **MCP server**: `chartwright-mcp` (installed with `pip install "chartwright[mcp]"`) - exposes fourteen tools covering the whole lifecycle, usable from any MCP client. -- **Open contract**: `chartwright schema` prints the spec's JSON Schema, so any LLM or - tool can generate valid specs. -- **Guardrails**: the AI proposes; the tool verifies, using your own Superset - login. Verification reads names (datasets, columns, metrics), not rows. +| You can | Instead of (Superset up to 6.1.0, [cited to source](docs/CONTRACTS.md)) | How it works | +|---|---|---| +| ● [Move dashboards between dev, staging, prod or another instance, including ones built in the UI](docs/MOVE-BETWEEN-INSTANCES.md) | Re-importing dashboard exports in the UI, which update the dashboard but not the charts already in it ([#34879](https://github.com/apache/superset/issues/34879), fixed after 6.1.0, not yet released) | [MOVE-BETWEEN-INSTANCES.md](docs/MOVE-BETWEEN-INSTANCES.md) | +| ● [Clone any dashboard onto another dataset, team or tenant](docs/CLONE-A-DASHBOARD.md) | Editing exported files, or copying the dashboard and re-pointing each chart at the new dataset by hand ([Stack Overflow](https://stackoverflow.com/q/69179618), [#20090](https://github.com/apache/superset/issues/20090)) | [CLONE-A-DASHBOARD.md](docs/CLONE-A-DASHBOARD.md) | +| ● [Describe a dashboard, or a change to one chart, to an AI agent and have it built, checked and updated in place, without learning Superset's editor](docs/AI-AGENTS.md) | Learning the chart editor, where tasks can take "hours of searching" ([Hacker News](https://news.ycombinator.com/item?id=39512272)), or Superset's MCP tools, which create a new dashboard on every build and can only add charts to an existing one ([#39864](https://github.com/apache/superset/discussions/39864)) | [AI-AGENTS.md](docs/AI-AGENTS.md) | +| ● [Lay out rows, columns and tabs by drawing them as text, and keep heights you adjust in Superset](docs/LAYOUT-GUIDE.md) | Nesting columns inside rows in the editor to place a tall chart beside two stacked ones ([2019](https://stackoverflow.com/q/55665527), [2026](https://stackoverflow.com/q/79905463)) | [LAYOUT-GUIDE.md](docs/LAYOUT-GUIDE.md) | +| ● [Review dashboard changes as a readable diff in a pull request, and check them in CI before they go live](docs/DEPLOY-FROM-GIT.md) | Exports full of instance-specific ids, which don't diff or merge cleanly in git ([#30190](https://github.com/apache/superset/discussions/30190)), and no settled way to deploy dashboards through CI/CD ([Stack Overflow](https://stackoverflow.com/q/79809865), unanswered) | [DEPLOY-FROM-GIT.md](docs/DEPLOY-FROM-GIT.md) | +| ● [Roll back any change, chart settings included, from a backup saved automatically before every apply](docs/HISTORY-AND-ROLLBACK.md) | Remembering to export before each change, then re-importing, which restores the dashboard but not its charts' settings ([#34879](https://github.com/apache/superset/issues/34879), fixed after 6.1.0, not yet released); re-importing deleted assets can also fail ([#44309](https://github.com/apache/superset/issues/44309)) | [HISTORY-AND-ROLLBACK.md](docs/HISTORY-AND-ROLLBACK.md) | +| ● [Rebuild your dashboards on a new Superset version from the same specs](docs/SUPERSET-VERSIONS.md) | Checking and repairing charts by hand when an upgrade leaves them blank in the editor ([#32725](https://github.com/apache/superset/discussions/32725)) | [SUPERSET-VERSIONS.md](docs/SUPERSET-VERSIONS.md) | +| ● [Create dashboards from code without reverse-engineering Superset's JSON](docs/DASHBOARDS-FROM-CODE.md) | Working out the undocumented layout JSON, `position_json` ([#32970](https://github.com/apache/superset/discussions/32970)) | [DASHBOARDS-FROM-CODE.md](docs/DASHBOARDS-FROM-CODE.md) | -## Quick start +## Install ```bash -pip install chartwright +pip install chartwright # Python 3.11 or newer + +# OPTIONAL +pip install "chartwright[mcp]" # adds the MCP server for AI clients, chartwright-mcp +pip install "chartwright[visual]" # for chartwright save-queries (CSV reports on built charts); + # then run: playwright install chromium ``` -Python 3.11 or newer; three dependencies; Windows, macOS, and Linux. +## Try it on your Superset -Create `~/.config/chartwright/profiles.toml` pointing at your Superset: +A dev or test instance is ideal. Add a profile to `~/.config/chartwright/profiles.toml`; it names the instance and where the password comes from, never the password itself: ```toml -[prod] -base_url = "https://superset.your-company.com" -username = "your-username" -password_env = "CHARTWRIGHT_PROD_PASSWORD" +[dev] # identifier +base_url = "https://superset-dev.example.com" +username = "chartwright" +password_env = "SUPERSET_DEV_PASSWORD" ``` -Point the example spec's `dataset` lines at a table your Superset already -knows, save it as `sales_overview.json`, then: +Export the password variable, then: ```bash -chartwright check sales_overview.json --profile prod # read-only: verify every reference -chartwright apply sales_overview.json --profile prod # build, import, verify +# Build and update a dashboard +chartwright validate spec.json # check the spec offline +chartwright check spec.json --profile # find every missing dataset, column or metric +chartwright plan spec.json --profile # show what apply would change +chartwright apply spec.json --profile # build or update the dashboard, then query every chart +chartwright restore backup.zip --profile # roll back to the backup taken before an apply + +# Start from a dashboard you already have +chartwright decompile --profile -o spec.json # a spec for a copy of it +chartwright adopt --profile -o spec.json # manage it where it is +chartwright absorb spec.json --profile # copy heights set in Superset into the spec + +# Design review +chartwright brief # sizing and layout guidance for writing a spec +chartwright advise spec.json # review the design; --fix applies the safe fixes +chartwright explain spec.json # where each filled-in setting came from +chartwright redesign --profile -o spec.json # decompile, review and fix in one step + +# Other +chartwright schema # the spec's JSON Schema +chartwright compile spec.json -o bundle.zip # build the import file offline +chartwright standards check specs/ # team standards (also show, assign, apply) +chartwright save-queries spec.json --profile # optional [visual]: makes CSV reports work +chartwright calibrate # suggest default heights from what absorb copied ``` -No Superset to point at? Clone this repo: `sandbox/up.sh` boots a disposable -Superset 6.1.0 with example data on `localhost:8098` (Docker required; -sign-in is admin/admin), and `tests/fixtures/kitchen_sink.json` applies to it -as-is. - -## What you get - -- **Deterministic dashboards**: the same spec always produces the identical - dashboard. Diff it in git, review it in a PR. -- **Verified at every step**: every reference is checked before anything is - written, and every chart's query runs once (an error fails the build, an - empty chart is named); failures say what went wrong and where. -- **Dashboards as code, in both directions**: `chartwright decompile` turns any live - dashboard into a spec; `chartwright plan` shows what differs between the spec and - the live dashboard, ready as a CI gate; `chartwright compile` builds the import - bundle offline, no server needed. -- **Recoverable by default**: every apply backs up the previous state first, - and an apply that fails while importing or updating charts restores it - automatically. Dashboards Chartwright did not create are never overwritten; - to bring a hand-built one under a spec, decompile it and build it at a new - slug. -- **What a spec can express**: 17 chart types, metrics as you write them, - per-chart filters, a native filter bar, tabs, markdown notes, a header and - footer shown on every tab, and layouts you can draw as ASCII sketches. -- **Environment promotion**: specs name their data (connection, schema, - table), so the same file applies to dev, staging, and production when they - share connection names. - -Every capability, with the CLI verb reference: [docs/FEATURES.md](https://github.com/debabsah/chartwright/blob/main/docs/FEATURES.md). - -## The design brain - -A spec can import perfectly and still produce a dashboard nobody can read: a -pie squeezed into two columns, a KPI buried under three tables, a bar chart -with forty category labels Superset silently drops. The design brain covers -size and scroll budgets, chart-choice limits, and layout composition. It -arrives as a brief the AI reads before authoring, and a critic that reviews -the result: - -```bash -chartwright brief --audience executive # the guidance, before writing a spec -chartwright advise spec.json --fix # the critique, with safe auto-repairs -chartwright redesign old-dash --profile prod # audit + repair a live dashboard -``` - -You stay in charge of it: suppress any rule for one chart in the spec, tune -the thresholds per audience or per deployment in `design.yaml`, and feed the -heights you drag in the UI back into the recommendations with `chartwright -calibrate`. Pass `--design off` and the output is byte-identical to a build -that never had it. The full rulebook and architecture: -[docs/DESIGN-BRAIN.md](https://github.com/debabsah/chartwright/blob/main/docs/DESIGN-BRAIN.md). - -## Drawing layouts as text - -The spec's `sketch` is the dashboard's shape, drawn with characters that you -and a model can both write: each letter is a chart, and repeating a sketch -line makes its charts taller. - -```text -KKKK LLLLLLLL -.... LLLLLLLL -``` - -![A short chart beside one twice as tall](https://raw.githubusercontent.com/debabsah/chartwright/main/docs/images/sketches/taller.svg) - -`L` spans both lines, so it renders twice as tall as `K`; the dots keep the -space under `K` deliberately empty. Every rule, drawn and explained: -[docs/LAYOUT-GUIDE.md](https://github.com/debabsah/chartwright/blob/main/docs/LAYOUT-GUIDE.md). - -## Testing and evidence +`chartwright --help` lists each command's options. -Every pull request and every push to main runs the full offline suite on Linux and -Windows, plus the full pipeline (apply, lifecycle soak, stale-tab adversary, -fault injection) against real Superset 4.1.4, 5.0.0, and 6.1.0 containers. -Chart options are checked against Superset's own source for every supported -version, so a Superset change surfaces here before it reaches your dashboards. +## Requirements and limits -Full evidence: [docs/VERIFICATION.md](https://github.com/debabsah/chartwright/blob/main/docs/VERIFICATION.md). +- **Sign-in:** a Superset user with database or LDAP login, or Preset API tokens; Chartwright doesn't sign in through SSO or OAuth. On an instance that uses SSO, ask your admin for an account that has a Superset password. All testing uses the Admin role. +- **Datasets:** datasets and database connections stay in Superset; create them on each instance first. A spec names each dataset by its database connection and table, so where connection names differ between instances, keep one copy of the spec per instance. +- **Ownership:** Chartwright changes only dashboards it built or that you adopted, so give each team its own slug prefix. To manage a UI-made dashboard where it is, run `chartwright adopt`; it names every setting the first apply will reset, before you apply. +- **Edits:** each `apply` writes the spec over the dashboard and undoes edits made in the UI, and takes a chart added there off the dashboard (the chart itself stays in Superset's Charts list); make lasting changes, and add new charts, in the spec. To keep chart heights you set by dragging in Superset, run `chartwright absorb` first; it copies them into the spec. +- **Status:** version 0.6.0, beta. CI builds on real Superset 4.1.4, 5.0.0 and 6.1.0 on every pull request ([how it's tested](docs/VERIFICATION.md)). -[docs/CONTRACTS.md](https://github.com/debabsah/chartwright/blob/main/docs/CONTRACTS.md) records how Superset itself behaves: what its -importer accepts, how dashboard settings are stored, what its chart plugins -expect. Every behavior is cited to its source line at 4.1.4, 5.0.0, and 6.1.0. -Release tags are immutable, so any line of it can be checked against a fresh -checkout of that tag. +Every limit, with what to do instead: [Limits](docs/LIMITS.md). -## Documentation +Every capability and command: [FEATURES.md](docs/FEATURES.md) · How it's tested: [VERIFICATION.md](docs/VERIFICATION.md) · Superset's behaviour, cited to source: [CONTRACTS.md](docs/CONTRACTS.md) · Compared with preset-cli, sup and Terraform: [COMPARISON.md](docs/COMPARISON.md) · Design rules: [DESIGN-BRAIN.md](docs/DESIGN-BRAIN.md) -- [docs/FEATURES.md](https://github.com/debabsah/chartwright/blob/main/docs/FEATURES.md): every capability, with the CLI verb - reference -- [docs/LAYOUT-GUIDE.md](https://github.com/debabsah/chartwright/blob/main/docs/LAYOUT-GUIDE.md): drawing layouts as text, - every rule illustrated -- [docs/DESIGN-BRAIN.md](https://github.com/debabsah/chartwright/blob/main/docs/DESIGN-BRAIN.md): the design rulebook, every - rule and threshold, and how to tune or switch them off -- [docs/VERIFICATION.md](https://github.com/debabsah/chartwright/blob/main/docs/VERIFICATION.md): what is tested, what it - caught, and how to reproduce it -- [docs/CONTRACTS.md](https://github.com/debabsah/chartwright/blob/main/docs/CONTRACTS.md): how Superset behaves on import, - on dashboard storage, and in chart options, cited to its source line at - each supported release -- [docs/COMPARISON.md](https://github.com/debabsah/chartwright/blob/main/docs/COMPARISON.md): what Chartwright, preset-cli, - sup, and the Terraform provider each automate, so you can pick by the job +[Changelog](CHANGELOG.md) · [Stability](docs/STABILITY.md) · [Contributing](CONTRIBUTING.md) · Apache-2.0 ([LICENSE](LICENSE), [NOTICE](NOTICE)) --- -Apache Superset, Superset, and the Superset logo are trademarks of the Apache -Software Foundation. No endorsement by the ASF is implied. +Apache Superset and Superset are trademarks of the Apache Software Foundation. No endorsement by the ASF is implied. diff --git a/docs/AI-AGENTS.md b/docs/AI-AGENTS.md new file mode 100644 index 0000000..db2b765 --- /dev/null +++ b/docs/AI-AGENTS.md @@ -0,0 +1,154 @@ +# Building and changing dashboards with an AI agent + +You describe a dashboard, or a change to one chart, in plain words. Your agent writes or edits the spec file, and you get the dashboard built in Superset with every reference checked first, each chart confirmed on the dashboard and its query run. Ask for a change and the same dashboard and charts are updated, with no copies left behind. + +## Connect your agent + +Install the MCP server, which adds the `chartwright-mcp` command (Python 3.11 or newer): + +```bash +pip install "chartwright[mcp]" +``` + +Register it with Claude Code. `CHARTWRIGHT_PROFILES` points the server at a profiles file holding only the instances you want the agent to reach (see [Keep production out of reach](#keep-production-out-of-reach)): + +```bash +claude mcp add chartwright -e CHARTWRIGHT_PROFILES="$HOME/.config/chartwright/agent-profiles.toml" -- chartwright-mcp +``` + +For Claude Desktop or any other MCP client, add the server to the client's config: + +```json +{ + "mcpServers": { + "chartwright": { + "command": "chartwright-mcp", + "env": {"CHARTWRIGHT_PROFILES": "/Users/you/.config/chartwright/agent-profiles.toml"} + } + } +} +``` + +- Use the absolute path to the profiles file; `~` isn't expanded there. +- If the client can't find `chartwright-mcp`, give the full path that `which chartwright-mcp` prints. +- The server reads passwords the same way the CLI does. A client started from your desktop doesn't see variables exported in a shell, so give the profile a `password_cmd` (a command that prints the password, such as your password manager's CLI) instead of `password_env`. + +### The Claude Code skill + +The repository also ships a Claude Code skill. With it, Claude reads the schema and the design brief, recommends a style for titles and colour that you can accept or change, writes only the spec, then runs `validate`, `check`, `advise` and `apply` in order, and edits the spec to change a dashboard later. It runs from a clone: + +```bash +git clone https://github.com/debabsah/chartwright && cd chartwright +python -m venv .venv && .venv/bin/pip install -e ".[mcp]" +.venv/bin/python install-skill.py +``` + +- The installer copies the skill to `~/.claude/skills/superset-dashboard/`, so it works from any directory, and writes `.mcp.json` in the clone to register the MCP server there. +- The skill keeps specs in the clone's `specs/` folder. Re-run `install-skill.py` after moving the clone. On Windows, run `.venv\Scripts\python install-skill.py`. + +## The 15 tools + +Each tool runs the same code as its CLI command and returns a JSON report. + +| MCP tool | CLI command | What the agent gets | +|---|---|---| +| `get_spec_schema` | `chartwright schema` | The JSON Schema every spec follows | +| `design_brief` | `chartwright brief` | The design brief to read before writing | +| `validate_spec` | `chartwright validate` | Field and layout errors, offline | +| `check_spec` | `chartwright check` | Every missing dataset, column or metric, plus design advice | +| `advise_spec` | `chartwright advise` | The readability review; pass a profile for the data-aware rules | +| `fix_spec` | `chartwright advise --fix` | The spec with safe fixes applied and design defaults filled in, returned instead of written to a file | +| `explain_spec` | `chartwright explain` | For each chart, where each display setting's value comes from and how to change it | +| `plan_dashboard` | `chartwright plan` | What a build would add, change or remove | +| `build_dashboard` | `chartwright apply` | The build report: dashboard URL, backup path, each chart's query result, the design advice; with `save_queries`, each chart's saved query for CSV reports | +| `decompile_dashboard` | `chartwright decompile` | A spec of a live dashboard, with what it couldn't carry | +| `adopt_dashboard` | `chartwright adopt` | A spec that takes over a dashboard made in the UI where it is, with what the first build resets | +| `redesign_dashboard` | `chartwright redesign` | A decompiled spec with the safe fixes applied and the remaining findings | +| `standards_check` | `chartwright standards check` | The spec reviewed under your organisation's standard ([standards](DESIGN-BRAIN.md)), offline | +| `standards_show` | `chartwright standards show` | A standard after its inheritance: each setting, where it was set and whether it is locked | +| `standards_apply` | `chartwright standards apply` | The spec with its standard's header, footer, CSS and settings written in | + +The agent passes the spec as JSON text and a profile name. `fix_spec`, `redesign_dashboard`, `adopt_dashboard` and `standards_apply` hand back a spec; the agent saves it and builds it. `build_dashboard` with `save_queries` needs the `chartwright[visual]` extra and `playwright install chromium` where the server runs. + +When something fails, a tool returns the same JSON error the CLI prints, so the agent can tell the kind of failure apart: `"stage": "profile"` for a missing profile or an unset password variable, code `api` for a failed sign-in or a Superset error, and code `unexpected` for anything else. + +## What gets checked + +Before writing the spec: + +- `design_brief` gives the agent sizing budgets, chart choice and layout rules for an executive, analytical or operational audience. It also marks two styles as recommendations, not rules: titles that name the measure, unit and time window, and colour by role (status in green, amber and red beside an arrow or a word). +- `advise_spec` reviews the spec for readability, such as a pie with too many slices or a chart too short for its axis labels. `fix_spec` applies the safe ones, such as heights and bar orientation, and fills design defaults the spec left unset, such as axis label formats; the agent keeps the spec it returns and edits that. On a `sketch` layout, those fixes write explicit heights that override the drawing; to keep the drawing, edit the sketch lines instead. + +Before anything is written to Superset, Chartwright looks up, on the instance, each dataset the spec names and each column and metric its charts and filters use. A dataset is matched by its database connection name, table and schema. The agent gets the full list of missing references in one response, not one at a time. Columns inside a custom SQL metric or filter can't be looked up this way; they come back under `unchecked_sql`. The check also confirms the spec's owners and theme exist and that the instance's Superset release can take every field. + +After the import, `build_dashboard`: + +- takes off any chart the spec doesn't name, such as one added in the UI, naming it in a warning, then confirms the dashboard holds exactly the spec's charts; +- applies each filter's chart scope; +- runs every chart's query, and names any chart whose query fails or returns no rows. + +When the dashboard already exists, a backup is taken first. If anything fails before or during the import, including the in-place chart updates, that backup is put back automatically. A failure at the linkage, scope, owners or query step leaves the new version live; restore the backup from the CLI (see [What the MCP server leaves to the CLI](#what-the-mcp-server-leaves-to-the-cli)). + +## Updated in place + +A dashboard's id comes from its `slug`, and each chart's from the slug plus the chart's name. Building the spec again, or retrying a failed build, reaches the same dashboard and the same charts, and charts keep their Superset ids, so links to them keep working. + +- Renaming a chart shows in `plan_dashboard` as one chart removed and one added; the build deletes the old chart and creates a new one. In a spec from `adopt_dashboard`, which names the dashboard and its charts by their own ids, a chart renamed together with its key in `dashboard.adopted.charts` keeps its id. +- Changing the slug builds a new dashboard and leaves the old one as it was; retire the old one in Superset. +- Builds stop at a slug that holds a dashboard Chartwright didn't build. To manage that dashboard where it is, have the agent run `adopt_dashboard` and show you what it lists under `resets` before the first build; otherwise choose another slug. +- Two specs with the same slug manage the same dashboard, so give the agent its own slug prefix. + +## Change one chart + +Ask for the change in words, such as "show only the top 5 regions in Revenue by Region". + +1. The agent edits that chart's entry in the spec, here `"row_limit": 5`, and leaves the rest alone. +2. `plan_dashboard` lists it under `charts_changed`. It also lists anything edited in the Superset UI since the last build, which the build would overwrite. +3. `build_dashboard` updates the charts in place and keeps their ids. + +Every build rewrites the dashboard from the spec, so edits made in the UI are undone; tags and a theme the spec leaves out stay as they are, and so do owners, apart from the account that builds, which each build adds. Make lasting changes, and add charts, in the spec; a chart added to the dashboard in the UI is taken off it on the next build and stays in Superset's Charts list. + +## What the agent sees + +Through Chartwright, the agent sees names and counts: + +- each missing dataset, column or metric by name; a missing saved metric lists the dataset's saved metrics, and a missing column suggests up to three close matches (in the error's `candidates` field) and lists the dataset's columns; +- specs of existing dashboards, including titles, markdown text, filter default values and the owners' usernames or emails; +- distinct-value counts per column, capped, when `advise_spec` or `redesign_dashboard` runs with a profile; +- the number of rows each chart's test query returned. + +No tool returns query rows. No tool lists a dataset's columns up front: the agent learns them from `check_spec`, whose error for a wrong column names the closest matches and the dataset's columns. To skip that round, put the column names in your request. The agent passes only a profile name; the password stays in the environment variable or password command the profile names, and no tool returns it. + +## Keep production out of reach + +Give the agent only a dev or test profile: + +- Point the MCP server's `CHARTWRIGHT_PROFILES` at a file that holds only those profiles. The server reads every profile in its file, and an unknown-profile error lists the names in it. +- With the skill, Claude runs the CLI from your shell; start that session with `CHARTWRIGHT_PROFILES` set to the same file, and keep production passwords out of its environment. +- Build on production yourself, or from CI, once you've reviewed the spec. + +## What the MCP server leaves to the CLI + +- **Restore:** there's no restore tool. Run `chartwright restore --profile `, using the `backup` path from the build report. +- **Kept UI heights:** run `chartwright absorb --profile ` before the agent's next build; it copies chart heights set in Superset into the spec. +- **Offline import ZIP:** run `chartwright compile` (see [DASHBOARDS-FROM-CODE.md](DASHBOARDS-FROM-CODE.md)). + +## Problems reported with Superset's own MCP tools + +| Superset 6.1.0's MCP service | Reported in | With Chartwright | +|---|---|---| +| `generate_dashboard` always creates a new dashboard, `add_chart_to_existing_dashboard` only appends charts, and no tool changes an existing dashboard's layout, filters or title, so each round of tweaks leaves another "My Dashboard (3)" | [#39864](https://github.com/apache/superset/discussions/39864) | `build_dashboard` updates the dashboard at the spec's slug, and its charts, in place | +| Agents reference columns the dataset doesn't have; `generate_chart` catches them one chart at a time, as it creates each chart | [#40920](https://github.com/apache/superset/discussions/40920) | Every reference in the whole spec is checked, and every missing one listed, before anything is written | +| `generate_dashboard` saves the dashboard unpublished with the agent's user as its only owner, and no tool changes that, so the person who asked may not see it | [#42001](https://github.com/apache/superset/discussions/42001) | Chartwright builds dashboards as published unless the spec sets `"published": false`, and Superset lists published dashboards for anyone with access to one of their datasets; `dashboard.owners` names the owners | +| With more than two `generate_chart` calls at once, a call can report failure for a chart it already saved, and the retry adds a duplicate (fixed on Superset's master branch after 6.1.0, not yet released) | [#42567](https://github.com/apache/superset/issues/42567) | Chart ids come from the slug and chart name, so a retried build updates the same charts | + +Superset's master branch adds MCP tools such as `update_dashboard`, none of them in a release yet. + +## Related + +- [DASHBOARDS-FROM-CODE.md](DASHBOARDS-FROM-CODE.md): the spec's fields, JSON output and exit codes +- [LAYOUT-GUIDE.md](LAYOUT-GUIDE.md): drawing layouts with `sketch` +- [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md): backups and `restore` +- [DESIGN-BRAIN.md](DESIGN-BRAIN.md): the rules behind the brief and the review +- [LIMITS.md](LIMITS.md): every limit, with what to do instead +- [CONTRACTS.md](CONTRACTS.md): Superset's behaviour, cited to source diff --git a/docs/CLONE-A-DASHBOARD.md b/docs/CLONE-A-DASHBOARD.md new file mode 100644 index 0000000..2322269 --- /dev/null +++ b/docs/CLONE-A-DASHBOARD.md @@ -0,0 +1,105 @@ +# Clone a dashboard onto another dataset, team or tenant + +Copy a spec, give the copy its own slug and point it at other datasets, and Chartwright builds a separate dashboard that reads that data. A short loop gives every tenant its own copy and keeps all the copies in step with one source file. Start from a spec you already have, or generate one from any dashboard on your instance, including one built in the UI. + +## Copy a spec onto other data + +1. Copy the spec to a new file. +2. Change `dashboard.slug`, and the title if you like. +3. Change the `dataset` block on each chart, and on each filter that has one (every type but time range). A dataset is named by its database connection, schema and table. If the spec has a `dashboard.adopted` block (written by `chartwright adopt`), delete it: it ties the spec to the original dashboard, and a new slug is refused while it's there. +4. Run `chartwright check copy.json --profile prod`, then `chartwright apply copy.json --profile prod`. + +| Change in the copy | Effect | +|---|---| +| `dashboard.slug` | A separate dashboard. Chart ids come from the slug and the chart name, so the copy gets charts of its own and the original is left as it is | +| `dashboard.title` | The name people see in Superset | +| `database`, `schema` or `table` in a `dataset` block | That chart or filter reads the new dataset. `check` confirms the dataset has every column and saved metric the spec names, and lists the columns inside custom SQL, which it can't check, under `unchecked_sql` | + +Two specs with the same slug manage the same dashboard, so give every copy a slug of its own, such as a team or tenant suffix. + +## One copy per tenant + +This loop assumes each tenant's tables sit in a schema of their own, with the same table names in every schema. It writes one spec per tenant from `orders.json` (the template), checks it, and builds it: + +```bash +for t in acme globex; do + jq --arg t "$t" ' + .dashboard.slug = "orders-\($t)" + | .dashboard.title = "Orders: \($t)" + | (.. | objects | select(has("table")) | .schema) = $t + ' orders.json > "orders-$t.json" + chartwright check "orders-$t.json" --profile prod --design off \ + && chartwright apply "orders-$t.json" --profile prod +done +``` + +- The jq line sets the schema in every `dataset` block, charts and filters alike. +- Where each tenant has its own database connection instead, set `.database`; where each has its own table, set `.table`. +- Tenant names become part of the slug, which takes lowercase letters, digits and hyphens. `chartwright validate` rejects any other slug before you touch Superset. +- Each tenant's datasets must already exist in Superset: register each tenant's table as a dataset first. When one is missing, `check` names it, lists tables of the same name in other schemas, and exits 1, so the loop skips that tenant's apply and moves on. + +## Update every copy + +Edit `orders.json` and run the loop again. Each apply finds its copy by slug and updates it in place: + +- charts keep their ids, so links and filter scopes keep working; +- a chart removed from the template is deleted from every copy; +- a chart renamed in the template is deleted and created again under its new name in every copy; +- each copy is backed up before it changes. + +To see what a run will change first, run `chartwright plan orders-acme.json --profile prod`. It also shows edits someone made to that copy in the UI, which the next apply overwrites. + +## Vary the copies with a script + +A spec holds literal values only: every slug, title and dataset name is written out in full, so the file you review is the dashboard that gets built. To vary anything per copy, generate the copies with a script, as the jq loop does, or with any language that writes JSON. Edit only the template, regenerate, and run `chartwright validate` on each generated file. + +## Start from a dashboard you already have + +Use this for any dashboard on your instance, including one built in the UI. To manage the original itself where it is, keeping its address and chart ids, use `chartwright adopt` instead ([MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md#take-over-a-dashboard-made-in-the-ui)); the steps below build a copy. + +1. Generate its spec: `chartwright decompile regional-sales --profile prod -o regional-sales.json`, using the slug or the numeric id. The spec goes to the file and the `losses` list to the terminal. +2. Read `losses` (below), then check the settings it doesn't mention. +3. Change `dashboard.slug`, for example to `regional-sales-v2`. Applying at the original slug is refused, because Chartwright didn't build that dashboard; to manage it at that slug, use `chartwright adopt` instead. +4. To clone it onto other data at the same time, change the `dataset` blocks as above. +5. Run `chartwright validate regional-sales.json`, then `check` and `apply` with your profile. +6. Compare the new dashboard with the original. Then point bookmarks, embedded links and scheduled reports at the new one, and delete the original in Superset. + +For a dashboard Chartwright built, edit the spec it was built from. If that spec is lost, `decompile` works there too, and applying the result at the same slug updates the dashboard; read `losses` first. + +`losses` lists what the spec can't carry, one entry per item, naming the chart, filter or layout part: + +- charts of a type outside the 15 Chartwright builds, or whose dataset can't be found, which are skipped; +- chart settings, filter defaults and filter chart scopes that weren't preserved, by name, including any chart setting changed from Superset's default that the spec can't hold, such as a rolling average or a forecast; +- filters scoped to some tabs, which the spec scopes to the whole dashboard; +- layout changes: nested tabs flattened, sizes rounded, and charts stacked beside a taller one flattened into a row when that part of the layout also holds something a `sketch` can't (text, a header, a divider, a white row, or a chart decompile skips); otherwise they come back as a `sketch`. + +A few dashboard settings are dropped without a note ([listed in LIMITS.md](LIMITS.md#settings-decompile-drops-without-a-note)), so check them on the original before you retire it. Where one matters, keep the original dashboard for it, or rework the chart with the fields the spec has (`chartwright schema` lists every field). Once a spec manages the dashboard, make changes in the spec and apply them, since each apply overwrites edits made in the UI. To move the copy to other instances, see [MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md). + +## Embedding and row-level security + +Both are untested with Chartwright, and neither is in the spec. Confirm them on your own test instance before you rely on them: + +- **Embedding:** build a copy, enable embedding on it in Superset, and note its settings. Change the spec, apply it again, then confirm the embedding settings are unchanged and the embedded dashboard still loads in your app. +- **Row-level security:** the rules live on datasets in Superset, and Chartwright never reads or changes them. Add a rule for a test role on one tenant's dataset, sign in as a user with that role, and confirm that tenant's copy shows only the rows the rule allows. + +## Limits + +| What's true today | What to do instead | +|---|---| +| A spec has no template variables | Generate the copies from one template with a script, as in the loop above | +| One dashboard per spec, and one spec per command (the `standards` commands take a folder) | Loop over the copies, as above | +| Each copy's datasets must already exist on the instance | Register them in Superset first; `check` lists the missing ones | +| A copy at a new slug gets new chart ids, so links to the original's charts don't follow it | To keep the original's address and ids, adopt it with `chartwright adopt` instead of copying it | +| `decompile` reads a dashboard from a live instance, not from a ZIP export | Import the ZIP into your test instance and decompile it there | +| A few dashboard settings are dropped by `decompile` without a note ([listed in LIMITS.md](LIMITS.md#settings-decompile-drops-without-a-note)) | Check them on the original before you retire it, and keep the original where one matters | +| A chart Chartwright built that leaves the spec is deleted, even if someone also added it to another dashboard | To show a chart on two dashboards, put it in both specs; each builds a chart of its own | +| A chart added in the UI to a copy is taken off it by the next apply (the chart stays in Superset's Charts list) | Add charts to the template and run the loop | +| Embedding and row-level security are untested | Confirm them on your test instance, as above | + +## Related + +- [MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md): profiles, sign-in, and moving a spec between dev, staging and prod +- [DEPLOY-FROM-GIT.md](DEPLOY-FROM-GIT.md): run the loop from CI +- [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md): backups and `restore` +- [DASHBOARDS-FROM-CODE.md](DASHBOARDS-FROM-CODE.md): the spec's fields, JSON output and exit codes for scripting +- [VERIFICATION.md](VERIFICATION.md): how it's tested diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index 891aae3..4ab6309 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -8,7 +8,7 @@ Facts checked 2026-10-03 against each project's public README and docs, and Supe | Tool | Where a dashboard starts | What gets checked | Built for | |---|---|---|---| -| [Chartwright](https://github.com/debabsah/chartwright) | a spec you (or an AI) write: a small JSON description of the dashboard you want | every dataset, column, and metric confirmed to exist before anything is built (a wrong column comes back with the closest real names); every chart compared to the spec and its query run once after; drift diff on demand | authoring dashboards as reviewable code on open-source Superset | +| [Chartwright](https://github.com/debabsah/chartwright) | a spec you (or an AI) write: a small JSON description of the dashboard you want | every dataset, column, and metric confirmed to exist before anything is built (a wrong column comes back with the closest real names); every chart confirmed on the dashboard and its query run once after; drift diff on demand | authoring dashboards as reviewable code on open-source Superset | | [preset-cli / superset-cli](https://github.com/preset-io/backend-sdk) | the Superset UI; you export the built dashboard as YAML and optionally add Jinja templates | warns when target files already exist (pass `--overwrite` to replace); `delete-assets` dry-runs by default, with optional best-effort rollback | syncing exported assets and dbt projects into Preset workspaces or standalone Superset | | [sup](https://github.com/preset-io/superset-sup) | the Superset UI; sync configs point at source and target workspaces | `sync run --dry-run` previews the sync operations | power-user and agent workflows: SQL from the terminal, asset search, backup and restore, cross-workspace sync (beta, per its README) | | [terraform-provider-superset](https://github.com/platacard/terraform-provider-superset) | Terraform HCL | Terraform's own plan and apply lifecycle | databases, datasets, roles, and users; dashboards and charts are not among its resources | @@ -26,11 +26,11 @@ Neither direction is better in general. If your dashboards are built by analysts The export-and-sync tools check transport concerns: existing targets, previewable operations. Chartwright's job is compilation, so what it verifies is content, at three points: -- **Before writing:** `chartwright check` signs in read-only and confirms every dataset, column, and metric the spec names actually exists. A misspelled column is a clear error before anything is created, with the closest real column names suggested, which matters most when an AI wrote the spec. -- **After writing:** `chartwright apply` compares the finished dashboard chart-by-chart against the spec and runs each chart's query once: an error fails the build, and a chart with no rows is named. -- **On demand:** `chartwright plan` diffs the spec against the live dashboard: the charts and filters added, changed or removed, plus title, layout, cross-filter and series-colour changes. It passes when they match and fails when they drifted, ready as a CI gate. +- **Before writing:** `chartwright check` signs in read-only and confirms every dataset, column, and metric the spec names actually exists (columns inside custom SQL are listed for you to check). A misspelled column is a clear error before anything is created, with the closest real column names suggested, which matters most when an AI wrote the spec. +- **After writing:** `chartwright apply` confirms every chart in the spec is on the finished dashboard and runs each chart's query once: an error fails the build, and a chart with no rows is named. +- **On demand:** `chartwright plan` diffs the spec against the live dashboard: the charts and filters added, changed or removed, plus title, layout, cross-filter, series-colour, CSS and dashboard-setting changes. It passes when they match and fails when they drifted, ready as a CI gate ([deploy from git](DEPLOY-FROM-GIT.md)). -`chartwright decompile` closes the loop from the other side: it turns a dashboard built in the UI into a spec, listing most of what it couldn't carry over, and you build that spec as a copy at a new slug. +`chartwright adopt` closes the loop from the other side: it turns a dashboard built in the UI into a spec that manages that same dashboard, listing what the first apply resets ([how](MOVE-BETWEEN-INSTANCES.md#take-over-a-dashboard-made-in-the-ui)); `chartwright decompile` does the same for a copy at a new slug. ## Tool notes @@ -38,7 +38,7 @@ The export-and-sync tools check transport concerns: existing targets, previewabl **sup** is Preset's newer CLI, self-described as beta, with a strong terminal experience: run SQL against any workspace database, search charts and datasets server-side, export chart data, back up and restore assets with dependency tracking, and machine-readable output modes aimed at scripts and AI agents. -**Superset's MCP service** ships with Superset 6.1.0 and gives an AI agent tools to create charts, create a dashboard from them, add charts to an existing dashboard, and update one chart. Superset's master branch adds tools that change existing dashboards, not yet in a release. Chartwright's own MCP server works the other way: the agent writes a spec, and every build updates the same dashboard in place. +**Superset's MCP service** ships with Superset 6.1.0 and gives an AI agent tools to create charts, create a dashboard from them, add charts to an existing dashboard, and update one chart. Superset's master branch adds tools that change existing dashboards, not yet in a release. Chartwright's own MCP server works the other way: the agent writes a spec, and every build updates the same dashboard in place ([AI agents](AI-AGENTS.md)). **terraform-provider-superset** brings the databases-datasets-roles-users layer of a Superset instance under Terraform. If your platform team already lives in Terraform, it can own that layer while a dashboard tool owns the dashboards. @@ -49,4 +49,4 @@ The tools compose because they hold different ends of the stack: - Manage connections, roles, and users with Terraform; author the dashboards on top of them as Chartwright specs. - Sync dataset definitions from your dbt project with preset-cli; point Chartwright specs at those datasets. - Keep ad-hoc SQL, asset search, and scheduled backups on sup; keep the dashboards you want under PR review on Chartwright specs, with `plan` as the merge gate. -- Migrating instances? `superset-cli export-assets` moves everything wholesale; `chartwright decompile` turns the dashboards you want to manage as code into specs from then on. +- Migrating instances? `superset-cli export-assets` moves everything wholesale; `chartwright adopt` turns the dashboards you want to manage as code into specs from then on. diff --git a/docs/DASHBOARDS-FROM-CODE.md b/docs/DASHBOARDS-FROM-CODE.md new file mode 100644 index 0000000..c3bb41e --- /dev/null +++ b/docs/DASHBOARDS-FROM-CODE.md @@ -0,0 +1,132 @@ +# Creating dashboards from code + +You write, or generate, a short JSON spec that names a dashboard's charts, filters and layout, and build it into a Superset dashboard without writing Superset's import files or layout JSON yourself. Every command returns JSON and an exit code, so your scripts can generate, check and build dashboards in a loop. + +## Read the schema + +```bash +chartwright schema > spec.schema.json +``` + +`chartwright schema` prints the JSON Schema every spec follows; generate and check your specs against it. The same schema is in the repository at `schema/dashboard_spec.schema.json`, and a test keeps the file identical to the command's output. + +The main fields: + +- `spec_version`: always `"1"`. +- `dashboard`: `title` and `slug` (lowercase kebab-case). The slug is the dashboard's identity: building a spec with the same slug updates the same dashboard. Optional settings include `cross_filters`, `label_colors`, `css`, `color_scheme`, `description`, `owners`, `tags` and `refresh_frequency` (`chartwright schema` lists them all). +- `charts`: each chart has + - `name`, unique in the dashboard and part of the chart's identity; + - `type`, one of 15: `big_number_total`, `big_number_trend`, `timeseries_line`, `timeseries_bar`, `timeseries_area`, `timeseries_scatter`, `bar`, `pie`, `table`, `pivot_table`, `heatmap`, `histogram`, `funnel`, `treemap`, `mixed`; + - `dataset`: the `database` connection name and `table`, plus `schema` where the table name alone is ambiguous; + - metrics written as an aggregate, `SUM`, `AVG`, `COUNT`, `COUNT_DISTINCT`, `MIN` or `MAX` of a column (`SUM(amount)`, optionally `SUM(amount) AS Revenue`), a custom SQL expression (`SQL(amount * 1.1) AS Gross`), or the name of a saved metric; + - optional `filters` (WHERE conditions on that chart, or custom SQL), `width` in twelfths of the page, and `height` in 40 px units. +- `filters`: the dashboard's filter bar, with `select`, `time_range`, `range`, `time_grain` and `time_column` filters. +- `layout`: exactly one of `rows`, `tabs` or `sketch`, plus an optional `header` and `footer`. In `rows`, charts without a `width` share the row equally. +- `design`: optional; the review's `audience`, the rule ids to `ignore`, and the `standard` the spec follows ([standards](DESIGN-BRAIN.md)). + +A small spec, two numbers above a bar chart, with a region filter: + +```json +{ + "spec_version": "1", + "dashboard": {"title": "Orders by Region", "slug": "orders-by-region"}, + "charts": [ + {"name": "Orders", "type": "big_number_total", "metric": "COUNT(*)", "number_format": ",.0f", + "dataset": {"database": "warehouse", "table": "orders"}}, + {"name": "Revenue", "type": "big_number_total", "metric": "SUM(amount)", "number_format": "$,.0f", + "dataset": {"database": "warehouse", "table": "orders"}}, + {"name": "Revenue by Region", "type": "bar", "x_column": "region", "metrics": ["SUM(amount)"], + "orientation": "horizontal", "row_limit": 10, "height": 7, + "dataset": {"database": "warehouse", "table": "orders"}} + ], + "filters": [ + {"type": "select", "name": "Region", "column": "region", + "dataset": {"database": "warehouse", "table": "orders"}} + ], + "layout": {"rows": [["Orders", "Revenue"], ["Revenue by Region"]]} +} +``` + +Change `warehouse`, `orders`, `amount` and `region` to a dataset and columns on your Superset. To draw the layout as text instead of `rows`, see [LAYOUT-GUIDE.md](LAYOUT-GUIDE.md). + +## Check and build + +```bash +chartwright validate orders-by-region.json # fields and layout +chartwright advise orders-by-region.json # readability review +chartwright check orders-by-region.json --profile dev # each dataset, column and metric is on the instance +chartwright apply orders-by-region.json --profile dev # build, then run each chart's query +``` + +`validate` and `advise` need no Superset. `check` lists every missing reference in one response. `apply` runs the same check before it writes anything, then confirms each chart is on the dashboard and runs its query. + +## Inspect the import ZIP + +```bash +chartwright compile orders-by-region.json -o orders-by-region.zip +``` + +`compile` writes the chart and dashboard files of the import ZIP without contacting Superset, so you can read what a spec turns into. The same spec always compiles to the same bytes, so any difference between two ZIPs comes from the specs. Its dataset ids are stand-ins, so use the ZIP for debugging only, and build with `apply`, which looks up the real ids. Without `-o`, the file is named after the slug. + +## Read the results + +Every command prints JSON on stdout, except `brief`, which prints the design brief as Markdown, and `explain`, `standards show` and `standards apply`, which print text unless you pass `--json`. A failed command names the `stage` where it stopped, and lists what it found. Exit codes: + +| Command | Exit 0 | Exit 1 | +|---|---|---| +| `validate` | The spec is well formed | Unreadable file, or field and layout errors | +| `compile` | ZIP written | Unreadable file, or field and layout errors | +| `check` | Each dataset, column and metric was found | Missing references, all listed; a field the instance's Superset release can't take; or design findings with `--design strict` | +| `plan` | No differences between spec and live dashboard | Any difference; no dashboard at the slug yet; missing references; or a dashboard Chartwright didn't build at the slug (take it over with `chartwright adopt`) | +| `apply` | Built, and every chart's query ran (charts returning no rows are named in `warnings`) | Stopped at the step named in `stage` | +| `advise` | No error findings (with `--strict`, no warnings either) | Findings that block | + +- Commands that sign in exit 1, with JSON, on profile or sign-in errors. +- Every command exits 2 on a mistyped command or option; that message goes to stderr as plain text. +- `chartwright --help` lists each command's options. + +## Build many dashboards + +Each spec holds one dashboard, and each command except the `standards` ones takes one spec. Loop over a folder to build several: + +```bash +for spec in specs/*.json; do + chartwright validate "$spec" > /dev/null || { echo "invalid: $spec"; exit 1; } +done +mkdir -p out +for spec in specs/*.json; do + chartwright apply "$spec" --profile dev > "out/$(basename "$spec")" || echo "failed: $spec" +done +``` + +Each build report lands in `out/`; a failed one names its `stage` and carries `resolution_errors` or `import_detail`. + +## Drive it from your own code + +Run Chartwright through its CLI or its MCP server. No Python API is documented, and the package's internal modules can change between releases; call the CLI from your program and parse its JSON: + +```python +import json, subprocess + +run = subprocess.run(["chartwright", "validate", "orders-by-region.json"], capture_output=True, text=True) +result = json.loads(run.stdout) +print(run.returncode, result.get("errors", "ok")) +``` + +To drive it from an AI agent or another MCP client, use the server's 15 tools, which run the same code as the CLI commands and return JSON reports ([AI-AGENTS.md](AI-AGENTS.md)). + +## Limits + +- One dashboard per spec, and one spec per command (the `standards` commands take a folder); loop over a folder of specs, as above. +- A spec names each dataset by its database connection name, table and schema exactly as they are on the instance; where names differ between instances, generate one copy of the spec per instance. +- Datasets and database connections aren't created from the spec; create them in Superset first, and `check` confirms they're there. +- `compile` output carries stand-in dataset ids; build with `apply`. + +## Related + +- [CONTRACTS.md](CONTRACTS.md): how Superset reads the import files and layout JSON, cited to source +- [LAYOUT-GUIDE.md](LAYOUT-GUIDE.md): `rows`, `tabs` and `sketch` layouts +- [DEPLOY-FROM-GIT.md](DEPLOY-FROM-GIT.md): running these commands in CI +- [CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md): generating copies of a spec for other datasets +- [AI-AGENTS.md](AI-AGENTS.md): the MCP server +- [LIMITS.md](LIMITS.md): every limit, with what to do instead diff --git a/docs/DEPLOY-FROM-GIT.md b/docs/DEPLOY-FROM-GIT.md new file mode 100644 index 0000000..6da203b --- /dev/null +++ b/docs/DEPLOY-FROM-GIT.md @@ -0,0 +1,215 @@ +# Deploy dashboards from git + +Keep one spec file per dashboard in your repository and review each dashboard change as a short diff in its pull request. Before merge, CI confirms the change against your Superset instance and shows what it would change there; after merge, CI applies it, staging first if you want. A nightly job tells you when someone has edited a deployed dashboard in the UI. + +## 1. Review a change as a diff + +A spec diff is short enough to read in a pull request: + +- You write only what you chose: the charts, filters, layout and datasets. Chartwright fills in the rest of Superset's settings when it builds. +- The file holds no ids. Chartwright derives the dashboard's id from its slug and each chart's id from the slug and the chart's name, so the same spec gets the same ids on every run and on every instance. The exception is a spec `chartwright adopt` wrote for a dashboard made in the UI: it names that dashboard and its charts by their own ids, in its `dashboard.adopted` block. +- Compiling a spec gives a byte-identical result each time: `chartwright compile spec.json -o out.zip` writes the import ZIP offline, and the same spec always writes the same bytes ([tested](VERIFICATION.md)). Its dataset ids are placeholders, so you deploy with `apply`. + +Changing a number format, a filter default and adding a filter reads like this: + +```diff +- {"name": "Revenue", "type": "big_number_total", "metric": "SUM(amount)", "number_format": "$,.0f", ++ {"name": "Revenue", "type": "big_number_total", "metric": "SUM(amount)", "number_format": "$,.2f", + "dataset": {"database": "warehouse", "table": "orders"}}, + {"name": "Orders", "type": "big_number_total", "metric": "COUNT(*)", "number_format": ",.0f", + "dataset": {"database": "warehouse", "table": "orders"}} + ], + "filters": [ +- {"type": "time_range", "name": "Order date", "default": "No filter"} ++ {"type": "time_range", "name": "Order date", "default": "Last quarter"}, ++ {"type": "select", "name": "Region", "column": "region", ++ "dataset": {"database": "warehouse", "table": "orders"}} + ], +``` + +If prod matches the old spec, `chartwright plan` in the pull request reports Revenue under changed charts, Order date under changed filters and Region under added filters. Renaming a chart gives it a new id, so `plan` shows the old name removed and the new one added, and `apply` deletes the old chart. In an adopted spec, a chart renamed together with its key in `dashboard.adopted.charts` keeps its id and is renamed in Superset. + +## 2. Add the workflow + +Commit a profiles file such as `ci/profiles.toml`. It names each instance and the environment variable its password comes from, never the password itself ([every sign-in option](MOVE-BETWEEN-INSTANCES.md)): + +```toml +[staging] +base_url = "https://superset-staging.example.com" +username = "chartwright-deploy" +password_env = "SUPERSET_STAGING_PASSWORD" + +[prod] +base_url = "https://superset.example.com" +username = "chartwright-deploy" +password_env = "SUPERSET_PROD_PASSWORD" +``` + +Store each password as a repository secret (Settings, then Secrets and variables, then Actions) under the name its `password_env` gives. Each job below copies only the secrets it needs into its environment. Preset workspaces work the same way, with `api_token_env` and `api_secret_env` naming two secrets. + +Put the specs in `dashboards/` and add this workflow: + +```yaml +# .github/workflows/dashboards.yml +name: dashboards +on: + pull_request: + paths: ["dashboards/**", "standards/**", "ci/profiles.toml"] + push: + branches: [main] + paths: ["dashboards/**", "standards/**", "ci/profiles.toml"] + schedule: + - cron: "0 5 * * *" # nightly comparison with prod + workflow_dispatch: + +env: + CHARTWRIGHT_PROFILES: ci/profiles.toml + +jobs: + review: + if: github.event_name == 'pull_request' + runs-on: ubuntu-latest + env: + SUPERSET_PROD_PASSWORD: ${{ secrets.SUPERSET_PROD_PASSWORD }} + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: {python-version: "3.12"} + - run: pip install "chartwright>=0.5" + - name: Validate and review each spec (offline) + run: | + for spec in dashboards/*.json; do + chartwright validate "$spec" + chartwright advise --strict "$spec" + done + - name: Check each spec against prod and show what would change + run: | + for spec in dashboards/*.json; do + chartwright check "$spec" --profile prod + rc=0 + chartwright plan "$spec" --profile prod > plan.json || rc=$? + { echo "### $spec"; echo '~~~json'; cat plan.json; echo '~~~'; } >> "$GITHUB_STEP_SUMMARY" + if [ "$rc" -ne 0 ] && ! jq -e '.dashboard == "create" or .dashboard == "update"' plan.json > /dev/null; then + echo "::error file=$spec::plan stopped; see the run summary" + exit 1 + fi + done + + deploy-staging: # optional; delete this job and the needs: line below to deploy straight to prod + if: github.event_name == 'push' + runs-on: ubuntu-latest + concurrency: dashboards-staging + env: + SUPERSET_STAGING_PASSWORD: ${{ secrets.SUPERSET_STAGING_PASSWORD }} + CHARTWRIGHT_BACKUP_DIR: backups + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: {python-version: "3.12"} + - run: pip install "chartwright>=0.5" + - run: for spec in dashboards/*.json; do chartwright apply "$spec" --profile staging --design strict; done + - uses: actions/upload-artifact@v4 + if: always() + with: {name: backups-staging, path: backups/, if-no-files-found: ignore} + + deploy-prod: + if: github.event_name == 'push' + needs: deploy-staging + runs-on: ubuntu-latest + concurrency: dashboards-prod + env: + SUPERSET_PROD_PASSWORD: ${{ secrets.SUPERSET_PROD_PASSWORD }} + CHARTWRIGHT_BACKUP_DIR: backups + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: {python-version: "3.12"} + - run: pip install "chartwright>=0.5" + - run: for spec in dashboards/*.json; do chartwright apply "$spec" --profile prod --design strict; done + - uses: actions/upload-artifact@v4 + if: always() + with: {name: backups-prod, path: backups/, if-no-files-found: ignore} + + drift: + if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + env: + SUPERSET_PROD_PASSWORD: ${{ secrets.SUPERSET_PROD_PASSWORD }} + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: {python-version: "3.12"} + - run: pip install "chartwright>=0.5" + - name: Compare prod with each spec + run: | + status=0 + for spec in dashboards/*.json; do + chartwright plan "$spec" --profile prod > plan.json || status=1 + { echo "### $spec"; echo '~~~json'; cat plan.json; echo '~~~'; } >> "$GITHUB_STEP_SUMMARY" + done + exit "$status" +``` + +What each job does: + +- **review**, on every pull request: `validate` confirms each spec's fields and layout and `advise --strict` reviews its design (step 4), both offline; then `check` confirms that the datasets, columns and metrics the spec names exist on prod, listing any that are missing, and `plan` writes what would change on prod to the run summary. +- **deploy-staging** (optional) and **deploy-prod**, on merge to main: apply each spec to staging, then, if that succeeds, to prod, keeping each backup as a run artifact (step 5). +- **drift**, nightly and on demand: compares prod with each spec (step 5). + +`concurrency` sits only on the two deploy jobs, so a deploy waits for the one already running, and a pull request never cancels a deploy. Each loop stops at the first spec that fails; fix it, and the next run deploys the rest. + +Before you commit the workflow, confirm it parses: + +```bash +python -c 'import sys, yaml; yaml.safe_load(open(sys.argv[1]))' .github/workflows/dashboards.yml +``` + +## 3. Read the exit codes + +The commands below print JSON. Those that sign in exit 1 on a profile or sign-in error, and every command exits 2 on a mistyped command or option. Otherwise: + +| Command | Exits 0 when | Exits 1 when | +|---|---|---| +| `validate` | The spec's fields and layout are valid | Any field, layout or sketch error | +| `check` | All the datasets, columns and metrics the spec names exist | Any is missing (all are listed); design findings, with `--design strict` | +| `plan` | Nothing differs between prod and the spec | Anything differs; the dashboard doesn't exist yet; a reference is missing; the slug holds a dashboard Chartwright didn't create | +| `apply` | The dashboard is built and every chart's query ran | A step failed (the report's `stage` names it); design findings, with `--design strict` | +| `advise` | No error findings (and no warnings, with `--strict`) | Findings at that level | + +Because `plan` exits 1 both when the dashboard would change and when it couldn't compare, the review job runs `check` first, which fails on sign-in, profile and missing-reference errors. It then accepts plan's exit 1 only when the report's `dashboard` field says `create` or `update`. A report with `blocked`, or with no `dashboard` field at all, fails the job. When `blocked` comes from the spec naming something prod doesn't have, the report lists it in `resolution_errors`, the same list `check` gives. + +## 4. Gate on design + +- `chartwright advise --strict spec.json` runs the design review offline and exits 1 on any error or warning; without `--strict`, only errors fail it. +- `chartwright apply --design strict` applies the same threshold and stops before it changes anything on the instance. `check --design strict` adds the gate to `check`. +- To keep a finding you chose deliberately, add its rule id to the spec's `design.ignore` list. The rules are in [DESIGN-BRAIN.md](DESIGN-BRAIN.md). +- Where your organisation keeps shared rules in a `standards/` folder, a rule a standard locks can't be ignored that way: record the exception in `standards/waivers.yaml`, with an owner, a reason and an expiry ([DESIGN-BRAIN.md](DESIGN-BRAIN.md), §18). + +## 5. Catch edits made in the Superset UI + +Run the drift job nightly. `plan` exits 1 when prod no longer matches a spec, so the job fails and the run summary shows what changed: + +- charts and filters added, changed or removed in the UI; +- title, layout, cross-filtering and series-colour changes; +- dashboard CSS and the dashboard's settings: colour scheme, description, certification, published or draft, refresh interval, filter bar and chart timestamps, and owners, tags and theme when the spec sets them; +- filters and filter scopes overwritten by a browser tab left open on an older copy of the dashboard ([tested on all three versions](VERIFICATION.md)). + +To undo those edits, re-run the deploy job or merge the next change: `apply` writes the spec back over them and keeps each chart's id. To keep an edit, copy it into the spec first; `chartwright absorb` does that for chart heights. + +Each apply to an existing dashboard backs it up first. On a CI runner that backup would vanish with the runner, so the deploy jobs set `CHARTWRIGHT_BACKUP_DIR` and upload it as an artifact, together with the record beside each ZIP that names its instance. To restore one, see [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md). + +## Limits + +- `plan` compares the settings a spec can hold. A UI edit to anything else, such as a rolling average or a forecast added to a chart, isn't reported and the next apply resets it, so make lasting changes in the spec. +- A chart added to a managed dashboard in the UI shows in `plan` as removed, and the next apply takes it off the dashboard, names it in a warning and leaves the chart itself in Superset's Charts list: add charts in the spec. +- A spec names each dataset by its database connection, schema and table, so staging must use prod's names for the same workflow to deploy to both. Where names differ, keep a copy of the specs per instance and point each deploy job at its own folder. +- The backup artifacts hold dashboard, chart and dataset definitions, and anyone who can read the repository's workflow runs can download them; keep the repository private, or drop the upload steps and run deploys from a machine you control. +- Chartwright signs in with a Superset account's own password (database or LDAP login) or with Preset API tokens, never through SSO or OAuth. On an instance where people sign in through SSO, ask your admin for a deploy account that has a Superset password. + +## Related + +- [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md): backups, automatic restore and `chartwright restore` +- [MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md): profiles, sign-in options and what must exist on each instance +- [VERIFICATION.md](VERIFICATION.md): how determinism, drift detection and repair are tested +- [CONTRACTS.md](CONTRACTS.md): the Superset behaviour `apply` and `plan` are built around, cited to source +- [LIMITS.md](LIMITS.md): every limit, with what to do instead diff --git a/docs/FEATURES.md b/docs/FEATURES.md index a2f0e00..629f91f 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -4,6 +4,12 @@ Chartwright builds Apache Superset dashboards from a small file called a spec, updates them from that same file later, and decompiles existing dashboards back into one. Everything below works from that one file. +Step-by-step guides: [move between instances](MOVE-BETWEEN-INSTANCES.md) · +[clone a dashboard](CLONE-A-DASHBOARD.md) · [AI agents](AI-AGENTS.md) · +[dashboards from code](DASHBOARDS-FROM-CODE.md) · [deploy from git](DEPLOY-FROM-GIT.md) · +[back up and roll back](HISTORY-AND-ROLLBACK.md) · [Superset versions](SUPERSET-VERSIONS.md) · +[limits](LIMITS.md). + ## Creating Dashboards with AI - **Context to Dashboard**: Ask in plain words; the dashboard is built from what is in front of you. - The analysis you just ran, a KPI contract document, a metrics definition page. diff --git a/docs/HISTORY-AND-ROLLBACK.md b/docs/HISTORY-AND-ROLLBACK.md new file mode 100644 index 0000000..6873fab --- /dev/null +++ b/docs/HISTORY-AND-ROLLBACK.md @@ -0,0 +1,99 @@ +# Back up and roll back dashboards + +Roll a dashboard back to how it was before any apply, including its charts' settings and its filter scopes. Every `chartwright apply` to a dashboard that already exists first saves the live dashboard as a ZIP, with no step on your part, and `chartwright restore` puts that ZIP back. + +## 1. Find the backup + +Each backup is saved at: + +```text +~/.config/chartwright/backups///.zip +``` + +- The timestamp is the local time of the apply, to the microsecond, such as `20261003T141502.123456.zip`, so the newest backup sorts last and no backup is ever overwritten. +- Beside each ZIP, a `.json` record names the instance it came from (its address), the profile, the slug and the dashboard id. Keep the two together: `restore` reads the record to check the instance. +- `apply` keeps the newest 50 backups of each dashboard per profile and deletes older ones, each ZIP with its record. Set `CHARTWRIGHT_BACKUP_KEEP` to keep more, or to 0 to keep them all. +- Set `CHARTWRIGHT_BACKUP_DIR` to keep backups somewhere else; the `//` folders are added under it. +- The JSON report from `apply` gives the file's path in its `backup` field. +- On macOS and Linux the default folder is readable only by you. If you set `CHARTWRIGHT_BACKUP_DIR`, protect that folder yourself: the ZIPs hold dashboard, chart and dataset definitions. + +`apply` saves the backup after it has confirmed every reference and that it owns the dashboard, and before it changes anything. The backup holds the dashboard exactly as it was live, including any edits made in the Superset UI since the last apply, so you can get back a UI edit that an apply overwrote. + +## 2. Know when apply restores it for you + +`apply` runs in steps: resolve, ownership, prepare, import, linkage, scope, owners and smoke. When a step fails, `apply` exits 1 and its report's `stage` names the step. Whether the backup goes back on its own depends on the step: + +| What fails | What's live afterwards | +|---|---| +| Superset refuses the import | The backup, restored automatically | +| Anything fails while charts that left the spec are deleted, while datasets are read, or during the import, whether Superset rejects a request, the connection drops or Chartwright hits a bug | The backup, restored automatically | +| Superset rejects an update to a chart that already existed, or the connection drops during it | The backup, restored automatically | +| The linkage step finds charts missing from the dashboard, or can't take off a chart the spec doesn't name (one added in the UI) | The new state; restore the backup by hand | +| Filter scopes or the spec's owners can't be set, or a chart's query fails in the smoke step | The new state; restore the backup by hand | + +An automatic restore is noted in the report's `warnings`; it also deletes any chart the failed apply had just created. If the automatic restore fails too, the warning names the backup to pass to `chartwright restore`. + +## 3. Restore a backup by hand + +```bash +ls ~/.config/chartwright/backups/prod/orders/ +chartwright restore ~/.config/chartwright/backups/prod/orders/20261003T141502.123456.zip --profile prod +``` + +`restore` does more than re-import the ZIP: + +- It confirms the backup holds a dashboard Chartwright built, or is one of its own backups in that profile's backup folder (which covers a dashboard you adopted), and refuses any other ZIP. +- It reads the record beside the ZIP and refuses a backup taken on another instance than the profile points at, naming both. Pass `--to-other-instance` to restore it there on purpose, for example onto a rebuilt instance at a new address. A backup with no record (from before 0.5.0) restores with a warning that the instance couldn't be checked. +- It imports the dashboard: its title, layout and filters as they were, linked to exactly the backup's charts. +- It writes each chart's settings and saved query back from the backup and points the chart at its backed-up dataset. Superset 6.1.0's importer leaves charts that already exist unchanged ([CONTRACTS.md](CONTRACTS.md), "When a bundle is imported"; [#34879](https://github.com/apache/superset/issues/34879), fixed after 6.1.0, not yet released), so a plain re-import brings back the dashboard but not its charts' settings. +- It deletes the charts Chartwright created after the backup that end up on no dashboard, and names any other chart it took off. +- It puts filters that apply to named charts back onto those charts. +- It prints a JSON report listing the charts it restored, and exits 0 on success and 1 on failure, naming any chart whose settings it couldn't put back. + +The [fault-injection tests](VERIFICATION.md) alter charts' settings and filter scopes, restore the backup, then require `plan` against the old spec to come back clean. + +## 4. Then revert the spec + +`restore` changes the dashboard in Superset, not your spec, so the next `apply` would build the newer version again. Revert the spec in git to match: + +```bash +git revert # or: git checkout -- dashboards/orders.json +chartwright plan dashboards/orders.json --profile prod +``` + +`plan` exits 0 when the restored dashboard matches the reverted spec. If it reports differences, the backup most likely held UI edits the spec never had: copy them into the spec, or apply the spec to replace them. When CI deploys your specs ([DEPLOY-FROM-GIT.md](DEPLOY-FROM-GIT.md)), merging the revert applies the reverted spec, which also rolls the dashboard back, rebuilt from the spec rather than from the backup. + +## 5. Undo an edit made in the Superset UI + +To undo a UI edit made since the last apply, apply the spec again: + +```bash +chartwright plan dashboards/orders.json --profile prod # shows what the UI edit changed +chartwright apply dashboards/orders.json --profile prod # writes the spec back over it +``` + +Backups are taken only when `apply` runs, so no backup holds the state from before a UI edit. The spec does, and that apply also backs up the edited dashboard first, in case you want the edit back. + +## Compared with re-importing a Superset export + +- In Superset 6.1.0, re-importing an export updates the dashboard but leaves its existing charts' settings unchanged (#34879 above). `restore` writes them back. +- Superset 6.1.0 keeps no copy of a dashboard before it changes; you export by hand first. Chartwright saves one before every apply to an existing dashboard. Superset's master branch adds version history and soft delete, not yet in a release. +- Re-importing an export after deleting its assets can fail ([#44309](https://github.com/apache/superset/issues/44309), open). `restore` imports through the same importer and may fail the same way when a dataset the backup's charts read has been deleted; recreate the dataset in Superset and apply the spec instead. + +## Limits + +- A backup covers one dashboard and its charts. To back up a whole instance, back up Superset's metadata database. +- The first apply at a slug saves no backup, since there was nothing to save; to undo it, delete the dashboard and its charts in Superset. +- `restore` deletes only the charts Chartwright created that it leaves on no dashboard. A chart made in Superset, or one that also sits on another dashboard, stays; the report names it, and you can delete it in Superset's chart list. +- `restore` saves no backup of the state it replaces; to return to the spec's version, apply the spec. +- A backup taken on Superset 6.1.0 doesn't import on 4.1.4 or 5.0.0 ([CONTRACTS.md](CONTRACTS.md)); after a downgrade, apply the spec instead. +- A backup written on a CI runner disappears with the runner; upload it as an artifact with its record, as the [deploy workflow](DEPLOY-FROM-GIT.md) does, and fetch it with `gh run download -n backups-prod -D backups`. The download keeps the `//` folders, so with `CHARTWRIGHT_BACKUP_DIR=backups` set the ZIP sits in that profile's backup folder and `restore` accepts it, which a dashboard you adopted needs. +- The MCP server has no restore tool; run `chartwright restore` from the command line. + +## Related + +- [DEPLOY-FROM-GIT.md](DEPLOY-FROM-GIT.md): review changes in pull requests, deploy from CI and compare prod with the specs nightly +- [MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md): profiles, and the steps `apply` runs +- [VERIFICATION.md](VERIFICATION.md): how backups, automatic restore and `restore` are tested +- [CONTRACTS.md](CONTRACTS.md): Superset's importer behaviour, cited to source +- [LIMITS.md](LIMITS.md): every limit, with what to do instead diff --git a/docs/LIMITS.md b/docs/LIMITS.md new file mode 100644 index 0000000..ba3a8d1 --- /dev/null +++ b/docs/LIMITS.md @@ -0,0 +1,53 @@ +# Limits + +Each limit below comes with what to do instead, so you can tell whether Chartwright fits your Superset setup before you point it at an instance you care about. It starts with how Chartwright decides which dashboards and charts it may change, because several of the limits follow from that. + +## How Chartwright decides what it owns + +- **The slug decides which dashboard a spec manages.** Chartwright derives a dashboard's uuid from its slug, and a chart's uuid from the slug plus the chart's name. It doesn't record which person, team or file built a dashboard, so two specs with the same slug manage the same dashboard. A spec written by `chartwright adopt` is the exception: its `dashboard.adopted` block names a dashboard made in the UI, and its charts, by their own uuids. +- **It changes only what it built or you adopted.** `apply` refuses to touch a dashboard at the spec's slug that Chartwright didn't create, and says to pick another slug or take the dashboard over with `chartwright adopt`. `absorb` and `restore` follow the same rule. +- **Charts that leave the spec are deleted.** When you remove or rename a chart in the spec, the next `apply` deletes the chart it had built, even if someone also added that chart to another dashboard. Charts Chartwright didn't build are never deleted: in an adopted spec, a chart that was there when you adopted the dashboard is only taken off it, and one renamed together with its key in `dashboard.adopted.charts` keeps its id; a chart Chartwright built after adoption is deleted only when no other dashboard uses it. +- **Give each team its own slug prefix**, such as `finance-` or `ops-`, so that two teams' specs never share a slug. + +## Every limit, with what to do instead + +| What's true today | What to do instead | +|---|---| +| **Sign-in:** Chartwright signs in through Superset's username-and-password login API (database or LDAP accounts) or with Preset API tokens. It can't complete an SSO or OAuth sign-in, and accounts created through SSO have no Superset password | Use an account that has a database or LDAP password. On an instance set up for SSO, ask its admin whether the login API accepts such an account. If it doesn't, use an instance that allows database or LDAP login, or a Preset workspace with API tokens | +| **Roles:** all testing signs in with the Admin role. Restricted roles are untested | Use an Admin account, such as a dedicated service account, or try your own role on a test instance first | +| **Datasets:** datasets and database connections must already exist on each instance. Chartwright reads them and never creates them | Create them on each instance first, in Superset or with the tooling you already use. `chartwright check` lists every missing one | +| **Connection names:** each dataset is matched by its exact database connection name and table (and schema, when the spec gives one). There's no per-instance mapping | Where names differ between instances, keep one copy of the spec per instance, or rewrite the `dataset` blocks with a script ([MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md)) | +| **UI edits:** every `apply` writes the spec over the dashboard. Layout, chart settings, dashboard CSS and the dashboard's settings go back to the spec's, and filters added in the UI are removed. Tags and a theme the spec leaves out stay as they are, and so do owners, except that each apply adds the account that applies | Make lasting changes in the spec. Run `chartwright plan` first to see what someone changed in Superset, and `chartwright absorb` to copy dragged chart heights into the spec | +| **Charts added in the UI:** the next `apply` takes a chart added in the UI off the dashboard, names it in a warning, and leaves the chart itself in Superset's Charts list | Add new charts to the spec. To keep one that was added in the UI, describe it in the spec: the next apply builds Chartwright's own chart from it, and you can delete the UI-made one | +| **Chart names:** a chart's name is its identity. Renaming a chart in the spec deletes the old chart and creates a new one with a new id (in an adopted spec, rename its key in `dashboard.adopted.charts` too, and the chart keeps its id) | Settle chart names before reports or links point at a chart. `plan` shows a rename as one chart removed and one added (in an adopted spec, as one chart changed) | +| **Decompile:** `decompile` lists what it can't carry over in its `losses`, including chart settings changed from Superset's defaults, but drops a few dashboard settings without a note ([listed below](#settings-decompile-drops-without-a-note)) | Before you retire the original, compare it with the copy for those settings | +| **Taking over a dashboard made in the UI:** the first apply of an adopted spec resets what a spec can't hold, such as a forecast on a chart or per-chart cross-filter scopes | `chartwright adopt` lists every such reset and refuses until you pass `--accept-reset`. For a setting you need, rework the chart with fields the spec has (`chartwright schema` lists them), or leave that dashboard unadopted; run `plan` before the first apply | +| **Decompile input:** `decompile` reads a live dashboard, not a ZIP export | Import the ZIP into a test instance and decompile it there | +| **Chart types and queries:** 17 chart types. Metrics are a saved metric, SUM, AVG, COUNT, COUNT_DISTINCT, MIN or MAX of a column, or a custom SQL expression (`SQL(...) AS Label`); a chart filter is a WHERE condition or custom SQL. The filter bar has five filter types: value picker, time range, numeric range, time grain and time column. `check` can't look inside custom SQL, so the columns it names are listed under `unchecked_sql` | For anything else, save the SQL as a metric or a virtual dataset in Superset, and name it in the spec | +| **`plan` detail:** `plan` names each changed chart but not the setting that changed (on an adopted spec it lists the changed settings under `chart_option_changes`). It compares only settings a spec can express | To see which setting changed, run `chartwright decompile --profile -o live.json` and diff `live.json` against your spec. Keep styling in the spec (`css`, `label_colors`, colour rules, text blocks) | +| **Heights from the UI:** `absorb` copies heights only | Set widths in the spec: a chart's `width`, or its run length in a `sketch` | +| **Design fixes on a sketch:** `advise --fix` writes an explicit height that overrides the sketch, so the drawing no longer matches | To keep the drawing true, make a chart taller by repeating its line in the sketch | +| **MCP server:** it has no `restore`, `absorb`, `calibrate` or `compile` tool. It does save a backup before every build of an existing dashboard | Run those from the command line, for example `chartwright restore --profile ` | +| **One dashboard per spec:** each spec describes one dashboard, and each command takes one spec, apart from the `standards` commands, which take a folder | Loop over a folder of specs in a shell script or a CI job | +| **Backups:** they're kept under `~/.config/chartwright/backups///` (or under `CHARTWRIGHT_BACKUP_DIR`), the newest 50 per dashboard. A backup holds one dashboard and its charts, with a record of the instance it came from; `restore` refuses one from another instance | Set `CHARTWRIGHT_BACKUP_KEEP` to keep more, or 0 to keep all; pass `--to-other-instance` to restore onto another instance on purpose. To back up the whole instance, back up Superset's metadata database | +| **Automatic restore:** it runs only when something fails while `apply` prepares or runs the import, in-place chart updates included. If the linkage check, setting the filter scopes or owners, or a chart query fails, the new version stays live | Run `chartwright restore` with the backup path printed in the `apply` report | +| **Embedding:** embedding settings aren't in the spec, and whether they survive `apply` is untested | Turn embedding on in Superset, re-apply on a test instance, and confirm the embedded dashboard still loads before you rely on it | +| **Row-level security:** datasets filtered by row-level security are untested | Apply on a test instance with your row-level security rules and role, and check each chart's result in the `apply` report | +| **Template variables:** a spec has none | Write one spec per copy with a script that changes the slug and the `dataset` blocks ([CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md)) | +| **Deleting dashboards:** there's no delete command, and changing a spec's slug builds a new dashboard while the old one stays (an adopted spec refuses a new slug) | Delete the dashboard in Superset, and its charts from Superset's chart list | +| **Scripting:** there's no documented Python API | Call the command line and read its JSON output and exit codes, or use the MCP server | +| **Superset releases:** tested on 4.1.4, 5.0.0 and 6.1.0 only | On any other release, apply your specs on a test instance first ([SUPERSET-VERSIONS.md](SUPERSET-VERSIONS.md)) | +| **Databases:** CI tests use Superset's example datasets, and Chartwright is also used against one production warehouse. Other database engines are untested | Apply on a test instance first. `apply` queries every chart, and its report names any chart that errors or comes back empty | + +## Settings decompile drops without a note + +- **Dashboard:** per-chart and dashboard-wide cross-filter scopes, charts exempt from auto-refresh, charts shown expanded, staggered refresh and the colour namespace. `chartwright adopt` lists each of these it finds as a reset. + +## Related + +- [SUPERSET-VERSIONS.md](SUPERSET-VERSIONS.md): what differs between 4.1.4, 5.0.0 and 6.1.0, and how to rebuild after an upgrade +- [CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md): decompile, and copies per team or tenant +- [MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md): profiles, sign-in and connection names +- [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md): backups and `restore` +- [VERIFICATION.md](VERIFICATION.md): how it's tested, and the coverage limits +- [CONTRACTS.md](CONTRACTS.md): the Superset behaviour Chartwright relies on, cited to source diff --git a/docs/MOVE-BETWEEN-INSTANCES.md b/docs/MOVE-BETWEEN-INSTANCES.md new file mode 100644 index 0000000..2380707 --- /dev/null +++ b/docs/MOVE-BETWEEN-INSTANCES.md @@ -0,0 +1,169 @@ +# Move dashboards between instances + +Keep one spec file per dashboard and build the same dashboard on dev, staging, prod or any other Superset instance from it. To move a dashboard someone built in the UI, adopt it first ([below](#take-over-a-dashboard-made-in-the-ui)), then build it on each instance. Before each move you can check the target without changing it, and each apply backs up the dashboard it replaces. + +## 1. Add a profile for each instance + +Profiles live in `~/.config/chartwright/profiles.toml`, or in the file the `CHARTWRIGHT_PROFILES` environment variable names. A profile gives the instance's address and says where its password comes from. The file holds no passwords, tokens or secrets, so you can commit it next to your specs. + +```toml +[dev] +base_url = "https://superset-dev.example.com" +username = "chartwright" +password_env = "SUPERSET_DEV_PASSWORD" + +[staging] +base_url = "https://superset-staging.example.com" +username_env = "SUPERSET_DEPLOY_USER" +password_env = "SUPERSET_STAGING_PASSWORD" +password_cmd = ["op", "read", "op://Work/superset-staging/password"] + +[prod] +base_url = "https://superset.example.com" +username_env = "SUPERSET_DEPLOY_USER" +password_env = "SUPERSET_PROD_PASSWORD" +auth_provider = "ldap" +ca_bundle = "/etc/ssl/certs/corp-root-ca.pem" +``` + +Pass the profile name to every command that talks to Superset: `--profile staging`. + +| To sign in with | Put in the profile | +|---|---| +| A password from an environment variable | `password_env`, naming any variable you already use | +| A password from your password manager | `password_cmd`, a command that prints the password; the list form runs without a shell, and `~` is expanded | +| Both | Set both, as `staging` does: the variable is used when it's set and the command runs otherwise, so CI can export the variable while laptops read the password manager | +| A username from the environment | `username_env` in place of `username` | +| LDAP | `auth_provider = "ldap"` (the default, `"db"`, is Superset's database login) | +| Preset API tokens | `api_token_env` and `api_secret_env`, naming the variables that hold the token and the secret; `base_url` is the workspace address | +| The login preset-cli already saved | `preset_credentials_file`, the path to the `credentials.yaml` preset-cli wrote | +| A corporate root certificate | `ca_bundle`, the path to the certificate file (the `REQUESTS_CA_BUNDLE` variable works too) | +| A proxy | The standard `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` environment variables, which every request follows | + +A Preset profile looks like this: + +```toml +[preset] +base_url = "https://..app.preset.io" +api_token_env = "PRESET_API_TOKEN" +api_secret_env = "PRESET_API_SECRET" +``` + +If a sign-in fails, the error names the cause: an unset variable, a failed command, a certificate problem (with the `ca_bundle` hint), a proxy error, or a login page from an SSO portal or proxy answering in place of Superset's login API. + +## 2. Create the data on the target + +A spec carries the dashboard. The data it reads stays in Superset, so each instance needs it before the first move. + +| Moves with the spec | Must already exist on the target | +|---|---| +| Title, slug, layout, tabs, header and footer rows, and markdown text | The database connection, under the name the spec uses | +| Every chart and its settings | Each dataset, matched by connection name, table and, when the spec gives one, schema | +| The filter bar with its defaults and chart scopes, the cross-filter setting, series colours | The columns and saved metrics the spec names | +| Dashboard CSS, colour scheme, description, certification, published or draft, refresh interval and tags | The owners the spec names, as accounts on the target, and the theme it names (Superset 6.0 and later) | + +Chartwright reads datasets and connections and never creates them. Create them in Superset, or with the tooling you already use for them, before you apply. When a spec leaves out `schema`, a table found in two schemas of the same connection is reported as ambiguous; add the schema to the spec. + +## 3. Check the target before you move + +```bash +chartwright check orders.json --profile prod --design off +chartwright plan orders.json --profile prod +``` + +`check` signs in, looks up every dataset, column and saved metric the spec names, and changes nothing (`--design off` leaves the design review out of its output). It exits 0 when everything resolves. Otherwise it exits 1 and lists every missing reference at once: + +- a missing dataset comes with any tables of the same name in other connections or schemas, which usually points at a naming difference; +- a missing saved metric comes with the metrics the dataset does have; +- a missing column comes with up to three close matches, such as `region` for `Region`, and the dataset's columns. + +`plan` shows what an apply would change on that instance: `create` when the dashboard isn't there yet, `blocked` when the slug belongs to a dashboard Chartwright didn't build or the spec names something the instance doesn't have or can't take, and otherwise the charts and filters added, changed and removed, plus title, layout, cross-filter, series-colour, CSS and dashboard-setting changes, including edits someone made in the UI there. It exits 0 when there's nothing to change and 1 otherwise. For a dashboard that doesn't exist yet, `plan` skips the reference lookup, so run `check` first. + +## 4. When connection names differ between instances + +A spec names each connection literally, and Chartwright has no per-instance name mapping. Where prod calls the connection something else, give prod its own copy of the spec, generated from the one you edit: + +```bash +jq '(.. | objects | select(has("table")) | select(.database == "warehouse") | .database) = "warehouse_prod"' \ + orders.json > orders.prod.json +chartwright apply orders.prod.json --profile prod +``` + +This renames the connection in every chart's and filter's `dataset` block and leaves the rest alone. For schema names, run the same command with `.schema` in both places. Keep the slug the same in every copy, so each instance has one dashboard at the same address. Generate the copy in your deploy job, or commit it and change both files in the same pull request. + +## 5. Apply + +```bash +chartwright apply orders.json --profile staging +chartwright apply orders.json --profile prod +``` + +The first apply on an instance creates the dashboard; each later apply of a spec with that slug updates it. `apply` runs these stages in order and stops at the first one that fails. Its JSON report names the stage and the backup file. + +| Stage | What happens | +|---|---| +| resolve | Each dataset, column and saved metric the spec names is looked up on this instance, as are the owners and theme it names, and every field is checked against the instance's Superset release. If anything is missing or can't be taken, all of it is listed and nothing is written | +| ownership | The apply stops if the slug belongs to a dashboard Chartwright didn't build or you didn't adopt | +| prepare | If the dashboard exists, it's exported to a backup file first, including any edits made in the UI. Charts Chartwright built that have left the spec are deleted, even if someone also added them to another dashboard | +| import | New charts are created. Charts Chartwright built before are updated in place and keep their ids | +| linkage | A chart the spec doesn't name, such as one added in the UI, is taken off the dashboard; then the charts on it are compared with the spec, because Superset's import can report success with charts missing | +| scope | Filters limited to named charts are pointed at those charts' ids on this instance | +| owners | The owners the spec names are set, when it names any | +| smoke | Every chart's query runs. An error fails the apply; a chart that returns no rows is a warning naming it | + +- If anything fails during the prepare or import stage, including the in-place chart updates, the backup is restored automatically, whatever the error. +- If linkage, scope, owners or smoke fails, the new version stays live. Run `chartwright restore --profile prod` to put the backup back, chart settings and filter scopes included. +- Backups are saved at `~/.config/chartwright/backups///.zip`, or under the folder `CHARTWRIGHT_BACKUP_DIR` names, each with a record of its instance beside it; the newest 50 of each dashboard are kept ([HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md)). + +Ownership comes from the slug: the dashboard's uuid is derived from its slug, and each chart's from the slug and the chart's name, unless the spec adopted a dashboard (below). Two specs with the same slug manage the same dashboard, so give each team its own slug prefix. + +## Take over a dashboard made in the UI + +Adopt it on the instance where it lives. The spec that `adopt` writes names the dashboard and its charts by their own ids, so applying it updates the same dashboard: its address, id and chart ids stay, and links and embeds keep pointing at it. + +1. Run `chartwright adopt regional-sales --profile dev -o regional-sales.json` (the slug or numeric id). It changes nothing in Superset. + - If the first apply would reset settings a spec can't hold, such as a forecast on a chart or per-chart cross-filter scopes, it lists each under `resets` and refuses. For a setting you need, rework the chart with fields the spec has (`chartwright schema` lists them), or leave the dashboard unadopted; pass `--accept-reset` to accept the rest. + - If some charts also sit on other dashboards, it refuses, since applying changes them there too: give those dashboards their own copies in Superset, or pass `--allow-shared`. +2. Run `chartwright plan regional-sales.json --profile dev`. It names each chart whose stored settings the first apply rewrites, under `chart_option_changes`. +3. Run `chartwright apply regional-sales.json --profile dev`. It backs the dashboard up first, as always. + +The adopted spec names the dashboard by its own id, so it applies wherever that dashboard is: on dev, and on any instance it reached through a Superset export. Where the slug is free, write a copy without the `dashboard.adopted` block and apply that copy, as in section 5; that instance gets a dashboard of its own at the same slug. The copy keeps the spec's `owners` and `theme`, so those accounts and that theme must exist there too, or edit them first: + +```bash +jq 'del(.dashboard.adopted)' regional-sales.json > regional-sales.copy.json +chartwright apply regional-sales.copy.json --profile staging +``` + +To build a separate copy beside the original instead, decompile it ([CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md#start-from-a-dashboard-you-already-have)), which also lists what `decompile` carries over and what it can't. + +## Compared with re-importing ZIP exports + +The first three Superset facts are cited to source in [CONTRACTS.md](CONTRACTS.md), under "When a bundle is imported" and "Differences the tool absorbs". + +- **Chart changes arrive.** Up to Superset 6.1.0, re-importing a dashboard export updates the dashboard but leaves charts that already exist on the target unchanged ([#34879](https://github.com/apache/superset/issues/34879), fixed after 6.1.0 and not yet released). Chartwright updates the charts it built through Superset's chart API, so new settings land and chart ids stay the same. +- **Missing charts are caught.** Superset 6.1.0 skips a chart whose dataset isn't in the bundle and still reports success. Chartwright adds the target's own dataset records to every import, then checks that every chart in the spec is on the dashboard. +- **Older releases are reachable.** A Superset 6.1.0 export doesn't import into 4.1.4 or 5.0.0. Chartwright builds from the spec and the target's own dataset records, and CI applies the same specs to all three releases; a field an older release can't take, such as tags or a theme before 6.0, is refused before anything is written ([SUPERSET-VERSIONS.md](SUPERSET-VERSIONS.md)). +- **There's a copy to go back to.** Superset 6.1.0 has no version history for dashboards (it's on Superset's master branch, not yet released). Chartwright saves a backup before every apply. + +## Limits + +| What's true today | What to do instead | +|---|---| +| Chartwright signs in with a Superset account's own password, through database or LDAP login, or with Preset API tokens, never through SSO or OAuth | On an instance where people sign in through SSO, ask your admin for an account that has a Superset password | +| CI's live runs sign in with a database login; LDAP and Preset sign-in aren't part of them | Run `chartwright check` against the instance first: it signs in and only reads | +| Datasets and connections must already exist on the target | Create them in Superset, or with your existing tooling, before the first apply | +| Connection and schema names are matched exactly, with no per-instance mapping | Generate a copy of the spec per instance, as in step 4 | +| The first apply of an adopted dashboard resets what a spec can't hold | `chartwright adopt` lists every such reset first and refuses until you pass `--accept-reset`, as above | +| Every apply overwrites edits made in the UI on that instance, dashboard CSS and settings included | Run `plan` first to see the edits, then copy the ones to keep into the spec; `chartwright absorb` copies chart heights set in the UI (heights only) | +| A chart added in the UI to a dashboard Chartwright manages is taken off it by the next apply (the chart stays in Superset's Charts list) | Add new charts to the spec | +| One dashboard per spec, and one spec per command (the `standards` commands take a folder) | Loop over a folder of specs ([DEPLOY-FROM-GIT.md](DEPLOY-FROM-GIT.md)) | +| A backup restores only onto the instance it came from | Pass `--to-other-instance` to `restore` to put it onto another one on purpose | + +## Related + +- [CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md): decompile in detail, and one copy per team or tenant +- [DEPLOY-FROM-GIT.md](DEPLOY-FROM-GIT.md): `check` and `plan` on pull requests, `apply` on merge +- [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md): backups, `restore` and checking for UI edits +- [SUPERSET-VERSIONS.md](SUPERSET-VERSIONS.md): one spec on 4.1.4, 5.0.0 and 6.1.0 +- [CONTRACTS.md](CONTRACTS.md): Superset's import behaviour, cited to source +- [VERIFICATION.md](VERIFICATION.md): how it's tested diff --git a/docs/STABILITY.md b/docs/STABILITY.md new file mode 100644 index 0000000..ab65771 --- /dev/null +++ b/docs/STABILITY.md @@ -0,0 +1,34 @@ +# Stability and deprecation + +What you can rely on when you upgrade Chartwright, and how a change that breaks something reaches you first. The promise below starts with version 1.0.0. Until then, versions are 0.x: a minor release may break something, and its changelog entry says what under "Upgrading from". + +## What stays compatible within a major version + +From 1.0.0, a 1.x release keeps these working as they did in every earlier 1.x release: + +- **Specs:** a spec that validates on 1.x validates on every later 1.x. New fields are optional. +- **Commands and flags:** every command and flag keeps its name and meaning, and every exit code keeps its meaning: 0 for success, 1 for a finding or a failure, and 2 for a command line the parser can't read. +- **JSON output:** every field a command prints keeps its name, type and meaning. New fields may appear, so read the fields you use and ignore the rest. +- **MCP tools:** every tool keeps its name, its parameters and the fields it returns. New tools, new optional parameters and new returned fields may appear. + +New capabilities arrive in minor releases (1.1, 1.2), fixes in patch releases (1.1.1). + +## How a removal reaches you + +Nothing on that list is removed without warning: + +1. A minor release marks it deprecated. Using it still works and prints a warning naming what replaces it. +2. It stays, deprecated and listed under "Deprecated" in the changelog, in every later 1.x release. +3. It goes only in the next major release (2.0.0), so at least one minor release warns you first; "Upgrading from 1.x" says what to change. + +## What changes outside that promise + +These follow Superset and good design practice, so they change in minor releases. Each change is in the changelog. + +- **Design findings:** the design review's rules, thresholds and wording improve over time, and a new rule can report something it didn't before. A pipeline that fails on warnings (`advise --strict`, `--design strict`) can therefore fail after an upgrade. The `design_brain` version in the advice output changes whenever findings can change; pin Chartwright's version in CI, and before you raise it, read that release's changelog entry for the design brain. +- **Supported Superset releases:** the releases Chartwright builds and tests against (today 4.1.4, 5.0.0 and 6.1.0, [how it's tested](VERIFICATION.md)) are added and retired as Superset releases, each change in the changelog. + +## Related + +- [CHANGELOG.md](../CHANGELOG.md): every release, with "Upgrading from" notes +- [LIMITS.md](LIMITS.md): what Chartwright doesn't do today, with what to do instead diff --git a/docs/SUPERSET-VERSIONS.md b/docs/SUPERSET-VERSIONS.md new file mode 100644 index 0000000..43768ab --- /dev/null +++ b/docs/SUPERSET-VERSIONS.md @@ -0,0 +1,132 @@ +# Superset versions and upgrades + +After you upgrade Superset, rebuild each dashboard from the spec you already have. The same spec applies to Superset 4.1.4, 5.0.0 and 6.1.0. After an upgrade, `chartwright apply` writes every chart's settings again from the spec, then confirms each chart is on the dashboard and runs its query. This works for dashboards that have a spec. For a dashboard someone made in Superset's UI, adopt it first (see [Dashboards made in the UI](#dashboards-made-in-the-ui)). + +## Why one spec works on all three releases + +Superset's backend doesn't define the settings a chart can have. Each chart type's frontend plugin defines them, and they change between releases: 6.1.0 renamed the big-number subtitle setting, for example. Chartwright deals with that as follows: + +- It writes the chart settings each release's plugins declare. The lists come from each plugin's source at 4.1.4, 5.0.0 and 6.1.0 (`tools/contracts/params-contract.json`). A setting only a newer release has is written only where an older release ignores it, and each such case is a named exception in the check below. +- CI compiles four example specs, which between them use all 17 chart types and the display and dashboard settings, and fails if any setting they produce is missing from a release's list without a named exception (`tools/params_drift.py`). +- `check`, `apply` and `plan` ask the instance for its release, then: + - refuse, before writing anything, a field it can't take: `tags` and `theme` before 6.0, `show_chart_timestamps` and a waterfall's `steps` before 6.1; + - warn about a field it ignores or draws differently: before 6.0, a trendline's `subtitle`, a table's `column_headers` and `hidden`, `show_value` on a stacked mixed chart, and a box plot's `row_limit`; before 6.1, `x_label_every`, a heatmap's `x_order` and `y_order` by value, a waterfall's labels, cell bars beside colour rules, and four colour-rule effects (the table below). +- `--superset-version` states the release when the instance doesn't report it, and `chartwright compile --superset-version 5.0.0` runs the same check offline. +- Each `apply` exports the dataset records from the instance it targets, so the bundle matches that instance. + +The Superset source behind each of these points is cited in [CONTRACTS.md](CONTRACTS.md) ("What chart options mean"). + +## Rebuild your dashboards after an upgrade + +1. **Before you upgrade, see what differs from each spec.** Run `chartwright plan orders.json --profile prod`. Exit 0 means the dashboard matches the spec. Exit 1 lists the charts and filters that were added, changed or removed (it also exits 1 when a reference is missing or sign-in fails), plus any title, layout, cross-filter, label-colour, CSS or dashboard-setting change. Edits made in the UI show up there too. The next `apply` overwrites them, so copy any edit you want to keep into the spec first. For chart heights dragged in the UI, `chartwright absorb orders.json --profile prod` does the copying. +2. **Try the new release on a test instance first.** Upgrade a test instance, then apply your specs there under the test instance's own profile. +3. **Upgrade production** with Superset's own upgrade steps. +4. **Check, then apply each spec:** + +```bash +# Before the upgrade: see what differs from each spec, including UI edits. +for f in specs/*.json; do + chartwright plan "$f" --profile prod +done + +# After the upgrade: check each spec's references, then rebuild it. +for f in specs/*.json; do + chartwright check "$f" --profile prod && chartwright apply "$f" --profile prod +done +``` + +| Step | What Chartwright checks or does | +|---|---| +| `plan` | Compares the live dashboard with the spec without changing anything. It names each changed chart, not the setting that changed, and it sees only settings a spec can express | +| `check` | Confirms that every dataset, column and saved metric the spec names, and the owners and theme it names, still exist on the upgraded instance, and that the release can take every field; lists every problem at once. Columns inside custom SQL, which it can't look up, are listed under `unchecked_sql` | +| `apply` | Saves a backup of the live dashboard. Updates each chart in place, so chart ids and links stay the same, writing every setting from the spec. Confirms every chart is linked to the dashboard, sets filter scopes, and queries every chart: a query error fails the run, and a chart with no rows is reported as a warning | + +When `apply` fails after the import and the chart updates, the new version stays live. The report gives the backup's path, and `chartwright restore --profile prod` puts the backup back. + +## What CI builds on each release + +On every pull request and every push to main, CI starts a real container of each release (4.1.4, 5.0.0 and 6.1.0) and runs these against it (`.github/workflows/ci.yml`, [VERIFICATION.md](VERIFICATION.md)): + +- **Builds of four specs**: an 18-chart spec with all 17 chart types, two tabs, a text block, WHERE filters on charts, two value pickers and a time range; a spec that sets the charts' display options, colour rules among them; one that sets the dashboard's settings, a header and a footer; and two waterfall bridges, which 4.1.4 and 5.0.0 must refuse. Each checks that every chart is linked and queries every chart, then applies the spec again to confirm chart ids don't change. +- **Other live checks**: a chart added in the UI taken off on the next apply, with restores matching their backups; a dashboard adopted in place; standards content written and read back; and, when the browser for them installs, locked text checked on the rendered page and each chart's query saved for reports. +- **25 cycles of random edits**: charts added, removed and renamed, filters added and removed, layouts switched between rows and tabs. After each cycle, `plan` must come back clean. +- **Stale-tab overwrites and injected faults**: these use a numeric range filter scoped to named charts. + +These spec fields have offline tests, but no live CI build uses them on any release: + +- **Layout:** `sketch` and sub-tabs. +- **Filters:** a value picker's `default` and `default_to_first`. +- **Colour:** `label_colors` and `cross_filters: true`. +- **Charts:** table `hidden`. + +## Differences between releases + +| What you use | On 6.1.0 | On 4.1.4 and 5.0.0 | +|---|---|---| +| `tags`, `theme` | Applied | Refused before anything is written (both need 6.0 or later) | +| `show_chart_timestamps` | Applied | Refused before anything is written | +| `x_label_every` | A label at every step | No effect, with a warning; Superset spaces the labels itself | +| A trendline's `subtitle`, a table's `column_headers` | Shown | No effect, with a warning | +| `show_value` on a stacked query of a mixed chart | Labels each stack's total | Labels every segment, with a warning; set `only_total: false` for the same labels on every release | +| `filter_bar_orientation: "horizontal"` | A filter bar across the top | Needs Superset's `HORIZONTAL_FILTER_BAR` feature flag | +| `owners` given as usernames | Found by username | Found by username only with Superset's `FAB_ADD_SECURITY_API` setting on; emails work on every release | +| Table `hidden` | The column is queried but not shown | The column is shown | +| Table colour rule with `apply_to` or `paint: "text"` | Paints another column, the whole row, or the text | These two settings aren't read, so the rule colours the tested value's own cell (from plugin source; not tested live) | +| Colour rule with any operator but `=` | A solid colour | The colour fades with distance from the threshold | +| A cell painted a dark colour | White text on it | Dark text, which a colour under 3:1 can make unreadable (`check` warns) | +| Cell bars beside colour rules on a table | Both drawn, except that a cell a rule fills loses its bar | No cell bars on any numeric column | +| A waterfall's `steps`, and its `total_label`, `increase_label` and `decrease_label` | The steps in your order, under your labels | `steps` refused before anything is written; the labels have no effect | +| A box plot's `row_limit` | Caps the rows | No effect, with a warning | +| A heatmap's `x_order` or `y_order` by value | Ranks each label by its total | No effect, with a warning | +| `label_colors` | Applied over the colour scheme | Unverified | +| Big-number `subtitle` | Shown. Chartwright writes the setting's older name, which 6.1.0 still displays | Shown | + +A spec can use settings only 6.1.0 reads and still apply to every release, because each setting it writes is declared by the table plugin of all three. This one does: + +```json +{ + "spec_version": "1", + "dashboard": {"title": "Order status", "slug": "ops-order-status"}, + "charts": [ + {"name": "Orders by region", "type": "table", + "metrics": ["SUM(amount) AS Revenue"], "groupby": ["region", "status"], + "hidden": ["status"], + "conditional_formatting": [ + {"metric": "Revenue", "operator": "<", "target": 1000, "color": "red", "apply_to": "row"} + ], + "dataset": {"database": "warehouse", "table": "orders"}} + ], + "layout": {"rows": [["Orders by region"]]} +} +``` + +On 6.1.0, a row with revenue under 1,000 turns red and the status column is hidden. On 4.1.4 and 5.0.0, the status column is shown and the rule colours only the Revenue cell, fading with distance from the threshold. + +## Moving from a newer Superset to an older one + +- **Superset exports:** a dashboard export from Superset 6.1.0 carries a `theme_uuid` field (`superset/commands/dashboard/export.py:165` at 6.1.0). The 4.1.4 and 5.0.0 import schemas don't declare that field, so their importers reject the export. Exports from an older release import into a newer one as they are ([CONTRACTS.md](CONTRACTS.md), "A 6.1.0 export does not import on older releases"). +- **Specs:** a spec applies to 4.1.4 and 5.0.0 the same way. Chartwright builds the bundle from the spec and takes the dataset records from the target instance. A field the older release can't take, such as `tags` or `theme`, is refused before anything is written, and one it ignores comes back as a warning (see the table above). +- **Backups:** a backup saved on 6.1.0 is a 6.1.0 export, so `restore` can't put it onto 4.1.4 or 5.0.0, and it refuses a backup from another instance anyway unless you pass `--to-other-instance`. Apply the spec there instead. + +## Dashboards made in the UI + +A dashboard with no spec gets none of this. Adopt it with `chartwright adopt --profile prod -o spec.json`: the spec updates that same dashboard, and adopt lists what its first apply resets ([MOVE-BETWEEN-INSTANCES.md](MOVE-BETWEEN-INSTANCES.md#take-over-a-dashboard-made-in-the-ui)). To build a copy beside the original instead, see [CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md). + +## Limits + +| What's true today | What to do instead | +|---|---| +| Tested on Superset 4.1.4, 5.0.0 and 6.1.0 only. On any other release, chart settings aren't checked against that release's plugins | Upgrade a test instance to that release and apply your specs there first. The linkage check and per-chart queries in the `apply` report show any chart that didn't land or doesn't return data | +| The CI build doesn't use every spec field (listed above) | Apply specs that use those fields on your test instance after the upgrade, and look at those charts before you upgrade production | +| `plan` names changed charts but not the setting that changed (on an adopted spec it lists them under `chart_option_changes`) | Run `chartwright decompile --profile prod -o live.json` and diff `live.json` against your spec | +| `plan` compares only settings a spec can express | Expect `apply` to reset the rest, and keep styling in the spec (`css`, `label_colors`, colour rules, text blocks) | +| Every `apply` overwrites edits made in the UI | Run `plan` before the upgrade and copy the edits you want into the spec | +| A failure after the import and the chart updates leaves the new version live | Run `chartwright restore` with the backup path from the report | + +## Related + +- [CLONE-A-DASHBOARD.md](CLONE-A-DASHBOARD.md): copy a dashboard, including one made in the UI +- [HISTORY-AND-ROLLBACK.md](HISTORY-AND-ROLLBACK.md): backups, `restore` and spotting UI edits +- [LIMITS.md](LIMITS.md): every limit, with what to do instead +- [CONTRACTS.md](CONTRACTS.md): the Superset behaviour Chartwright relies on, cited to source at each release +- [VERIFICATION.md](VERIFICATION.md): how each release is tested diff --git a/docs/images/sundown.webp b/docs/images/sundown.webp new file mode 100644 index 0000000..ec01aa9 Binary files /dev/null and b/docs/images/sundown.webp differ