diff --git a/README.md b/README.md index 04aba583..5e0449a0 100644 --- a/README.md +++ b/README.md @@ -608,6 +608,39 @@ mb content-translation upload --file translations.csv --profile prod --json | --------------- | ------------------------------------------------------- | | `--file ` | Complete content translation dictionary CSV (required). | +## Data sensitivity + +Propose a `data_sensitivity` label for every field of a table, a schema, or a database with Metabot's LLM through `/api/ee/data-sensitivity`. These commands require write access to the database, the `data_sensitivity` premium feature (Metabase v64+), and a configured AI provider. A scan is a dry run: nothing is written, the response is the proposal diffed against each field's current label. Apply one with `mb field update --body '{"data_sensitivity":"PII"}'`. + +Each field comes back with a status: `agree` (the proposal matches the current label), `disagree`, `new` (no current label yet), `abstain` (the model was unsure), or `dropped` (no usable answer). Text output prints a summary line and a table with one cell per LLM output, `current -> proposed` where they differ, covering every field with a changed label or semantic type; `--json` returns the whole result, including `usage` with `input_tokens` (the whole prompt, cache buckets included), `output_tokens`, `cache_creation_tokens`, `cache_read_tokens`, and `total_tokens`; the text summary breaks the input down by cache bucket when any of it was cached. Every scan spends provider tokens. A whole-database result is often larger than the default `--max-bytes`, so narrow it with `--status disagree,new` or `--schema`, raise `--max-bytes`, or scan one table at a time. + +| Flag | Description | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `--schema ` | `scan-db` only: scan only the tables in this schema. | +| `--status ` | Keep only fields with these statuses (`agree`, `disagree`, `new`, `abstain`, `dropped`). Text default: every field with a changed label or semantic type; JSON default: all. Counts and token usage stay the server's totals for the scan. | +| `--timeout ` | HTTP timeout for the single synchronous scan request (default 600000). | + +### `mb data-sensitivity scan-db ` + +Scan every active table of a database, or only those in one schema. The request is synchronous and runs as long as the scan, so raise `--timeout` for a large database. A table the server could not classify appears as one row with the error message in the Sensitivity cell, and the run still exits 0. + +```sh +mb data-sensitivity scan-db 1 +mb data-sensitivity scan-db 1 --schema public +mb data-sensitivity scan-db 1 --status disagree,new --json +mb data-sensitivity scan-db 1 --json --fields counts,failed +``` + +### `mb data-sensitivity scan-table ` + +Scan every active field of one table. + +```sh +mb data-sensitivity scan-table 3 +mb data-sensitivity scan-table 3 --status disagree,new --json +mb data-sensitivity scan-table 3 --json --fields counts,usage +``` + ## Cards CRUD plus query execution on `/api/card`. A "card" is a Metabase question, model, or metric. The `query` subcommand runs the card and either returns Metabase's JSON envelope or streams a raw CSV / XLSX export. diff --git a/packages/cli/skill-data/core/SKILL.md b/packages/cli/skill-data/core/SKILL.md index 49e0ebff..0d34956b 100644 --- a/packages/cli/skill-data/core/SKILL.md +++ b/packages/cli/skill-data/core/SKILL.md @@ -1,6 +1,6 @@ --- name: core -description: Foundations for driving Metabase from the terminal with the `mb` CLI — authentication and named profiles, the flag/output/`--json` conventions every command shares, JSON body input, command discovery via `--help` (add `--json` for machine-readable schemas), and the per-resource footguns (db, table, field, upload, content translation, card, dashboard, collection, segment, measure, timeline, alert, subscription, library, setting, search, eid). Load first for any `mb` task; it routes to the specialized skills for deeper work. +description: Foundations for driving Metabase from the terminal with the `mb` CLI — authentication and named profiles, the flag/output/`--json` conventions every command shares, JSON body input, command discovery via `--help` (add `--json` for machine-readable schemas), and the per-resource footguns (db, table, field, upload, content translation, data sensitivity, card, dashboard, collection, segment, measure, timeline, alert, subscription, library, setting, search, eid). Load first for any `mb` task; it routes to the specialized skills for deeper work. allowed-tools: Read, Write, Edit, Bash, AskUserQuestion --- @@ -11,8 +11,8 @@ The official Metabase CLI (`mb`) drives a Metabase instance over its REST API: a Top-level command groups (run `mb --help` to discover verbs): ``` -auth | db | table | field | upload | content-translation | query | card | dashboard | snippet | segment | measure | collection | library -document | timeline | timeline-event | transform | transform-job | transform-tag | alert | subscription | setting +auth | db | table | field | upload | content-translation | data-sensitivity | query | card | dashboard | snippet | segment | measure | collection +library | document | timeline | timeline-event | transform | transform-job | transform-tag | alert | subscription | setting search | git-sync | setup | eid | uuid | upgrade | skills ``` @@ -130,6 +130,7 @@ Routine verb shapes (list / get / create / update), every flag, and output schem - **field has no `list`.** Fields are per-table — get them via `table get --include fields`. Never enumerate fields across a whole db (context blow-up). `field summary` is live cardinality `{field_id, count, distincts}`; `field values` is the cached distinct set (`has_more_values: true` ⇒ truncated cache). `field update` patches metadata only (`base_type` isn't editable) — this is where you set a column's `semantic_type` or foreign-key target. - **upload (CSV → tables).** `upload csv --file ` creates a new table + model (prints `{model_id, table_id}`); `upload append ` / `upload replace --file ` add to / overwrite a table **previously created by upload** (columns must match). The destination db+schema is admin-configured, not per-call — check with `mb setting get uploads-settings --json` (`db_id: null` ⇒ uploads off/unconfigured; needs admin to read). `--collection ` only sets the model's collection. Max 50 MB. Errors: **"The uploads database is not configured."** = no db has uploads enabled; **"Uploads are not enabled."** = the append/replace target isn't an uploaded table. - **content-translation.** EE-only (`content_translation` premium feature), admin-only, and separate from Remote Sync. `content-translation download > translations.csv` streams the complete active dictionary; `content-translation upload --file translations.csv` replaces every active translation with the file's contents. Always upload the canonical complete CSV, never a partial patch. An empty dictionary downloads as Metabase's four-row sample dictionary — don't re-upload it as real translations. Metabase limits dictionaries to 1.5 MiB. +- **data-sensitivity.** EE-only (`data_sensitivity` feature, v64+, AI provider). `scan-db ` / `scan-table ` ask the LLM for a `data_sensitivity` label per field; dry run: spends tokens, writes nothing. Act on `--status disagree,new`; apply via `field update` (→ `metadata`). - **card.** `dataset_query` is the **flat** `mbql/query` value, not a legacy `{type:"query",query:…}` envelope (→ `mbql`). `--export-format csv|xlsx` streams the raw export (pipe to a file), bypassing the JSON envelope. `archive` is the only delete; unarchive with `update --body '{"archived":false}'`. `visualization_settings` keys are scoped by `display` and aren't pre-flighted — see `visualization`. - **dashboard.** Dashcards round-trip through `PUT /api/dashboard/:id` (no per-dashcard endpoint): `update-dashcard ` patches one safely; `update --body '{"dashcards":[…]}'` replaces the whole set (omitted ids are deleted server-side; negative ids for new cards). Every dashcard must include `card_id`, including existing rows; use `card_id:null` plus a `visualization_settings.virtual_card` block (`{display:"text"|"heading"|"link"|…}`) for non-question cards. `create` accepts the **same** `dashcards` array in its initial body, so lay out the whole dashboard in one call. `create`/`update` pre-flight every positive `card_id` and exit **2** with `{ok:false,errors:[…]}` on a bad ref (non-bypassable). `dashboard get ` (or `--full`) hydrates dashcards/tabs; `list` omits them. **The grid is 24 columns wide:** each dashcard's `{col, row, size_x, size_y}` is in grid units — **full-width is `size_x: 24`** (`size_x: 12` is half a row, the usual cause of a card filling only half the width). Keep `col + size_x ≤ 24`, start a full-width stack's `col` at 0, and don't overlap (the server stores collisions as sent — no auto-fix). Layout patterns and per-chart default sizes → the `dashboard` skill; load it before composing any `dashcards` array. - **dashboard parameters (filters).** A dashboard's `parameters` array holds its filter widgets; they're part of the dashboard record, so read them with `dashboard get --fields parameters --json` (no separate verb). **Editing replaces the _whole_ array** (like dashcards), so it's a read-modify-write loop and omitting a parameter deletes it. A parameter only filters a card once it is **mapped** onto that dashcard's `parameter_mappings` — an unmapped parameter is an inert widget. `type` is a **closed enum**; an unlisted value is a hard parse error that echoes the full allowed set back to you. `dashboard parameter-values [--query ]` fetches a widget's selectable values (`{values, has_more_values}`; `--query` is a case-insensitive substring search). Parameter types, ids, mapping targets, and value sources → the `dashboard` skill; load it before authoring a `parameters` array. diff --git a/packages/cli/skill-data/metadata/SKILL.md b/packages/cli/skill-data/metadata/SKILL.md index dfc42274..04887e98 100644 --- a/packages/cli/skill-data/metadata/SKILL.md +++ b/packages/cli/skill-data/metadata/SKILL.md @@ -27,6 +27,7 @@ This is the whole point of the skill. Each edit below is a key in the `field upd | `visibility_type: "details-only"` | hidden in table views, shown in the single-record detail view (for long blobs) | | `coercion_strategy: ` | **actually casts** the column — the only entry here that changes the value's type (below) | | `display_name` / `description` | the human label and help text shown everywhere | +| `data_sensitivity: "PII"` (or `"PCI_FIN"`, …, `"PUBLIC"`, `null`) | records how sensitive the column is — metadata only, no query or display effect; `null` means never scanned, `PUBLIC` reviewed and clean | `table update` carries the table-level equivalents: `display_name`, `description`, `visibility_type` (`hidden` / `technical` / `cruft` — hides the whole table from the builder), `field_order`, and `entity_type`. @@ -59,6 +60,14 @@ mb field update 42 --body '{"coercion_strategy":"Coercion/UNIXSeconds->DateTime" The full semantic-type catalog — every value grouped by the base type it attaches to, plus the `has_field_values` and `visibility_type` value tables and the exact writable-key lists — is in `references/semantic-types.md` (`mb skills get metadata --full`). +## Data sensitivity labels + +`data_sensitivity` is a per-column label (`SEC_KEY`, `SYS_TELEMETRY`, `PHI`, `BIO_GEN`, `PCI_FIN`, `SENS_PERS`, `PII`, `CORP_IP`, `BIZ_CONF`, `PUBLIC`, most severe first) recorded as metadata: it changes nothing about queries, formatting, or access. `null` means the column has never been scanned; `PUBLIC` means it was scanned or reviewed and nothing sensitive was found. A value a person set through `field update` is ground truth; a scanner reports where it differs but never overrides it. `mb data-sensitivity scan-table ` / `scan-db ` (EE, `data_sensitivity` feature, v64+) ask the LLM to propose a label per field as a dry run; apply one with: + +```bash +mb field update 42 --body '{"data_sensitivity":"PII"}' --profile --json +``` + ## Sync, scan, fingerprint — three different refreshes When a column looks stale or missing, know which one you need (`db` verbs, mechanics in `core`): diff --git a/packages/cli/src/commands/data-sensitivity/examples.test.ts b/packages/cli/src/commands/data-sensitivity/examples.test.ts new file mode 100644 index 00000000..43c5a013 --- /dev/null +++ b/packages/cli/src/commands/data-sensitivity/examples.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from "vitest"; + +import type { + DataSensitivityDatabaseResult, + DataSensitivityTableResult, +} from "@metabase/client/domain/data-sensitivity"; + +import { applyProjection } from "../../output/projection"; +import { + dataSensitivityDatabaseView, + dataSensitivityTableView, +} from "../../output/views/data-sensitivity"; +import { getMetabaseAugment } from "../../runtime/command-augment"; +import type { ResourceView } from "../../output/view"; + +import scanDb from "./scan-db"; +import scanTable from "./scan-table"; + +const TABLE: DataSensitivityTableResult = { + table_id: 3, + table_name: "PEOPLE", + schema: "PUBLIC", + database_id: 1, + model: "anthropic/claude-haiku-4-5-20251001", + requests: 1, + usage: { + input_tokens: 1, + output_tokens: 1, + cache_read_tokens: 0, + cache_creation_tokens: 0, + total_tokens: 2, + }, + sample_error: null, + counts: { fields: 0, agree: 0, disagree: 0, new: 0, abstain: 0, dropped: 0, semantic_changed: 0 }, + fields: [], +}; + +const DATABASE: DataSensitivityDatabaseResult = { + database_id: 1, + schema: null, + tables: [TABLE], + counts: TABLE.counts, + usage: TABLE.usage, + requests: 1, + failed: 0, +}; + +// The projection walks plain objects only, so an example advertising a path through `tables` or +// `fields` would throw for every user who copied it. +function fieldsPathsInExamples(cmd: object): string[][] { + const examples = getMetabaseAugment(cmd)?.examples ?? []; + return examples.flatMap((example) => { + const tokens = example.split(/\s+/); + const flagIndex = tokens.indexOf("--fields"); + const value = flagIndex === -1 ? undefined : tokens[flagIndex + 1]; + return value === undefined ? [] : [value.split(",")]; + }); +} + +function projectsEveryExample(cmd: object, fixture: T, view: ResourceView): void { + const paths = fieldsPathsInExamples(cmd); + expect(paths.length).toBeGreaterThan(0); + for (const fields of paths) { + expect(() => applyProjection(fixture, view, false, fields)).not.toThrow(); + } +} + +describe("data-sensitivity --fields examples", () => { + it("scan-db advertises only paths the projection can resolve", () => { + projectsEveryExample(scanDb, DATABASE, dataSensitivityDatabaseView); + }); + + it("scan-table advertises only paths the projection can resolve", () => { + projectsEveryExample(scanTable, TABLE, dataSensitivityTableView); + }); +}); diff --git a/packages/cli/src/commands/data-sensitivity/index.ts b/packages/cli/src/commands/data-sensitivity/index.ts new file mode 100644 index 00000000..18242d8d --- /dev/null +++ b/packages/cli/src/commands/data-sensitivity/index.ts @@ -0,0 +1,11 @@ +import { defineCommandGroup } from "../group"; + +export default defineCommandGroup({ + name: "data-sensitivity", + description: "Propose data sensitivity labels for fields with the LLM (dry run, nothing written)", + skills: [{ skill: "metadata", purpose: "what a data_sensitivity label is and how to apply one" }], + subCommands: { + "scan-db": () => import("./scan-db").then((mod) => mod.default), + "scan-table": () => import("./scan-table").then((mod) => mod.default), + }, +}); diff --git a/packages/cli/src/commands/data-sensitivity/scan-db.ts b/packages/cli/src/commands/data-sensitivity/scan-db.ts new file mode 100644 index 00000000..45ae7bfa --- /dev/null +++ b/packages/cli/src/commands/data-sensitivity/scan-db.ts @@ -0,0 +1,56 @@ +import { DataSensitivityDatabaseResult } from "@metabase/client/domain/data-sensitivity"; + +import { filterResult, formatDataSensitivityReport } from "../../output/data-sensitivity-report"; +import { renderSummary } from "../../output/render"; +import { dataSensitivityDatabaseView } from "../../output/views/data-sensitivity"; +import { connectionFlags, outputFlags, profileFlag } from "../flags"; +import { parseId } from "../parse-id"; +import { defineMetabaseCommand } from "../runtime"; + +import { parseScanFlags, scanFlags } from "./scan-flags"; + +const OVERSIZE_HINT = + "narrow the scan with `--schema `, keep fewer rows with `--status disagree,new`, raise " + + "`--max-bytes`, or scan one table at a time with `mb data-sensitivity scan-table `"; + +export default defineMetabaseCommand({ + meta: { + name: "scan-db", + description: + "Propose a data sensitivity label for every field of a database with the LLM (dry run)", + }, + details: + 'Dry run: nothing is written, the response is the proposal. For every active table (or only those in --schema) the server builds one packet of names, types, descriptions, fingerprints and a few sample values, asks the LLM for a label per field, and diffs each proposal against the field\'s current data_sensitivity label, which the model never sees. Statuses: agree, disagree, new (no current label yet), abstain (model unsure), dropped (no usable answer). The request is synchronous and runs as long as the scan, so raise --timeout for large databases; every request spends provider tokens. A table the server could not classify appears as an error entry and the run still exits 0. Apply a proposal with `mb field update --body \'{"data_sensitivity":"PII"}\'`.', + capabilities: { minVersion: 64, tokenFeature: "data_sensitivity" }, + args: { + ...outputFlags, + ...profileFlag, + ...connectionFlags, + ...scanFlags, + id: { type: "positional", description: "Database id", required: true }, + schema: { type: "string", description: "Scan only the tables in this schema" }, + }, + outputSchema: DataSensitivityDatabaseResult, + examples: [ + "mb data-sensitivity scan-db 1", + "mb data-sensitivity scan-db 1 --schema public", + "mb data-sensitivity scan-db 1 --status disagree,new --json", + "mb data-sensitivity scan-db 1 --json --fields counts,failed", + ], + async run({ args, ctx, getClient }) { + const id = parseId(args.id); + const scan = parseScanFlags(args); + const client = await getClient(); + const result = await client.dataSensitivity.classifyDatabase( + id, + { schema: args.schema }, + { timeoutMs: scan.timeoutMs }, + ); + renderSummary( + filterResult(result, scan.statuses), + dataSensitivityDatabaseView, + () => formatDataSensitivityReport(result, scan.statuses), + { ...ctx, oversizeHint: OVERSIZE_HINT }, + ); + }, +}); diff --git a/packages/cli/src/commands/data-sensitivity/scan-flags.ts b/packages/cli/src/commands/data-sensitivity/scan-flags.ts new file mode 100644 index 00000000..6fc87a51 --- /dev/null +++ b/packages/cli/src/commands/data-sensitivity/scan-flags.ts @@ -0,0 +1,35 @@ +import type { ParsedArgs } from "citty"; + +import { DataSensitivityStatus } from "@metabase/client/domain/data-sensitivity"; +import { DEFAULT_TIMEOUT_MS } from "@metabase/client/poll"; + +import { parseEnumCsv } from "../../runtime/csv"; +import { parseId } from "../parse-id"; + +export const scanFlags = { + status: { + type: "string", + description: `Keep only fields with these statuses, comma separated: ${DataSensitivityStatus.options.join(" | ")} (text default: every field with a changed label or semantic type; JSON default: all)`, + }, + timeout: { + type: "string", + description: "HTTP timeout in ms for the single synchronous scan request", + // One synchronous request covers the whole scan, so its budget is the one a polled wait gets, + // not the transport's per-request default. + default: String(DEFAULT_TIMEOUT_MS), + }, +} as const; + +type ScanArgs = ParsedArgs; + +interface ScanOptions { + statuses: DataSensitivityStatus[] | null; + timeoutMs: number; +} + +export function parseScanFlags(args: ScanArgs): ScanOptions { + return { + statuses: parseEnumCsv(args.status, DataSensitivityStatus, "--status") ?? null, + timeoutMs: parseId(args.timeout, "timeout"), + }; +} diff --git a/packages/cli/src/commands/data-sensitivity/scan-table.ts b/packages/cli/src/commands/data-sensitivity/scan-table.ts new file mode 100644 index 00000000..03735209 --- /dev/null +++ b/packages/cli/src/commands/data-sensitivity/scan-table.ts @@ -0,0 +1,46 @@ +import { DataSensitivityTableResult } from "@metabase/client/domain/data-sensitivity"; + +import { filterResult, formatDataSensitivityReport } from "../../output/data-sensitivity-report"; +import { renderSummary } from "../../output/render"; +import { dataSensitivityTableView } from "../../output/views/data-sensitivity"; +import { connectionFlags, outputFlags, profileFlag } from "../flags"; +import { parseId } from "../parse-id"; +import { defineMetabaseCommand } from "../runtime"; + +import { parseScanFlags, scanFlags } from "./scan-flags"; + +export default defineMetabaseCommand({ + meta: { + name: "scan-table", + description: + "Propose a data sensitivity label for every field of a table with the LLM (dry run)", + }, + details: + "Dry run: nothing is written, the response is the proposal. The server builds one packet of the table's names, types, descriptions, fingerprints and a few sample values, asks the LLM for a label per field, and diffs each proposal against the field's current data_sensitivity label, which the model never sees. Statuses: agree, disagree, new (no current label yet), abstain (model unsure), dropped (no usable answer). The request is synchronous and spends provider tokens. Apply a proposal with `mb field update --body '{\"data_sensitivity\":\"PII\"}'`.", + capabilities: { minVersion: 64, tokenFeature: "data_sensitivity" }, + args: { + ...outputFlags, + ...profileFlag, + ...connectionFlags, + ...scanFlags, + id: { type: "positional", description: "Table id", required: true }, + }, + outputSchema: DataSensitivityTableResult, + examples: [ + "mb data-sensitivity scan-table 3", + "mb data-sensitivity scan-table 3 --status disagree,new --json", + "mb data-sensitivity scan-table 3 --json --fields counts,usage", + ], + async run({ args, ctx, getClient }) { + const id = parseId(args.id); + const scan = parseScanFlags(args); + const client = await getClient(); + const result = await client.dataSensitivity.classifyTable(id, { timeoutMs: scan.timeoutMs }); + renderSummary( + filterResult(result, scan.statuses), + dataSensitivityTableView, + () => formatDataSensitivityReport(result, scan.statuses), + ctx, + ); + }, +}); diff --git a/packages/cli/src/commands/field/values.ts b/packages/cli/src/commands/field/values.ts index b3cbe50b..62ae228e 100644 --- a/packages/cli/src/commands/field/values.ts +++ b/packages/cli/src/commands/field/values.ts @@ -1,4 +1,5 @@ import { FieldValues } from "@metabase/client/domain/field"; +import { plural } from "../../output/format"; import { fieldValuesView } from "../../output/views/field"; import { formatScalar, renderSummary } from "../../output/render"; import { connectionFlags, outputFlags, profileFlag } from "../flags"; @@ -34,7 +35,7 @@ export default defineMetabaseCommand({ } const more = values.has_more_values === true ? " (more available; rescan for the full set)" : ""; - const header = `Field ${fieldId} has ${count} cached value${count === 1 ? "" : "s"}${more}:`; + const header = `Field ${fieldId} has ${plural(count, "cached value")}${more}:`; const lines = values.values.map((row) => ` ${formatScalar(row[0])}`); return [header, ...lines].join("\n"); }, diff --git a/packages/cli/src/main.ts b/packages/cli/src/main.ts index 420a0ace..d81080ff 100644 --- a/packages/cli/src/main.ts +++ b/packages/cli/src/main.ts @@ -19,6 +19,7 @@ const main: CommandDef = defineCommand({ upload: () => import("./commands/upload").then((mod) => mod.default), "content-translation": () => import("./commands/content-translation").then((mod) => mod.default), + "data-sensitivity": () => import("./commands/data-sensitivity").then((mod) => mod.default), card: () => import("./commands/card").then((mod) => mod.default), dashboard: () => import("./commands/dashboard").then((mod) => mod.default), subscription: () => import("./commands/subscription").then((mod) => mod.default), diff --git a/packages/cli/src/output/data-sensitivity-report.test.ts b/packages/cli/src/output/data-sensitivity-report.test.ts new file mode 100644 index 00000000..65ec7043 --- /dev/null +++ b/packages/cli/src/output/data-sensitivity-report.test.ts @@ -0,0 +1,245 @@ +import { describe, expect, it } from "vitest"; + +import type { + DataSensitivityDatabaseResult, + DataSensitivityFieldResult, + DataSensitivityTableResult, +} from "@metabase/client/domain/data-sensitivity"; + +import { filterResult, formatDataSensitivityReport } from "./data-sensitivity-report"; + +const USAGE = { + input_tokens: 1200, + output_tokens: 90, + cache_read_tokens: 0, + cache_creation_tokens: 0, + total_tokens: 1290, +}; + +interface FieldSpec { + name: string; + status: DataSensitivityFieldResult["status"]; + current: DataSensitivityFieldResult["current"]; + proposed: DataSensitivityFieldResult["proposed"]; + semantic_changed: boolean; +} + +function field(spec: FieldSpec): DataSensitivityFieldResult { + return { + field_id: spec.name.length, + name: spec.name, + display_name: null, + base_type: "type/Text", + current: spec.current, + proposed: spec.proposed, + status: spec.status, + semantic_changed: spec.semantic_changed, + }; +} + +const ID = field({ + name: "ID", + status: "agree", + current: { + data_sensitivity: "PUBLIC", + human_set: false, + state: "classifier", + semantic_type: "type/PK", + }, + proposed: { + data_sensitivity: "PUBLIC", + confidence: "high", + semantic_type: "type/PK", + reasoning: null, + }, + semantic_changed: false, +}); + +const CITY = field({ + name: "CITY", + status: "disagree", + current: { + data_sensitivity: "PUBLIC", + human_set: false, + state: "classifier", + semantic_type: null, + }, + proposed: { data_sensitivity: "PII", confidence: "high", semantic_type: null, reasoning: null }, + semantic_changed: false, +}); + +const EMAIL = field({ + name: "EMAIL", + status: "new", + current: { data_sensitivity: null, human_set: false, state: "unscanned", semantic_type: null }, + proposed: { + data_sensitivity: "PII", + confidence: "high", + semantic_type: "type/Email", + reasoning: null, + }, + semantic_changed: true, +}); + +const NOTES = field({ + name: "NOTES", + status: "abstain", + current: { + data_sensitivity: "PUBLIC", + human_set: false, + state: "classifier", + semantic_type: null, + }, + proposed: { data_sensitivity: null, confidence: "low", semantic_type: null, reasoning: null }, + semantic_changed: false, +}); + +const BLOB = field({ + name: "BLOB", + status: "dropped", + current: { data_sensitivity: null, human_set: false, state: "unscanned", semantic_type: null }, + proposed: { data_sensitivity: null, confidence: null, semantic_type: null, reasoning: null }, + semantic_changed: false, +}); + +const SSN = field({ + name: "SSN", + status: "agree", + current: { data_sensitivity: "PII", human_set: true, state: "human", semantic_type: "type/Name" }, + proposed: { + data_sensitivity: "PII", + confidence: "high", + semantic_type: "type/Category", + reasoning: null, + }, + semantic_changed: true, +}); + +const PEOPLE: DataSensitivityTableResult = { + table_id: 3, + table_name: "PEOPLE", + schema: "PUBLIC", + database_id: 1, + model: "anthropic/claude-haiku-4-5-20251001", + requests: 1, + usage: USAGE, + sample_error: null, + counts: { fields: 6, agree: 2, disagree: 1, new: 1, abstain: 1, dropped: 1, semantic_changed: 2 }, + fields: [ID, CITY, EMAIL, NOTES, BLOB, SSN], +}; + +const ORDERS_ERROR = { + table_id: 4, + table_name: "ORDERS", + schema: null, + error: "boom", + error_code: "skipped", +}; + +const DATABASE: DataSensitivityDatabaseResult = { + database_id: 1, + schema: null, + tables: [PEOPLE, ORDERS_ERROR], + counts: PEOPLE.counts, + usage: USAGE, + requests: 1, + failed: 1, +}; + +const TABLE_SUMMARY = + "Scanned table PUBLIC.PEOPLE, 6 fields: 2 agree, 1 disagree, 1 new, 1 unsure, 1 no answer, 2 semantic type changes. 1 request, 1200 in / 90 out tokens."; + +const DATABASE_SUMMARY = + "Scanned 2 tables (1 failed), 6 fields: 2 agree, 1 disagree, 1 new, 1 unsure, 1 no answer, 2 semantic type changes. 1 request, 1200 in / 90 out tokens."; + +describe("filterResult", () => { + it("hands the result back untouched when no filter was asked for", () => { + expect(filterResult(DATABASE, null)).toBe(DATABASE); + }); + + it("keeps only the fields whose status was asked for and leaves the counts as the server sent them", () => { + expect(filterResult(PEOPLE, ["disagree", "new"])).toEqual({ + ...PEOPLE, + fields: [CITY, EMAIL], + }); + }); + + it("filters every table's fields and keeps the table errors", () => { + expect(filterResult(DATABASE, ["abstain"])).toEqual({ + ...DATABASE, + tables: [{ ...PEOPLE, fields: [NOTES] }, ORDERS_ERROR], + }); + }); +}); + +describe("formatDataSensitivityReport", () => { + it("breaks the input tokens down by cache bucket when the provider cached part of the prompt", () => { + const cached = { ...PEOPLE, usage: { ...USAGE, cache_read_tokens: 1100 } }; + expect(formatDataSensitivityReport(cached, [])).toBe( + TABLE_SUMMARY.replace("1200 in", "1200 in (0 cache write, 1100 cache read)"), + ); + }); + + it("shows every field with a changed label or semantic type when no filter was asked for", () => { + expect(formatDataSensitivityReport(PEOPLE, null)).toBe( + [ + TABLE_SUMMARY, + "┌───────┬───────────┬────────────────┬──────────────────┐", + "│ Field │ Base type │ Sensitivity │ Semantic type │", + "├───────┼───────────┼────────────────┼──────────────────┤", + "│ CITY │ Text │ PUBLIC -> PII │ │", + "├───────┼───────────┼────────────────┼──────────────────┤", + "│ EMAIL │ Text │ -> PII │ -> Email │", + "├───────┼───────────┼────────────────┼──────────────────┤", + "│ NOTES │ Text │ PUBLIC -> ? │ │", + "├───────┼───────────┼────────────────┼──────────────────┤", + "│ BLOB │ Text │ -> (no answer) │ │", + "├───────┼───────────┼────────────────┼──────────────────┤", + "│ SSN │ Text │ PII* │ Name -> Category │", + "└───────┴───────────┴────────────────┴──────────────────┘", + "* set by a person", + ].join("\n"), + ); + }); + + it("adds the table column in database scope and renders a failed table as one row", () => { + expect(formatDataSensitivityReport(DATABASE, ["agree"])).toBe( + [ + DATABASE_SUMMARY, + "┌───────────────┬───────┬───────────┬─────────────┬──────────────────┐", + "│ Table │ Field │ Base type │ Sensitivity │ Semantic type │", + "├───────────────┼───────┼───────────┼─────────────┼──────────────────┤", + "│ PUBLIC.PEOPLE │ ID │ Text │ PUBLIC │ PK │", + "├───────────────┼───────┼───────────┼─────────────┼──────────────────┤", + "│ PUBLIC.PEOPLE │ SSN │ Text │ PII* │ Name -> Category │", + "├───────────────┼───────┼───────────┼─────────────┼──────────────────┤", + "│ ORDERS │ │ │ boom │ │", + "└───────────────┴───────┴───────────┴─────────────┴──────────────────┘", + "* set by a person", + ].join("\n"), + ); + }); + + it("omits the footnote when no shown label was set by a person", () => { + expect(formatDataSensitivityReport(PEOPLE, ["new"])).toBe( + [ + TABLE_SUMMARY, + "┌───────┬───────────┬─────────────┬───────────────┐", + "│ Field │ Base type │ Sensitivity │ Semantic type │", + "├───────┼───────────┼─────────────┼───────────────┤", + "│ EMAIL │ Text │ -> PII │ -> Email │", + "└───────┴───────────┴─────────────┴───────────────┘", + ].join("\n"), + ); + }); + + it("prints only the summary when no row survives the filter", () => { + expect(formatDataSensitivityReport({ ...PEOPLE, fields: [ID] }, null)).toBe(TABLE_SUMMARY); + }); + + it("reports a failed sample on a table scan", () => { + expect( + formatDataSensitivityReport({ ...PEOPLE, fields: [ID], sample_error: "timeout" }, null), + ).toBe(`${TABLE_SUMMARY} Sample values unavailable: timeout`); + }); +}); diff --git a/packages/cli/src/output/data-sensitivity-report.ts b/packages/cli/src/output/data-sensitivity-report.ts new file mode 100644 index 00000000..b8fc8106 --- /dev/null +++ b/packages/cli/src/output/data-sensitivity-report.ts @@ -0,0 +1,221 @@ +import { + type DataSensitivityCounts, + type DataSensitivityDatabaseResult, + type DataSensitivityFieldResult, + type DataSensitivityStatus, + type DataSensitivityTableEntry, + type DataSensitivityTableResult, + isDataSensitivityTableError, +} from "@metabase/client/domain/data-sensitivity"; + +import { plural, qualifiedName } from "./format"; +import { renderTable } from "./table"; +import type { ColumnDef } from "./view"; + +type DataSensitivityResult = DataSensitivityTableResult | DataSensitivityDatabaseResult; + +type StatusFilter = ReadonlyArray | null; + +const ARROW = "->"; +const UNSURE_CELL = "?"; +const NO_ANSWER_CELL = "(no answer)"; +const HUMAN_SET_MARK = "*"; +const HUMAN_SET_FOOTNOTE = `${HUMAN_SET_MARK} set by a person`; +const TYPE_PREFIX = "type/"; + +// Each LLM output gets one cell: the value alone when nothing changes, `current -> proposed` when +// it does. An error row has no field, so its field-level cells are null and the sensitivity cell +// carries the message. +interface DataSensitivityRow { + table: string; + field: string | null; + base_type: string | null; + sensitivity: string | null; + semantic_type: string | null; + human_set: boolean; +} + +const fieldColumns: ColumnDef[] = [ + { key: "field", label: "Field" }, + { key: "base_type", label: "Base type" }, + { key: "sensitivity", label: "Sensitivity" }, + { key: "semantic_type", label: "Semantic type" }, +]; + +const tableColumn: ColumnDef = { key: "table", label: "Table" }; + +function isDatabaseResult(result: DataSensitivityResult): result is DataSensitivityDatabaseResult { + return "tables" in result; +} + +function stripTypePrefix(value: string | null): string | null { + if (value === null) { + return null; + } + return value.startsWith(TYPE_PREFIX) ? value.slice(TYPE_PREFIX.length) : value; +} + +function diffCell( + current: string | null, + proposed: string | null, + humanSet: boolean, +): string | null { + const shown = current === null ? null : `${current}${humanSet ? HUMAN_SET_MARK : ""}`; + if (proposed === null || proposed === current) { + return shown; + } + return shown === null ? `${ARROW} ${proposed}` : `${shown} ${ARROW} ${proposed}`; +} + +function sensitivityCell(field: DataSensitivityFieldResult): string | null { + const current = field.current.data_sensitivity; + const humanSet = field.current.human_set; + switch (field.status) { + case "abstain": { + return diffCell(current, UNSURE_CELL, humanSet); + } + case "dropped": { + return diffCell(current, NO_ANSWER_CELL, humanSet); + } + default: { + return diffCell(current, field.proposed.data_sensitivity, humanSet); + } + } +} + +// The server reports the effective semantic type as the proposal, so a null proposal only happens +// when the field has none and none was suggested. +function semanticTypeCell(field: DataSensitivityFieldResult): string | null { + const current = stripTypePrefix(field.current.semantic_type); + const proposed = stripTypePrefix(field.proposed.semantic_type); + return diffCell(current, proposed ?? current, false); +} + +function entryRows(entry: DataSensitivityTableEntry): DataSensitivityRow[] { + const table = qualifiedName(entry.schema, entry.table_name); + if (isDataSensitivityTableError(entry)) { + return [ + { + table, + field: null, + base_type: null, + sensitivity: entry.error, + semantic_type: null, + human_set: false, + }, + ]; + } + return entry.fields.map((field) => ({ + table, + field: field.name, + base_type: stripTypePrefix(field.base_type), + sensitivity: sensitivityCell(field), + semantic_type: semanticTypeCell(field), + human_set: field.current.human_set, + })); +} + +function flattenRows(result: DataSensitivityResult): DataSensitivityRow[] { + const entries: DataSensitivityTableEntry[] = isDatabaseResult(result) ? result.tables : [result]; + return entries.flatMap(entryRows); +} + +type FieldPredicate = (field: DataSensitivityFieldResult) => boolean; + +// Table errors are not field rows, so no filter drops them. +function filterFields(result: DataSensitivityResult, keep: FieldPredicate): DataSensitivityResult { + if (isDatabaseResult(result)) { + return { + ...result, + tables: result.tables.map((entry) => + isDataSensitivityTableError(entry) + ? entry + : { ...entry, fields: entry.fields.filter(keep) }, + ), + }; + } + return { ...result, fields: result.fields.filter(keep) }; +} + +// A null filter keeps every field and hands the result back untouched. +export function filterResult( + result: DataSensitivityTableResult, + statuses: StatusFilter, +): DataSensitivityTableResult; +export function filterResult( + result: DataSensitivityDatabaseResult, + statuses: StatusFilter, +): DataSensitivityDatabaseResult; +export function filterResult( + result: DataSensitivityResult, + statuses: StatusFilter, +): DataSensitivityResult; +export function filterResult( + result: DataSensitivityResult, + statuses: StatusFilter, +): DataSensitivityResult { + if (statuses === null) { + return result; + } + const keep = new Set(statuses); + return filterFields(result, (field) => keep.has(field.status)); +} + +// Text output is for a person deciding what to change, so without an explicit filter it shows +// every field where either LLM output differs from what is stored. +function hasProposedChange(field: DataSensitivityFieldResult): boolean { + return field.status !== "agree" || field.semantic_changed; +} + +function countsSentence(counts: DataSensitivityCounts): string { + return ( + `${plural(counts.fields, "field")}: ${counts.agree} agree, ${counts.disagree} disagree, ` + + `${counts.new} new, ${counts.abstain} unsure, ${counts.dropped} no answer, ` + + `${plural(counts.semantic_changed, "semantic type change")}.` + ); +} + +// `input_tokens` is the total prompt, cache buckets included, so the breakdown only appears when +// the provider cached any of it. +function usageSentence(result: DataSensitivityResult): string { + const { input_tokens, output_tokens, cache_creation_tokens, cache_read_tokens } = result.usage; + const cached = + cache_creation_tokens === 0 && cache_read_tokens === 0 + ? "" + : ` (${cache_creation_tokens} cache write, ${cache_read_tokens} cache read)`; + return `${plural(result.requests, "request")}, ${input_tokens} in${cached} / ${output_tokens} out tokens.`; +} + +// Counts and usage are the server's totals for the whole scan, so the summary is the same +// whatever the filter kept. +function summaryLine(result: DataSensitivityResult): string { + if (isDatabaseResult(result)) { + const failed = result.failed === 0 ? "" : ` (${result.failed} failed)`; + return ( + `Scanned ${plural(result.tables.length, "table")}${failed}, ` + + `${countsSentence(result.counts)} ${usageSentence(result)}` + ); + } + const sample = + result.sample_error === null ? "" : ` Sample values unavailable: ${result.sample_error}`; + return ( + `Scanned table ${qualifiedName(result.schema, result.table_name)}, ` + + `${countsSentence(result.counts)} ${usageSentence(result)}${sample}` + ); +} + +export function formatDataSensitivityReport( + result: DataSensitivityResult, + statuses: StatusFilter, +): string { + const shown = + statuses === null ? filterFields(result, hasProposedChange) : filterResult(result, statuses); + const rows = flattenRows(shown); + const summary = summaryLine(result); + if (rows.length === 0) { + return summary; + } + const columns = isDatabaseResult(result) ? [tableColumn, ...fieldColumns] : fieldColumns; + const footnote = rows.some((row) => row.human_set) ? `\n${HUMAN_SET_FOOTNOTE}` : ""; + return `${summary}\n${renderTable(rows, columns)}${footnote}`; +} diff --git a/packages/cli/src/output/format.test.ts b/packages/cli/src/output/format.test.ts index ef26ab70..e1fd06cd 100644 --- a/packages/cli/src/output/format.test.ts +++ b/packages/cli/src/output/format.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { ConfigError } from "@metabase/client/errors"; -import { resolveFormat } from "./format"; +import { plural, qualifiedName, resolveFormat } from "./format"; describe("resolveFormat", () => { it("forces json when --json is set", () => { @@ -34,3 +34,23 @@ describe("resolveFormat", () => { expect(resolveFormat({ json: undefined, format: undefined, isTty: false })).toBe("json"); }); }); + +describe("plural", () => { + it("leaves the noun singular for exactly one and pluralizes every other count", () => { + expect([plural(0, "table"), plural(1, "table"), plural(2, "table")]).toEqual([ + "0 tables", + "1 table", + "2 tables", + ]); + }); +}); + +describe("qualifiedName", () => { + it("prefixes the schema when there is one and leaves a bare name otherwise", () => { + expect([ + qualifiedName("public", "orders"), + qualifiedName(null, "orders"), + qualifiedName(undefined, "orders"), + ]).toEqual(["public.orders", "orders", "orders"]); + }); +}); diff --git a/packages/cli/src/output/format.ts b/packages/cli/src/output/format.ts index 2d259bb1..f28eba33 100644 --- a/packages/cli/src/output/format.ts +++ b/packages/cli/src/output/format.ts @@ -23,3 +23,11 @@ export function resolveFormat({ json, format, isTty }: FormatInputs): Format { } return isTty ? "text" : "json"; } + +export function plural(count: number, noun: string): string { + return `${count} ${noun}${count === 1 ? "" : "s"}`; +} + +export function qualifiedName(schema: string | null | undefined, name: string): string { + return schema ? `${schema}.${name}` : name; +} diff --git a/packages/cli/src/output/query-result.ts b/packages/cli/src/output/query-result.ts index 2af08aef..113ae670 100644 --- a/packages/cli/src/output/query-result.ts +++ b/packages/cli/src/output/query-result.ts @@ -1,5 +1,6 @@ import type { CardQueryResult } from "@metabase/client/domain/card"; +import { plural } from "./format"; import { formatScalar, renderRows } from "./table"; export function formatQueryResult(result: CardQueryResult): string { @@ -10,7 +11,7 @@ export function formatQueryResult(result: CardQueryResult): string { } const head = result.data.cols.map((col) => col.display_name ?? col.name); const rowCount = result.row_count ?? result.data.rows.length; - const summary = `${rowCount} row${rowCount === 1 ? "" : "s"}.`; + const summary = `${plural(rowCount, "row")}.`; if (result.data.rows.length === 0) { return summary; } diff --git a/packages/cli/src/output/views/data-sensitivity.ts b/packages/cli/src/output/views/data-sensitivity.ts new file mode 100644 index 00000000..b047e42f --- /dev/null +++ b/packages/cli/src/output/views/data-sensitivity.ts @@ -0,0 +1,24 @@ +import { + DataSensitivityDatabaseResult, + DataSensitivityTableResult, +} from "@metabase/client/domain/data-sensitivity"; + +import type { ResourceView } from "../view"; + +export const dataSensitivityTableView: ResourceView = { + compactPick: DataSensitivityTableResult, + tableColumns: [ + { key: "table_id", label: "Table" }, + { key: "table_name", label: "Name" }, + { key: "requests", label: "Requests" }, + ], +}; + +export const dataSensitivityDatabaseView: ResourceView = { + compactPick: DataSensitivityDatabaseResult, + tableColumns: [ + { key: "database_id", label: "Database" }, + { key: "requests", label: "Requests" }, + { key: "failed", label: "Failed" }, + ], +}; diff --git a/packages/cli/src/output/views/transform.ts b/packages/cli/src/output/views/transform.ts index 8cadb28c..2b34956a 100644 --- a/packages/cli/src/output/views/transform.ts +++ b/packages/cli/src/output/views/transform.ts @@ -6,6 +6,7 @@ import { TransformTarget, } from "@metabase/client/domain/transform"; +import { qualifiedName } from "../format"; import { MALFORMED_CELL } from "../table"; import type { ResourceView } from "../view"; @@ -14,8 +15,7 @@ function formatTarget(value: unknown): string { if (!parsed.success) { return MALFORMED_CELL; } - const { schema, name } = parsed.data; - return schema ? `${schema}.${name}` : name; + return qualifiedName(parsed.data.schema, parsed.data.name); } export const transformRunView: ResourceView = { diff --git a/packages/cli/src/runtime/command-help.test.ts b/packages/cli/src/runtime/command-help.test.ts index aac33b4f..ea15afbc 100644 --- a/packages/cli/src/runtime/command-help.test.ts +++ b/packages/cli/src/runtime/command-help.test.ts @@ -325,6 +325,8 @@ const ALL_COMMANDS = [ "upload replace", "content-translation download", "content-translation upload", + "data-sensitivity scan-db", + "data-sensitivity scan-table", "card list", "card get", "card query", @@ -450,6 +452,7 @@ const CONTENT_TRANSLATION_CAPABILITIES = { minVersion: 58, tokenFeature: "content_translation", } as const; +const DATA_SENSITIVITY_CAPABILITIES = { minVersion: 64, tokenFeature: "data_sensitivity" } as const; const TRANSFORM_CAPABILITIES = { minVersion: 59 } as const; const TRANSFORM_JOB_SET_ACTIVE_CAPABILITIES = { minVersion: 61 } as const; @@ -525,6 +528,19 @@ describe("command tree contract", () => { }); }); + it("gates every data sensitivity command on v64 and its premium feature", async () => { + const entries = await allEntries(); + const capabilities = Object.fromEntries( + entries + .filter((entry) => entry.command.startsWith("data-sensitivity ")) + .map((entry) => [entry.command, entry.capabilities]), + ); + expect(capabilities).toEqual({ + "data-sensitivity scan-db": DATA_SENSITIVITY_CAPABILITIES, + "data-sensitivity scan-table": DATA_SENSITIVITY_CAPABILITIES, + }); + }); + it("carries the transform version gates through to every transform command", async () => { const entries = await allEntries(); const transformCapabilities = Object.fromEntries( diff --git a/packages/client/README.md b/packages/client/README.md index a450b28f..0c76dee8 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -407,10 +407,13 @@ deliberately absent. A second class of schema describes a single response shape that has no compact pair: `DashboardDetail`, `DatabaseSyncResult`, `CollectionTreeNode`, `FieldSummary`, `SettingValue`, -`TableQueryMetadata`, `SessionProperties`, and `TokenFeatures`. +`TableQueryMetadata`, `SessionProperties`, `TokenFeatures`, `DataSensitivityTableResult`, +`DataSensitivityDatabaseResult`, and the `DataSensitivityFieldResult`, `DataSensitivityTableError`, +and `DataSensitivityCounts` shapes they compose. Request bodies (`CreateInput`, `UpdateInput`) and the domain vocabulary enums -(`FieldBaseType`, `SearchModel`, `CollectionItemModel`, …) live in the same modules and are reached +(`FieldBaseType`, `DataSensitivityLabel`, `DataSensitivityStatus`, `SearchModel`, +`CollectionItemModel`, …) live in the same modules and are reached at `@metabase/client/domain/`. ### Utilities diff --git a/packages/client/src/client.ts b/packages/client/src/client.ts index 4ad45d80..ea5b0744 100644 --- a/packages/client/src/client.ts +++ b/packages/client/src/client.ts @@ -3,6 +3,7 @@ import { cardResource } from "./resources/card"; import { collectionResource } from "./resources/collection"; import { contentTranslationResource } from "./resources/content-translation"; import { dashboardResource } from "./resources/dashboard"; +import { dataSensitivityResource } from "./resources/data-sensitivity"; import { databaseResource } from "./resources/database"; import { datasetResource } from "./resources/dataset"; import { documentResource } from "./resources/document"; @@ -37,6 +38,7 @@ export function createClient(config: ClientCredentials, options: ClientOptions) collection: collectionResource(transport), contentTranslation: contentTranslationResource(transport), dashboard: dashboardResource(transport), + dataSensitivity: dataSensitivityResource(transport), database: databaseResource(transport), dataset: datasetResource(transport), document: documentResource(transport), diff --git a/packages/client/src/domain/data-sensitivity.ts b/packages/client/src/domain/data-sensitivity.ts new file mode 100644 index 00000000..a602b548 --- /dev/null +++ b/packages/client/src/domain/data-sensitivity.ts @@ -0,0 +1,118 @@ +import { z } from "zod"; + +import { FieldBaseType, FieldSemanticType } from "./field"; + +// Most severe first: the order is the precedence an automated classifier applies when several +// categories match one column. `null` is unscanned; `PUBLIC` is scanned and found clean. +export const DataSensitivityLabel = z.enum([ + "SEC_KEY", + "SYS_TELEMETRY", + "PHI", + "BIO_GEN", + "PCI_FIN", + "SENS_PERS", + "PII", + "CORP_IP", + "BIZ_CONF", + "PUBLIC", +]); +export type DataSensitivityLabel = z.infer; + +export const DataSensitivityStatus = z.enum(["agree", "disagree", "new", "abstain", "dropped"]); +export type DataSensitivityStatus = z.infer; + +export const DataSensitivityConfidence = z.enum(["high", "medium", "low"]); +export type DataSensitivityConfidence = z.infer; + +const DataSensitivityLabelState = z.enum(["human", "classifier", "unscanned"]); + +const DataSensitivityUsage = z.object({ + input_tokens: z.number().int(), + output_tokens: z.number().int(), + cache_read_tokens: z.number().int(), + cache_creation_tokens: z.number().int(), + total_tokens: z.number().int(), +}); + +export const DataSensitivityCounts = z.object({ + fields: z.number().int(), + agree: z.number().int(), + disagree: z.number().int(), + new: z.number().int(), + abstain: z.number().int(), + dropped: z.number().int(), + semantic_changed: z.number().int(), +}); +export type DataSensitivityCounts = z.infer; + +export const DataSensitivityFieldResult = z + .object({ + field_id: z.number().int().positive(), + name: z.string(), + display_name: z.string().nullable(), + base_type: FieldBaseType, + current: z.object({ + data_sensitivity: DataSensitivityLabel.nullable(), + human_set: z.boolean(), + state: DataSensitivityLabelState, + semantic_type: FieldSemanticType.nullable(), + }), + proposed: z.object({ + data_sensitivity: DataSensitivityLabel.nullable(), + confidence: DataSensitivityConfidence.nullable(), + semantic_type: FieldSemanticType.nullable(), + reasoning: z.string().nullable(), + }), + status: DataSensitivityStatus, + semantic_changed: z.boolean(), + }) + .loose(); +export type DataSensitivityFieldResult = z.infer; + +export const DataSensitivityTableResult = z + .object({ + table_id: z.number().int().positive(), + table_name: z.string(), + schema: z.string().nullable(), + database_id: z.number().int().positive(), + model: z.string(), + requests: z.number().int(), + usage: DataSensitivityUsage, + sample_error: z.string().nullable(), + counts: DataSensitivityCounts, + fields: z.array(DataSensitivityFieldResult), + }) + .loose(); +export type DataSensitivityTableResult = z.infer; + +export const DataSensitivityTableError = z + .object({ + table_id: z.number().int().positive(), + table_name: z.string(), + schema: z.string().nullable(), + error: z.string(), + error_code: z.string().nullable(), + }) + .loose(); +export type DataSensitivityTableError = z.infer; + +export type DataSensitivityTableEntry = DataSensitivityTableResult | DataSensitivityTableError; + +export function isDataSensitivityTableError( + entry: DataSensitivityTableEntry, +): entry is DataSensitivityTableError { + return "error" in entry; +} + +export const DataSensitivityDatabaseResult = z + .object({ + database_id: z.number().int().positive(), + schema: z.string().nullable(), + tables: z.array(z.union([DataSensitivityTableResult, DataSensitivityTableError])), + counts: DataSensitivityCounts, + usage: DataSensitivityUsage, + requests: z.number().int(), + failed: z.number().int(), + }) + .loose(); +export type DataSensitivityDatabaseResult = z.infer; diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index 356dcb0a..fcfc8cbd 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -104,6 +104,17 @@ export { } from "./domain/collection"; export { ContentTranslationUploadResult } from "./domain/content-translation"; export { CronUiDisplayType } from "./domain/cron"; +export { + DataSensitivityConfidence, + DataSensitivityCounts, + DataSensitivityDatabaseResult, + DataSensitivityFieldResult, + DataSensitivityLabel, + DataSensitivityStatus, + DataSensitivityTableError, + DataSensitivityTableResult, + isDataSensitivityTableError, +} from "./domain/data-sensitivity"; export { Dashboard, DashboardCompact, diff --git a/packages/client/src/resources/data-sensitivity.test.ts b/packages/client/src/resources/data-sensitivity.test.ts new file mode 100644 index 00000000..81f33607 --- /dev/null +++ b/packages/client/src/resources/data-sensitivity.test.ts @@ -0,0 +1,141 @@ +import { describe, expect, it } from "vitest"; + +import { createClient } from "../client"; +import type { ClientCredentials } from "../http/transport"; +import { captureFetch, jsonResponse, TEST_USER_AGENT } from "../testing/fetch-capture"; + +const CREDENTIALS: ClientCredentials = { + url: "https://mb.example.com/metabase", + credential: { kind: "apiKey", apiKey: "mb_wire_test_key" }, +}; + +const USAGE = { + input_tokens: 1200, + output_tokens: 90, + cache_read_tokens: 0, + cache_creation_tokens: 0, + total_tokens: 1290, +}; + +const COUNTS = { + fields: 1, + agree: 0, + disagree: 0, + new: 1, + abstain: 0, + dropped: 0, + semantic_changed: 0, +}; + +const TABLE_RESULT = { + table_id: 3, + table_name: "PEOPLE", + schema: "PUBLIC", + database_id: 1, + model: "anthropic/claude-haiku-4-5-20251001", + requests: 1, + usage: USAGE, + sample_error: null, + counts: COUNTS, + fields: [ + { + field_id: 12, + name: "EMAIL", + display_name: "Email", + base_type: "type/Text", + current: { + data_sensitivity: null, + human_set: false, + state: "unscanned", + semantic_type: "type/Email", + }, + proposed: { + data_sensitivity: "PII", + confidence: "high", + semantic_type: null, + reasoning: "Personal email addresses.", + }, + status: "new", + semantic_changed: false, + }, + ], +}; + +const DATABASE_RESULT = { + database_id: 1, + schema: null, + tables: [TABLE_RESULT], + counts: COUNTS, + usage: USAGE, + requests: 1, + failed: 0, +}; + +const JSON_REQUEST_HEADERS = { + accept: "application/json", + "content-type": "application/json", + "user-agent": TEST_USER_AGENT, + "x-api-key": "mb_wire_test_key", +}; + +const JSON_READ_HEADERS = { + accept: "application/json", + "user-agent": TEST_USER_AGENT, + "x-api-key": "mb_wire_test_key", +}; + +function clientOver(responses: Array) { + const capture = captureFetch(responses); + const mb = createClient(CREDENTIALS, { + userAgent: TEST_USER_AGENT, + fetchImpl: capture.fetch, + }); + return { mb, capture }; +} + +describe("data-sensitivity resource wire requests", () => { + it("posts the table classification without a body", async () => { + const { mb, capture } = clientOver([jsonResponse(TABLE_RESULT)]); + + await mb.dataSensitivity.classifyTable(3); + + expect(capture.calls).toEqual([ + { + url: "https://mb.example.com/metabase/api/ee/data-sensitivity/table/3", + method: "POST", + headers: JSON_READ_HEADERS, + body: null, + }, + ]); + }); + + it("posts the database classification with an empty body when no schema is given", async () => { + const { mb, capture } = clientOver([jsonResponse(DATABASE_RESULT)]); + + await mb.dataSensitivity.classifyDatabase(1); + + expect(capture.calls).toEqual([ + { + url: "https://mb.example.com/metabase/api/ee/data-sensitivity/database/1", + method: "POST", + headers: JSON_REQUEST_HEADERS, + body: "{}", + }, + ]); + }); + + it("posts the database classification with the schema in the body", async () => { + const { mb, capture } = clientOver([jsonResponse(DATABASE_RESULT)]); + + await mb.dataSensitivity.classifyDatabase(1, { schema: "public" }); + + expect(capture.calls).toEqual([ + { + url: "https://mb.example.com/metabase/api/ee/data-sensitivity/database/1", + method: "POST", + headers: JSON_REQUEST_HEADERS, + body: '{"schema":"public"}', + }, + ]); + }); +}); diff --git a/packages/client/src/resources/data-sensitivity.ts b/packages/client/src/resources/data-sensitivity.ts new file mode 100644 index 00000000..979593b4 --- /dev/null +++ b/packages/client/src/resources/data-sensitivity.ts @@ -0,0 +1,42 @@ +import { + DataSensitivityDatabaseResult, + DataSensitivityTableResult, +} from "../domain/data-sensitivity"; +import type { RequestOptions, Transport } from "../http/transport"; + +const DATA_SENSITIVITY_PATH = "/api/ee/data-sensitivity"; + +export interface DataSensitivityDatabaseParams { + schema?: string | undefined; +} + +// Every path parameter is a numeric id, so none needs encoding. Neither method opts into retries: a +// resend spends provider tokens on a scan the server may already be running. +export function dataSensitivityResource(transport: Transport) { + /** Classify every active field of a table with the LLM and diff the proposals against the current `data_sensitivity` labels. Synchronous; writes nothing. */ + async function classifyTable( + id: number, + options: RequestOptions = {}, + ): Promise { + return transport.requestParsed( + DataSensitivityTableResult, + `${DATA_SENSITIVITY_PATH}/table/${id}`, + { ...options, method: "POST" }, + ); + } + + /** Classify every active table of a database, or only those in `schema`, with the LLM and diff the proposals against the current `data_sensitivity` labels. Synchronous; writes nothing. */ + async function classifyDatabase( + id: number, + params: DataSensitivityDatabaseParams = {}, + options: RequestOptions = {}, + ): Promise { + return transport.requestParsed( + DataSensitivityDatabaseResult, + `${DATA_SENSITIVITY_PATH}/database/${id}`, + { ...options, method: "POST", body: { schema: params.schema } }, + ); + } + + return { classifyTable, classifyDatabase }; +} diff --git a/tests/e2e/data-sensitivity.e2e.test.ts b/tests/e2e/data-sensitivity.e2e.test.ts new file mode 100644 index 00000000..c535f29b --- /dev/null +++ b/tests/e2e/data-sensitivity.e2e.test.ts @@ -0,0 +1,118 @@ +import { afterEach, beforeAll, describe, expect, it } from "vitest"; + +import { readBootstrap, type E2EBootstrap } from "./bootstrap-data"; +import { cliErrorCategory, cliErrorMessage } from "./cli-error"; +import { cleanupConfigHome, mkTempConfigHome, runCli } from "./run-cli"; +import { SEEDED } from "./seed/seeded"; +import { requireServer } from "./server-gate"; + +const EE_UNAVAILABLE = requireServer("data-sensitivity › against EE endpoints", { + minVersion: 64, + tokenFeature: "data_sensitivity", +}); + +// The compose stack configures no AI provider, so on a server that grants the feature the first +// gate the scan reaches after the write check is the provider pre-flight. +const NO_PROVIDER_MESSAGE = "No AI provider is configured for Metabot."; + +describe("data-sensitivity arg validation e2e (no Metabase contact required)", () => { + const tempDirs: string[] = []; + + afterEach(async () => { + await Promise.all(tempDirs.splice(0).map(cleanupConfigHome)); + }); + + async function makeIsolatedConfigHome(): Promise { + const dir = await mkTempConfigHome(); + tempDirs.push(dir); + return dir; + } + + it("scan-db with a non-integer id fails fast with ConfigError", async () => { + const result = await runCli({ + args: ["data-sensitivity", "scan-db", "abc", "--json"], + configHome: await makeIsolatedConfigHome(), + }); + + expect(result.exitCode).toBe(2); + expect(cliErrorMessage(result.stderr)).toContain('invalid id: "abc" (expected integer)'); + expect(result.stdout).toBe(""); + }); + + it("scan-table with an unknown --status fails fast with ConfigError", async () => { + const result = await runCli({ + args: ["data-sensitivity", "scan-table", "1", "--status", "disagree,bogus", "--json"], + configHome: await makeIsolatedConfigHome(), + }); + + expect(result.exitCode).toBe(2); + expect(cliErrorMessage(result.stderr)).toBe( + "invalid --status value: bogus (expected one of: agree, disagree, new, abstain, dropped)", + ); + expect(result.stdout).toBe(""); + }); + + it("scan-db with a zero --timeout fails fast with ConfigError", async () => { + const result = await runCli({ + args: ["data-sensitivity", "scan-db", "1", "--timeout", "0", "--json"], + configHome: await makeIsolatedConfigHome(), + }); + + expect(result.exitCode).toBe(2); + expect(cliErrorMessage(result.stderr)).toBe("invalid timeout: 0 (must be ≥ 1)"); + expect(result.stdout).toBe(""); + }); +}); + +describe.skipIf(EE_UNAVAILABLE !== null)("data-sensitivity e2e against EE endpoints", () => { + let bootstrap: E2EBootstrap; + const tempDirs: string[] = []; + + beforeAll(async () => { + bootstrap = await readBootstrap(); + }); + + afterEach(async () => { + await Promise.all(tempDirs.splice(0).map(cleanupConfigHome)); + }); + + async function makeIsolatedConfigHome(): Promise { + const dir = await mkTempConfigHome(); + tempDirs.push(dir); + return dir; + } + + function authEnv(): Record { + return { + MB_URL: bootstrap.baseUrl, + MB_API_KEY: bootstrap.adminApiKey, + }; + } + + it("scan-db stops at the provider pre-flight when no AI provider is configured", async () => { + const result = await runCli({ + args: ["data-sensitivity", "scan-db", String(SEEDED.warehouseDbId), "--json"], + configHome: await makeIsolatedConfigHome(), + env: authEnv(), + }); + + expect(result.exitCode).toBe(1); + expect(cliErrorCategory(result.stderr)).toBe("http"); + expect(cliErrorMessage(result.stderr)).toBe(NO_PROVIDER_MESSAGE); + expect(result.stdout).toBe(""); + }); + + it("scan-table against a missing table id surfaces a 404 HttpError", async () => { + const result = await runCli({ + args: ["data-sensitivity", "scan-table", "9999999", "--json"], + configHome: await makeIsolatedConfigHome(), + env: authEnv(), + }); + + expect(result.exitCode).toBe(1); + expect(cliErrorMessage(result.stderr)).toBe( + "Not found: POST /api/ee/data-sensitivity/table/9999999.", + ); + expect(result.stdout).toBe(""); + }); +});