From 74d0a06398eaa155c0499a3ec18c6d4915182291 Mon Sep 17 00:00:00 2001 From: Mofei Zhu Date: Tue, 29 Sep 2026 15:42:28 +0300 Subject: [PATCH 01/20] Add mapbox workflow: install and run multi-step workflows (beta) A workflow is a workflow.yaml of command and script steps, installed from a local directory or a GitHub repository's workflow/// into ~/.mapbox/workflows/, then run by name. Command steps run this binary again with --output json, so they resolve credentials, timeouts and path encoding exactly as a typed command does. The first published workflow is workflow/beta/copy-style, which copies a style between accounts using two credential profiles. --- CHANGELOG.md | 10 + docs/commands.md | 208 +++++++ src/main.rs | 15 + src/schema.rs | 8 + src/workflow/definition.rs | 598 ++++++++++++++++++ src/workflow/mod.rs | 494 +++++++++++++++ src/workflow/runner.rs | 601 ++++++++++++++++++ src/workflow/store.rs | 644 ++++++++++++++++++++ src/workflow/template.rs | 351 +++++++++++ tests/source_guards.rs | 10 +- tests/workflow.rs | 295 +++++++++ workflow/README.md | 94 +++ workflow/beta/copy-style/README.md | 21 + workflow/beta/copy-style/scripts/prepare.py | 67 ++ workflow/beta/copy-style/workflow.yaml | 65 ++ 15 files changed, 3480 insertions(+), 1 deletion(-) create mode 100644 src/workflow/definition.rs create mode 100644 src/workflow/mod.rs create mode 100644 src/workflow/runner.rs create mode 100644 src/workflow/store.rs create mode 100644 src/workflow/template.rs create mode 100644 tests/workflow.rs create mode 100644 workflow/README.md create mode 100644 workflow/beta/copy-style/README.md create mode 100644 workflow/beta/copy-style/scripts/prepare.py create mode 100644 workflow/beta/copy-style/workflow.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 40c6a8c..ecbe170 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,16 @@ that may never merge. They are not releases and are not listed here. ## Unreleased +- `mapbox workflow`, beta: install and run workflows, which are named, + multi-step recipes of `mapbox` commands and scripts defined in a + `workflow.yaml`. `install` copies one from a local directory or from a + GitHub repository's `workflow///` (by default `mapbox/cli`; + `GH_TOKEN`/`GITHUB_TOKEN` for a private one) into `~/.mapbox/workflows/`, + and `list`, `show`, `run` and `uninstall` work on what is installed. The + first one published is `copy-style`, which copies a style between + accounts. The command and the `version: 1` format are beta and may change. + Nothing changes for a script that does not use them. + - New command: `mapbox styles download > style.zip` saves a style as a ZIP with its sprite icons and custom fonts. `mapbox auth login` now also asks for the `styles:download` scope it needs, so log in again diff --git a/docs/commands.md b/docs/commands.md index c0fd1c1..c865928 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -111,6 +111,12 @@ nests, and is typed `mapbox styles draft get`. [tilesets.get-mvt](#mapbox-tilesets-get-mvt) · [tilesets.query](#mapbox-tilesets-query) +**[Workflows](#workflows)** — [workflow.list](#mapbox-workflow-list) · +[workflow.show](#mapbox-workflow-show) · +[workflow.install](#mapbox-workflow-install) · +[workflow.uninstall](#mapbox-workflow-uninstall) · +[workflow.run](#mapbox-workflow-run) + **[Tilesets CLI](#tilesets-cli)** — [tilesets-cli](#mapbox-tilesets-cli-args) Then [Errors](#errors) — the shape a failure takes in each mode. @@ -3852,6 +3858,208 @@ list drops the `--daily` suggestion once it's already in effect. --- +## Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts, +defined in a `workflow.yaml`. **Beta:** the command and the `version: 1` +format may change. The format and the rules a workflow directory follows are +in [workflow/README.md](../workflow/README.md). + +None ships inside the binary. `install` copies one into +`~/.mapbox/workflows//`, and every other subcommand works on what is +installed there. A `beta` workflow says so on stderr when it is installed or +run. + +--- + +### `mapbox workflow list` + +Lists the installed workflows. One that no longer loads, because a file was +edited by hand or this CLI no longer reads its format, is listed with its +error instead of a summary. + +#### Outputs + + + + +
textjson
+ +``` +copy-style (beta) + Copy a style from one account to another +``` + + + +```json +[ + { + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "github:mapbox/cli@main", + "stage": "beta", + "summary": "Copy a style from one account to another" + } +] +``` + +
+ +--- + +### `mapbox workflow show` + +Describes an installed workflow: its inputs, with their types and defaults, +and its steps. When a step names a command or an argument this build no +longer has, the problems are listed at the end and under `problems`. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | + +--- + +### `mapbox workflow install` + +Installs a workflow from GitHub or from a local directory. Everything is read +and checked before anything is written: the layout, the `workflow.yaml`, and +every command step against this build's commands. A workflow that fails any +of these is `invalid_workflow`, with every problem listed at once. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `SOURCE` | A workflow name, looked up in the repository's `workflow///`, or a path to a local workflow directory: anything with a `/` or starting with `.`. | +| `--repo` | GitHub repository to install from, as `OWNER/REPO`. Defaults to `mapbox/cli`. | +| `--ref` | Branch, tag or commit to install from. Defaults to `main`. | +| `--force` | Replace a workflow that is already installed. | +| `--dry-run` | Check the workflow and list the files it would write, then exit. | + +The repository is read as one tarball through the GitHub API. For a private +repository, such as `mapbox/cli` today, set `GH_TOKEN` or `GITHUB_TOKEN`. It is +sent only to `api.github.com`. Without one, a private repository answers 404, +exactly as a ref that does not exist does, and the error says both. + +**An installed workflow stops the install** unless `--force` is given. The new +copy is staged and renamed into place, so an interrupted install leaves the +old copy or the new one. Only regular files are taken. A symlink in a local +directory is refused, and an archive entry whose path would leave the +workflow's directory stops the read. + +#### Examples + +```sh +export GITHUB_TOKEN="$(gh auth token)" +mapbox workflow install copy-style + +mapbox workflow install copy-style --ref v0.4.0 + +mapbox workflow install ./workflow/beta/copy-style --force +``` + +#### Outputs + + + + +
textjson
+ +``` +Installed `copy-style` from +/Users/me/dev/cli/workflow/beta/copy-style +into /Users/me/.mapbox/workflows/copy-style. +Run it with `mapbox workflow run copy-style`. +``` + + + +```json +{ + "dry_run": false, + "files": [ + "README.md", + "scripts/prepare.py", + "workflow.yaml" + ], + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "/Users/me/dev/cli/workflow/beta/copy-style", + "stage": "beta" +} +``` + +
+ +--- + +### `mapbox workflow uninstall` + +Removes an installed workflow's directory. At a terminal it asks first, and +`--yes` skips the question. Only a plain workflow name is accepted, and only a +directory `install` wrote is removed. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | +| `--dry-run` | Say what it would remove, then exit without removing it. | + +--- + +### `mapbox workflow run` + +Runs an installed workflow's steps in order. The first step that fails stops +the run with `workflow_failed`, naming the step, and exit code 1. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | +| `--input`, `-i` | `KEY=VALUE` for one of the workflow's inputs, repeatable. The value is read as the input's declared type. | +| `--dry-run` | Check the workflow and its inputs and print the plan, then exit without running a step. | + +**stdout holds only the result**: the workflow's `outputs`, or the last +step's output if it declares none, rendered like any other result. Each +step's progress line and anything a step writes to stderr go to stderr. + +Each command step is this binary run again with `--output json`, so it +resolves its token and applies its timeouts as the same command typed by +hand would, and appears in `mapbox history` as its own run. The globals given +to `workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, +`--yes`, `--debug`, `--token`) reach every command step that does not set its +own. A token typed as `--token` goes to the step's environment, not its +command line. + +#### Examples + +```sh +mapbox workflow run copy-style \ + --input style_id=cmm28c5rm00bj01qz9hwp69qc \ + --input from_profile=source \ + --input to_profile=target + +mapbox workflow run copy-style -i style_id=cmm28c5rm00bj01qz9hwp69qc \ + -i from_profile=source -i to_profile=target --dry-run +``` + +`--dry-run` prints the plan: the inputs as they were read, and each step with +its arguments as written, since nothing has run to fill them in. + +``` +Would run `copy-style`: + 1. source mapbox styles get + 2. body python3 scripts/prepare.py + 3. created mapbox styles create +``` + +--- + ## Tilesets CLI ### `mapbox tilesets-cli ` diff --git a/src/main.rs b/src/main.rs index de67a96..9723730 100644 --- a/src/main.rs +++ b/src/main.rs @@ -39,6 +39,7 @@ mod telemetry; mod tilesets_cli; mod uninstall; mod update_check; +mod workflow; use output::{CliError, Mode}; use remedy::Remedy; @@ -637,6 +638,7 @@ fn build_app(specs: &[ServiceSpec]) -> Command { app = app.subcommand(account_usage::command()); + app = app.subcommand(workflow::command()); app.subcommand(tilesets_cli::command()) } @@ -1135,6 +1137,19 @@ fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) - Some(("show", show_matches)) => history::show(show_matches, mode)?, _ => unreachable!("`history` sets subcommand_required(true)"), }, + // Ahead of the generic service arm too. Its own requests go to + // GitHub, and each command step is a child `mapbox` that resolves + // its credentials itself — so nothing is loaded here. + Some((workflow::COMMAND, workflow_matches)) => workflow::run( + app, + workflow_matches, + workflow::RunFlags { + globals: matches, + debug, + assume_yes, + }, + mode, + )?, // Also ahead of the generic service arm: read-only except for the // opt-in `--verify` request, and needs no credential load of its own // — it reports what one would resolve to, not what a fresh one diff --git a/src/schema.rs b/src/schema.rs index de31d92..1a726c1 100644 --- a/src/schema.rs +++ b/src/schema.rs @@ -387,6 +387,14 @@ fn commands(app: &Command, specs: &[ServiceSpec], path: &[String]) -> Vec>; + +/// A checked workflow. Nothing constructs one except [`parse`]. +#[derive(Debug, Clone)] +pub struct Workflow { + pub name: String, + pub summary: String, + pub description: Option, + pub inputs: BTreeMap, + pub steps: Vec, + pub outputs: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Input { + #[serde(rename = "type")] + pub kind: InputType, + #[serde(default)] + pub description: Option, + #[serde(default)] + pub required: bool, + #[serde(default)] + pub default: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum InputType { + String, + Number, + Boolean, +} + +impl InputType { + pub fn as_str(self) -> &'static str { + match self { + InputType::String => "string", + InputType::Number => "number", + InputType::Boolean => "boolean", + } + } + + pub fn accepts(self, value: &Value) -> bool { + matches!( + (self, value), + (InputType::String, Value::String(_)) + | (InputType::Number, Value::Number(_)) + | (InputType::Boolean, Value::Bool(_)) + ) + } +} + +#[derive(Debug, Clone)] +pub struct Step { + pub id: String, + pub name: Option, + pub action: Action, + pub stdin: Option, +} + +#[derive(Debug, Clone)] +pub enum Action { + Command { + /// `styles get` as `["styles", "get"]`. + path: Vec, + args: Map, + }, + Script { + /// Relative to `scripts/`. + script: PathBuf, + interpreter: String, + args: Vec, + }, +} + +impl Step { + /// How progress and plans name the step. + pub fn label(&self) -> String { + match &self.action { + Action::Command { path, .. } => format!("mapbox {}", path.join(" ")), + Action::Script { + script, + interpreter, + .. + } => format!("{interpreter} {SCRIPTS_DIR}/{}", slash_path(script)), + } + } +} + +/// The file as written, before anything is checked. +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Raw { + version: u64, + name: String, + summary: String, + #[serde(default)] + description: Option, + #[serde(default)] + inputs: BTreeMap, + steps: Vec, + #[serde(default)] + outputs: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct RawStep { + id: String, + #[serde(default)] + name: Option, + #[serde(default)] + command: Option, + #[serde(default)] + script: Option, + #[serde(default)] + interpreter: Option, + #[serde(default)] + args: Option, + #[serde(default)] + stdin: Option, +} + +/// A workflow name is one directory name, lower-case and dash-separated — +/// `uninstall` turns it into a path it deletes, so nothing else may pass. +pub fn is_workflow_name(name: &str) -> bool { + !name.is_empty() + && name.len() <= 64 + && name.starts_with(|c: char| c.is_ascii_lowercase() || c.is_ascii_digit()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') +} + +/// Input names and step ids, which expressions refer to. +fn is_identifier(name: &str) -> bool { + name.starts_with(|c: char| c.is_ascii_lowercase()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_') +} + +/// A path that stays inside the directory it is relative to. +pub fn is_contained(path: &Path) -> bool { + !path.as_os_str().is_empty() + && path + .components() + .all(|component| matches!(component, Component::Normal(_))) +} + +pub fn slash_path(path: &Path) -> String { + path.components() + .map(|part| part.as_os_str().to_string_lossy()) + .collect::>() + .join("/") +} + +/// The interpreter a script runs under when the step names none. A short, +/// fixed list: guessing further is how a script ends up run by the wrong +/// program. +fn default_interpreter(script: &Path) -> Option<&'static str> { + match script.extension()?.to_str()? { + "sh" => Some("sh"), + "py" => Some("python3"), + "js" | "mjs" => Some("node"), + _ => None, + } +} + +/// Checks `files` as the workflow called `name`, and returns it or every +/// problem found — all of them at once, so fixing a workflow is not one +/// install attempt per mistake. +pub fn parse(name: &str, files: &Files) -> Result> { + let mut problems = vec![]; + + for path in files.keys() { + let allowed = path == Path::new(DEFINITION_FILE) + || path == Path::new(README_FILE) + || path.starts_with(SCRIPTS_DIR); + if !allowed { + problems.push(format!( + "`{}` does not belong in a workflow: only {DEFINITION_FILE}, {README_FILE} \ + and {SCRIPTS_DIR}/ do", + slash_path(path) + )); + } + } + + let Some(bytes) = files.get(Path::new(DEFINITION_FILE)) else { + problems.push(format!("there is no {DEFINITION_FILE}")); + return Err(problems); + }; + let raw: Raw = match serde_yaml::from_slice(bytes) { + Ok(raw) => raw, + Err(e) => { + problems.push(format!("{DEFINITION_FILE} is not valid: {e}")); + return Err(problems); + } + }; + + if raw.version != SCHEMA_VERSION { + problems.push(format!( + "`version: {}` is not one this CLI reads; it reads `version: {SCHEMA_VERSION}`", + raw.version + )); + } + if raw.name != name { + problems.push(format!( + "`name: {}` must match the directory it is in, `{name}`", + raw.name + )); + } + if !is_workflow_name(&raw.name) { + problems.push(format!( + "`name: {}` must be lower-case letters, digits and dashes", + raw.name + )); + } + if raw.summary.trim().is_empty() || raw.summary.contains('\n') { + problems.push("`summary` must be one non-empty line".to_string()); + } + + for (input, spec) in &raw.inputs { + if !is_identifier(input) { + problems.push(format!( + "input `{input}` must be lower-case letters, digits and underscores" + )); + } + if let Some(default) = &spec.default { + if spec.required { + problems.push(format!( + "input `{input}` is required and has a default; it can only be one" + )); + } + if !spec.kind.accepts(default) { + problems.push(format!( + "input `{input}`'s default is not a {}", + spec.kind.as_str() + )); + } + } + } + + if raw.steps.is_empty() { + problems.push("`steps` is empty".to_string()); + } + + let mut steps = vec![]; + let mut seen: BTreeSet = BTreeSet::new(); + let mut referenced_scripts: BTreeSet = BTreeSet::new(); + for raw_step in raw.steps { + let id = raw_step.id.clone(); + if !is_identifier(&id) { + problems.push(format!( + "step `{id}`: the id must be lower-case letters, digits and underscores" + )); + } + if seen.contains(&id) { + problems.push(format!("step `{id}` appears twice")); + } + + for value in raw_step.args.iter().chain(raw_step.stdin.iter()) { + check_references( + value, + &format!("step `{id}`"), + &raw.inputs, + &seen, + &mut problems, + ); + } + + let action = match (raw_step.command, raw_step.script) { + (Some(command), None) => { + if raw_step.interpreter.is_some() { + problems.push(format!( + "step `{id}`: `interpreter` only applies to a `script` step" + )); + } + let path: Vec = command.split_whitespace().map(String::from).collect(); + if path.is_empty() { + problems.push(format!("step `{id}`: `command` is empty")); + } + let args = match raw_step.args { + None => Map::new(), + Some(Value::Object(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a command's `args` is a mapping of argument name to value" + )); + Map::new() + } + }; + Some(Action::Command { path, args }) + } + (None, Some(script)) => { + let script = PathBuf::from(script); + let under_scripts = Path::new(SCRIPTS_DIR).join(&script); + if !is_contained(&script) { + problems.push(format!( + "step `{id}`: `script: {}` must be a path inside {SCRIPTS_DIR}/", + script.display() + )); + } else if !files.contains_key(&under_scripts) { + problems.push(format!( + "step `{id}`: there is no {SCRIPTS_DIR}/{}", + slash_path(&script) + )); + } + referenced_scripts.insert(under_scripts); + + let interpreter = match raw_step + .interpreter + .or_else(|| default_interpreter(&script).map(String::from)) + { + Some(interpreter) + if !interpreter.is_empty() + && !interpreter.contains(char::is_whitespace) => + { + interpreter + } + Some(interpreter) => { + problems.push(format!( + "step `{id}`: `interpreter: {interpreter}` must be one program name" + )); + String::new() + } + None => { + problems.push(format!( + "step `{id}`: name an `interpreter` for {}; only .sh, .py and .js \ + have a default", + slash_path(&script) + )); + String::new() + } + }; + let args = match raw_step.args { + None => vec![], + Some(Value::Array(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a script's `args` is a list of values" + )); + vec![] + } + }; + Some(Action::Script { + script, + interpreter, + args, + }) + } + (Some(_), Some(_)) => { + problems.push(format!( + "step `{id}` has both `command` and `script`; a step is one or the other" + )); + None + } + (None, None) => { + problems.push(format!("step `{id}` needs a `command` or a `script`")); + None + } + }; + + seen.insert(id.clone()); + if let Some(action) = action { + steps.push(Step { + id, + name: raw_step.name, + action, + stdin: raw_step.stdin, + }); + } + } + + if let Some(outputs) = &raw.outputs { + check_references(outputs, "`outputs`", &raw.inputs, &seen, &mut problems); + } + + for path in files.keys().filter(|path| path.starts_with(SCRIPTS_DIR)) { + if !referenced_scripts.contains(path) { + problems.push(format!( + "`{}` is not used by any step; a workflow ships only the scripts it runs", + slash_path(path) + )); + } + } + + if !problems.is_empty() { + return Err(problems); + } + Ok(Workflow { + name: raw.name, + summary: raw.summary, + description: raw.description, + inputs: raw.inputs, + steps, + outputs: raw.outputs, + }) +} + +/// Every expression in `value` must name a declared input or a step that +/// has already run by the time `value` is read. +fn check_references( + value: &Value, + owner: &str, + inputs: &BTreeMap, + earlier_steps: &BTreeSet, + problems: &mut Vec, +) { + let references = match template::references(value) { + Ok(references) => references, + Err(e) => { + problems.push(format!("{owner}: {e}")); + return; + } + }; + for reference in references { + match &reference { + Reference::Input(name) if !inputs.contains_key(name) => problems.push(format!( + "{owner}: `{}` names no declared input", + template::display(&reference) + )), + Reference::Step { id, .. } if !earlier_steps.contains(id) => problems.push(format!( + "{owner}: `{}` names no earlier step", + template::display(&reference) + )), + _ => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const MINIMAL: &str = "\ +version: 1 +name: demo +summary: A demo +inputs: + style_id: { type: string, required: true } +steps: + - id: fetch + command: styles get + args: { style-id: '${{ inputs.style_id }}' } + - id: shape + script: shape.py + stdin: '${{ steps.fetch.output }}' +outputs: + id: '${{ steps.shape.output.id }}' +"; + + fn files(entries: &[(&str, &str)]) -> Files { + entries + .iter() + .map(|(path, body)| (PathBuf::from(path), body.as_bytes().to_vec())) + .collect() + } + + fn problems(entries: &[(&str, &str)]) -> Vec { + parse("demo", &files(entries)).expect_err("should be refused") + } + + #[test] + fn a_minimal_workflow_parses() { + let workflow = parse( + "demo", + &files(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + (README_FILE, ""), + ]), + ) + .unwrap(); + assert_eq!(workflow.steps.len(), 2); + assert_eq!(workflow.steps[0].label(), "mapbox styles get"); + assert_eq!(workflow.steps[1].label(), "python3 scripts/shape.py"); + } + + #[test] + fn the_name_must_match_the_directory() { + let found = parse( + "other", + &files(&[(DEFINITION_FILE, MINIMAL), ("scripts/shape.py", "")]), + ) + .unwrap_err(); + assert!(found.iter().any(|p| p.contains("must match")), "{found:?}"); + } + + #[test] + fn stray_and_unused_files_are_refused() { + let found = problems(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + ("scripts/old.py", ""), + ("notes.txt", ""), + ]); + assert!( + found.iter().any(|p| p.contains("scripts/old.py")), + "{found:?}" + ); + assert!(found.iter().any(|p| p.contains("notes.txt")), "{found:?}"); + } + + #[test] + fn a_missing_script_is_refused() { + let found = problems(&[(DEFINITION_FILE, MINIMAL)]); + assert!( + found.iter().any(|p| p.contains("no scripts/shape.py")), + "{found:?}" + ); + } + + #[test] + fn a_script_path_cannot_leave_scripts() { + let yaml = MINIMAL.replace("script: shape.py", "script: ../workflow.yaml"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("inside scripts/")), + "{found:?}" + ); + } + + #[test] + fn a_reference_must_point_backwards() { + let yaml = MINIMAL.replace( + "args: { style-id: '${{ inputs.style_id }}' }", + "args: { style-id: '${{ steps.shape.output.id }}' }", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no earlier step")), + "{found:?}" + ); + } + + #[test] + fn an_undeclared_input_is_refused() { + let yaml = MINIMAL.replace("inputs.style_id }}'", "inputs.nope }}'"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no declared input")), + "{found:?}" + ); + } + + #[test] + fn an_unknown_field_is_refused() { + let yaml = MINIMAL.replace("summary: A demo", "summary: A demo\nretries: 3"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!(found.iter().any(|p| p.contains("retries")), "{found:?}"); + } + + #[test] + fn an_unknown_extension_needs_an_interpreter() { + let yaml = MINIMAL.replace("shape.py", "shape.rb"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.rb", "")]); + assert!(found.iter().any(|p| p.contains("interpreter")), "{found:?}"); + } + + #[test] + fn workflow_names_are_one_plain_directory_name() { + for good in ["copy-style", "a", "v2-sync"] { + assert!(is_workflow_name(good), "{good}"); + } + for bad in ["", "..", "a/b", "/tmp", "Copy", "-x", "a_b", "a.b"] { + assert!(!is_workflow_name(bad), "{bad}"); + } + } +} diff --git a/src/workflow/mod.rs b/src/workflow/mod.rs new file mode 100644 index 0000000..48fbcc8 --- /dev/null +++ b/src/workflow/mod.rs @@ -0,0 +1,494 @@ +//! `mapbox workflow` — install and run workflows: named, multi-step recipes +//! of `mapbox` commands and scripts. Beta. +//! +//! A workflow is a directory holding a `workflow.yaml` and the scripts it +//! runs (see [`definition`] for the schema and the layout rules). None ships +//! inside the binary. `install` copies one in, from a local directory or a +//! GitHub repository's `workflow///`, and `run` runs only what +//! is installed. +//! +//! What this refuses to be: a scheduler, a retry engine, or a language. +//! Steps run in order and the first failure stops the run. Logic belongs in +//! a script step. + +pub mod definition; +pub mod runner; +pub mod store; +pub mod template; + +use anyhow::Result; +use clap::{Arg, ArgAction, ArgMatches, Command}; +use serde_json::{json, Value}; + +use crate::confirm; +use crate::executor; +use crate::output::{self, field_lines, Mode}; + +pub const COMMAND: &str = "workflow"; + +const NAME_ARG: &str = "name"; +const SOURCE_ARG: &str = "source"; +const REPO_ARG: &str = "repo"; +const REF_ARG: &str = "ref"; +const FORCE_ARG: &str = "force"; +const INPUT_ARG: &str = "input"; + +pub fn command() -> Command { + let name = || { + Arg::new(NAME_ARG) + .value_name("NAME") + .required(true) + .help("Name of an installed workflow") + }; + Command::new(COMMAND) + .about("Install and run multi-step workflows of mapbox commands and scripts (beta)") + .long_about( + "Install and run workflows: named, multi-step recipes of mapbox commands and \ + scripts, defined in a workflow.yaml.\n\n\ + A workflow runs only once it is installed, from a local directory or from a \ + GitHub repository's workflow/// directory. Workflows and this \ + command are beta: their format may change.", + ) + .subcommand_required(true) + .subcommand(Command::new("list").about("List installed workflows")) + .subcommand( + Command::new("show") + .about("Describe an installed workflow: its inputs and steps") + .arg(name()), + ) + .subcommand( + Command::new("install") + .about("Install a workflow from a local directory or from GitHub") + .arg( + Arg::new(SOURCE_ARG) + .value_name("SOURCE") + .required(true) + .help( + "A workflow name, fetched from GitHub, or a path to a local \ + workflow directory (anything with a `/` or starting with `.`)", + ), + ) + .arg( + Arg::new(REPO_ARG) + .long(REPO_ARG) + .value_name("OWNER/REPO") + .help(format!( + "GitHub repository to install from [default: {}]. Reads \ + GH_TOKEN or GITHUB_TOKEN for a private one", + store::DEFAULT_REPO + )), + ) + .arg( + Arg::new(REF_ARG) + .long(REF_ARG) + .value_name("REF") + .help(format!( + "Branch, tag or commit to install from [default: {}]", + store::DEFAULT_REF + )), + ) + .arg( + Arg::new(FORCE_ARG) + .long(FORCE_ARG) + .action(ArgAction::SetTrue) + .help("Replace a workflow that is already installed"), + ) + .arg(executor::dry_run_arg( + "Check the workflow and list the files it would write, then exit", + )), + ) + .subcommand( + Command::new("uninstall") + .about("Remove an installed workflow") + .arg(name()) + .arg(executor::dry_run_arg( + "Say what it would remove, then exit without removing it", + )), + ) + .subcommand( + Command::new("run") + .about("Run an installed workflow") + .arg(name()) + .arg( + Arg::new(INPUT_ARG) + .long(INPUT_ARG) + .short('i') + .value_name("KEY=VALUE") + .action(ArgAction::Append) + .help("A value for one of the workflow's inputs, repeatable"), + ) + .arg(executor::dry_run_arg( + "Check the workflow and its inputs and print the plan, then exit \ + without running a step", + )), + ) +} + +/// What the globals say, for the subcommands that need them. +pub struct RunFlags<'a> { + pub globals: &'a ArgMatches, + pub debug: bool, + pub assume_yes: bool, +} + +pub fn run(app: &Command, matches: &ArgMatches, flags: RunFlags, mode: Mode) -> Result<()> { + match matches.subcommand() { + Some(("list", _)) => list(mode), + Some(("show", m)) => show(app, name(m), mode), + Some(("install", m)) => install(app, m, flags.debug, mode), + Some(("uninstall", m)) => { + uninstall(name(m), executor::wants_dry_run(m), flags.assume_yes, mode) + } + Some(("run", m)) => run_workflow(app, m, flags.globals, mode), + _ => unreachable!("`workflow` sets subcommand_required(true)"), + } +} + +fn name(matches: &ArgMatches) -> &str { + matches.get_one::(NAME_ARG).expect("required") +} + +fn stage_label(stage: Option<&str>) -> String { + stage.unwrap_or("local").to_string() +} + +fn warn_if_beta(workflow: &str, stage: Option<&str>) { + if stage == Some("beta") { + output::progress(&format!( + "`{workflow}` is a beta workflow: its inputs and outputs may change." + )); + } +} + +fn source_label(meta: &store::Meta) -> String { + match &meta.git_ref { + Some(git_ref) => format!("{}@{git_ref}", meta.source), + None => meta.source.clone(), + } +} + +fn list(mode: Mode) -> Result<()> { + let installed = store::list()?; + let mut rows = vec![]; + let mut text = String::new(); + for (name, loaded) in &installed { + match loaded { + Ok(found) => { + rows.push(json!({ + "name": name, + "stage": stage_label(found.meta.stage.as_deref()), + "summary": found.workflow.summary, + "source": source_label(&found.meta), + "path": found.root, + })); + text.push_str(&format!( + "{name} ({})\n {}\n", + stage_label(found.meta.stage.as_deref()), + found.workflow.summary + )); + } + Err(e) => { + rows.push(json!({ "name": name, "error": format!("{e:#}") })); + text.push_str(&format!("{name} (broken)\n {e:#}\n")); + } + } + } + if installed.is_empty() { + text = "No workflows installed. Install one with `mapbox workflow install `.\n" + .to_string(); + } + output::emit(mode, text.trim_end(), Value::Array(rows)) +} + +fn show(app: &Command, name: &str, mode: Mode) -> Result<()> { + let found = store::load(name)?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + + let inputs: Vec = workflow + .inputs + .iter() + .map(|(name, input)| { + json!({ + "name": name, + "type": input.kind.as_str(), + "required": input.required, + "default": input.default, + "description": input.description, + }) + }) + .collect(); + let steps: Vec = workflow + .steps + .iter() + .map(|step| json!({ "id": step.id, "name": step.name, "run": step.label() })) + .collect(); + + let mut text = field_lines( + &[ + ("Name", workflow.name.clone()), + ("Stage", stage_label(found.meta.stage.as_deref())), + ("Summary", workflow.summary.clone()), + ("Source", source_label(&found.meta)), + ("Path", found.root.display().to_string()), + ], + output::result_in_color(), + ); + if let Some(description) = &workflow.description { + text.push_str(&format!("\n\n{}", description.trim_end())); + } + text.push_str("\n\nInputs:"); + if workflow.inputs.is_empty() { + text.push_str(" none"); + } + for (name, input) in &workflow.inputs { + let mut line = format!("\n {name} ({}", input.kind.as_str()); + if input.required { + line.push_str(", required"); + } + if let Some(default) = &input.default { + line.push_str(&format!(", default {default}")); + } + line.push(')'); + if let Some(description) = &input.description { + line.push_str(&format!(" {description}")); + } + text.push_str(&line); + } + text.push_str("\n\nSteps:"); + for (index, step) in workflow.steps.iter().enumerate() { + let title = step.name.as_deref().unwrap_or(&step.id); + text.push_str(&format!("\n {}. {title} {}", index + 1, step.label())); + } + if !problems.is_empty() { + text.push_str("\n\nThis CLI cannot run it:"); + for problem in &problems { + text.push_str(&format!("\n - {problem}")); + } + } + + output::emit( + mode, + &text, + json!({ + "name": workflow.name, + "stage": stage_label(found.meta.stage.as_deref()), + "summary": workflow.summary, + "description": workflow.description, + "source": source_label(&found.meta), + "path": found.root, + "inputs": inputs, + "steps": steps, + "problems": problems, + }), + ) +} + +/// A source with a path separator or a leading `.` is a directory; anything +/// else is a name to look up on GitHub. A workflow name can hold neither, +/// so the two never overlap. +fn is_local_source(source: &str) -> bool { + source.starts_with('.') || source.contains('/') || source.contains('\\') +} + +fn install(app: &Command, matches: &ArgMatches, debug: bool, mode: Mode) -> Result<()> { + let source = matches.get_one::(SOURCE_ARG).expect("required"); + let repo = matches.get_one::(REPO_ARG); + let git_ref = matches.get_one::(REF_ARG); + let force = matches.get_flag(FORCE_ARG); + let dry_run = executor::wants_dry_run(matches); + + let package = if is_local_source(source) { + if repo.is_some() || git_ref.is_some() { + return Err(output::CliError::new( + "invalid_arguments", + "`--repo` and `--ref` name a GitHub source; this one is a local directory.", + ) + .into()); + } + store::read_local(std::path::Path::new(source))? + } else { + store::fetch_github( + source, + repo.map_or(store::DEFAULT_REPO, String::as_str), + git_ref.map_or(store::DEFAULT_REF, String::as_str), + debug, + )? + }; + let workflow = &package.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let files: Vec = package + .files + .keys() + .map(|path| definition::slash_path(path)) + .collect(); + let target = store::root_path()?.join(&workflow.name); + let summary = json!({ + "name": workflow.name, + "stage": stage_label(package.meta.stage.as_deref()), + "source": source_label(&package.meta), + "path": target, + "files": files, + "dry_run": dry_run, + }); + + if dry_run { + if target.exists() && !force { + return Err(store::already_installed(&workflow.name, &target)); + } + warn_if_beta(&workflow.name, package.meta.stage.as_deref()); + let text = format!( + "Would install `{}` from {} into {}:\n{}", + workflow.name, + source_label(&package.meta), + target.display(), + files + .iter() + .map(|file| format!(" {file}")) + .collect::>() + .join("\n") + ); + return output::emit(mode, &text, summary); + } + + let target = store::install(&package, force)?; + warn_if_beta(&workflow.name, package.meta.stage.as_deref()); + let text = format!( + "Installed `{}` from {} into {}.\nRun it with `mapbox workflow run {}`.", + workflow.name, + source_label(&package.meta), + target.display(), + workflow.name + ); + output::emit(mode, &text, summary) +} + +fn uninstall(name: &str, dry_run: bool, assume_yes: bool, mode: Mode) -> Result<()> { + let found = store::installed_dir(name)?; + let Some(dir) = found else { + // `load` words the error, with the commands to try instead. + return store::load(name).map(|_| ()); + }; + if dry_run { + return output::emit( + mode, + &format!("Would remove {}.", dir.display()), + json!({ "name": name, "path": dir, "dry_run": true }), + ); + } + confirm::destructive_local_action( + &format!("Remove the workflow at {}?", dir.display()), + assume_yes, + )?; + let dir = store::uninstall(name)?; + output::emit( + mode, + &format!("Removed `{name}` from {}.", dir.display()), + json!({ "name": name, "path": dir, "dry_run": false }), + ) +} + +fn run_workflow( + app: &Command, + matches: &ArgMatches, + globals: &ArgMatches, + mode: Mode, +) -> Result<()> { + let found = store::load(name(matches))?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let pairs: Vec = matches + .get_many::(INPUT_ARG) + .into_iter() + .flatten() + .cloned() + .collect(); + let inputs = runner::read_inputs(workflow, &pairs)?; + + if executor::wants_dry_run(matches) { + let plan = runner::plan(workflow, &inputs); + let text = + std::iter::once(format!("Would run `{}`:", workflow.name)) + .chain( + workflow.steps.iter().enumerate().map(|(index, step)| { + format!(" {}. {} {}", index + 1, step.id, step.label()) + }), + ) + .collect::>() + .join("\n"); + return output::emit(mode, &text, plan); + } + + warn_if_beta(&workflow.name, found.meta.stage.as_deref()); + let inherited = runner::Inherited::from_matches(globals); + let result = runner::run(app, workflow, &found.root, &inputs, &inherited)?; + output::emit_value(mode, &result, None, None, None) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::Path; + + #[test] + fn a_name_and_a_path_never_look_alike() { + for local in [ + "./copy-style", + "../x", + "workflow/beta/copy-style", + "/abs", + ".", + "a\\b", + ] { + assert!(is_local_source(local), "{local}"); + } + assert!(!is_local_source("copy-style")); + } + + /// Every workflow this repository publishes is one `install` accepts. + /// The rules are the same as for anybody's, so this is also what + /// catches a stray file or a step naming a command that was renamed. + #[test] + fn every_published_workflow_is_valid() { + let app = crate::build_app(&crate::spec::effective_services().expect("bundled specs")); + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("workflow"); + let mut checked = 0; + for stage in std::fs::read_dir(&root) + .expect("workflow/ exists") + .flatten() + { + if !stage.path().is_dir() { + continue; + } + let stage_name = stage.file_name().to_string_lossy().into_owned(); + assert!( + store::STAGES.contains(&stage_name.as_str()), + "workflow/{stage_name}/ is not a stage; add it to store::STAGES or move it" + ); + for dir in std::fs::read_dir(stage.path()) + .expect("read stage") + .flatten() + { + let package = store::read_local(&dir.path()) + .unwrap_or_else(|e| panic!("{}: {e:#}", dir.path().display())); + let problems = runner::command_problems(&app, &package.workflow); + assert!( + problems.is_empty(), + "{}: {problems:#?}", + dir.path().display() + ); + assert_eq!(package.meta.stage.as_deref(), Some(stage_name.as_str())); + checked += 1; + } + } + assert!(checked > 0, "no workflows found under workflow/"); + } +} diff --git a/src/workflow/runner.rs b/src/workflow/runner.rs new file mode 100644 index 0000000..aa96576 --- /dev/null +++ b/src/workflow/runner.rs @@ -0,0 +1,601 @@ +//! Runs a checked workflow, one step at a time. +//! +//! A `command` step is this binary run again as a child, with `--output json` +//! and the step's arguments turned into flags. That is the point rather than +//! a shortcut: the child resolves its token, applies its timeouts, encodes +//! its path segments and records its run exactly as a command typed by hand +//! would, so a workflow cannot reach the API any way a person could not. +//! +//! Only the child's stdout is captured, as the step's output. Its stderr and +//! stdin are the terminal's, so progress, warnings and a confirmation prompt +//! reach the person running the workflow — unless the step feeds `stdin` +//! itself. +//! +//! A `script` step runs its file from the installed workflow's `scripts/` +//! under the interpreter the definition settled on. It is given `MAPBOX_CLI`, +//! the path to this binary, so it can call commands of its own. + +use std::collections::BTreeMap; +use std::ffi::OsString; +use std::io::Write; +use std::path::Path; +use std::process::{Command as Process, Stdio}; + +use anyhow::{anyhow, Context as _, Result}; +use clap::{Arg, ArgAction, Command}; +use serde_json::{json, Map, Value}; + +use super::definition::{Action, InputType, Step, Workflow, SCRIPTS_DIR}; +use super::template::{self, Context}; +use crate::auth; +use crate::output::{self, CliError}; + +/// Global options a step may not set, because the runner owns them or +/// because they would put a credential in a file. +const RESERVED: &[(&str, &str)] = &[ + ("output", "the runner reads every step as JSON"), + ("quiet", "the runner sets it"), + ( + "schema", + "a workflow runs commands rather than describing them", + ), + ( + "dry-run", + "use `mapbox workflow run --dry-run` for the whole workflow", + ), + ( + "token", + "a token written into a workflow is a secret in a file; use `profile`", + ), +]; + +/// The globals of the `workflow run` line that each command step inherits +/// unless it sets its own. +#[derive(Debug, Default, Clone)] +pub struct Inherited { + /// Only a token typed as `--token`; one from the environment reaches the + /// child by being in its environment already. + pub typed_token: Option, + pub options: Vec<(&'static str, String)>, + pub flags: Vec<&'static str>, +} + +impl Inherited { + pub fn from_matches(matches: &clap::ArgMatches) -> Self { + let mut inherited = Inherited { + typed_token: auth::typed_token(matches), + ..Default::default() + }; + for option in ["profile", "username", crate::http::TIMEOUT_ARG] { + let typed = + matches.value_source(option) == Some(clap::parser::ValueSource::CommandLine); + if let (true, Some(value)) = (typed, matches.get_one::(option)) { + inherited.options.push((option, value.clone())); + } + } + for flag in ["use-login", "debug", crate::confirm::ARG] { + if matches.get_flag(flag) { + inherited.flags.push(flag); + } + } + inherited + } +} + +/// Turns `--input key=value` pairs into the workflow's inputs, typed as the +/// definition declares, with defaults filled in. +pub fn read_inputs(workflow: &Workflow, pairs: &[String]) -> Result> { + let mut given: Map = Map::new(); + for pair in pairs { + let (key, raw) = pair + .split_once('=') + .ok_or_else(|| invalid_input(format!("`{pair}` is not KEY=VALUE"), workflow))?; + let Some(spec) = workflow.inputs.get(key) else { + return Err(invalid_input( + format!("`{}` has no input called `{key}`", workflow.name), + workflow, + )); + }; + let value = match spec.kind { + InputType::String => Value::String(raw.to_string()), + InputType::Number => raw + .parse::() + .map(Value::from) + .or_else(|_| raw.parse::().map(Value::from)) + .map_err(|_| { + invalid_input(format!("input `{key}` is a number, not `{raw}`"), workflow) + })?, + InputType::Boolean => match raw { + "true" => Value::Bool(true), + "false" => Value::Bool(false), + _ => { + return Err(invalid_input( + format!("input `{key}` is `true` or `false`, not `{raw}`"), + workflow, + )) + } + }, + }; + if given.insert(key.to_string(), value).is_some() { + return Err(invalid_input( + format!("input `{key}` is given twice"), + workflow, + )); + } + } + + let mut missing = vec![]; + for (name, spec) in &workflow.inputs { + if given.contains_key(name) { + continue; + } + match &spec.default { + Some(default) => { + given.insert(name.clone(), default.clone()); + } + None if spec.required => missing.push(name.as_str()), + None => { + given.insert(name.clone(), Value::Null); + } + } + } + if !missing.is_empty() { + let flags: Vec = missing + .iter() + .map(|name| format!("--input {name}=…")) + .collect(); + return Err(invalid_input( + format!("missing required input: {}", flags.join(" ")), + workflow, + )); + } + Ok(given) +} + +fn invalid_input(message: String, workflow: &Workflow) -> anyhow::Error { + CliError::new("invalid_input", message) + .with_remedy( + crate::remedy::Remedy::default() + .with_action(Some(format!("mapbox workflow show {}", workflow.name))), + ) + .into() +} + +/// What `--dry-run` prints: the plan, with the inputs resolved and every +/// later reference left as written, since nothing has run to fill it. +pub fn plan(workflow: &Workflow, inputs: &Map) -> Value { + let steps: Vec = workflow + .steps + .iter() + .map(|step| { + let (kind, args) = match &step.action { + Action::Command { args, .. } => ("command", Value::Object(args.clone())), + Action::Script { args, .. } => ("script", Value::Array(args.clone())), + }; + json!({ "id": step.id, "kind": kind, "run": step.label(), "args": args, "stdin": step.stdin }) + }) + .collect(); + json!({ "workflow": workflow.name, "inputs": inputs, "steps": steps, "outputs": workflow.outputs }) +} + +/// Runs every step and returns what `outputs` resolves to, or the last +/// step's output when the workflow declares none. +pub fn run( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, +) -> Result { + let exe = std::env::current_exe().context("Could not find this program's own path")?; + let mut outputs: BTreeMap = BTreeMap::new(); + let total = workflow.steps.len(); + + for (index, step) in workflow.steps.iter().enumerate() { + let title = step.name.as_deref().unwrap_or(&step.id); + output::progress(&format!( + "[{}/{total}] {title} ({})", + index + 1, + step.label() + )); + + let context = Context { + inputs, + steps: &outputs, + }; + let stdin = step + .stdin + .as_ref() + .map(|value| template::resolve(value, &context)) + .transpose() + .map_err(|e| step_failure(step, e))?; + + let mut process = match &step.action { + Action::Command { path, args } => { + let args = match template::resolve(&Value::Object(args.clone()), &context) + .map_err(|e| step_failure(step, e))? + { + Value::Object(args) => args, + _ => unreachable!("an object resolves to an object"), + }; + let argv = + command_argv(app, path, &args, inherited).map_err(|e| step_failure(step, e))?; + let mut process = Process::new(&exe); + process.args(argv); + process + } + Action::Script { + script, + interpreter, + args, + } => { + let args = template::resolve(&Value::Array(args.clone()), &context) + .map_err(|e| step_failure(step, e))?; + let mut process = Process::new(interpreter); + process.arg(root.join(SCRIPTS_DIR).join(script)); + for value in args.as_array().into_iter().flatten() { + process.arg(template::as_text(value).unwrap_or_default()); + } + process + .env("MAPBOX_CLI", &exe) + .env("MAPBOX_WORKFLOW_ROOT", root); + process + } + }; + if let Some(token) = &inherited.typed_token { + process.env(auth::CLAP_TOKEN_ENV, token); + } + + let value = execute(&mut process, stdin.as_ref(), step)?; + outputs.insert(step.id.clone(), value); + } + + let last = workflow + .steps + .last() + .and_then(|step| outputs.get(&step.id)) + .cloned() + .unwrap_or(Value::Null); + match &workflow.outputs { + Some(declared) => template::resolve( + declared, + &Context { + inputs, + steps: &outputs, + }, + ) + .map_err(|e| CliError::new("workflow_failed", format!("`outputs`: {e:#}")).into()), + None => Ok(last), + } +} + +fn step_failure(step: &Step, err: anyhow::Error) -> anyhow::Error { + CliError::new( + "workflow_failed", + format!("Step `{}` ({}): {err:#}", step.id, step.label()), + ) + .into() +} + +/// Spawns one step and reads its stdout as its output: JSON when it parses, +/// the text otherwise, nothing when there is none. +fn execute(process: &mut Process, stdin: Option<&Value>, step: &Step) -> Result { + // A script with nothing to read gets nothing, rather than a terminal it + // might block on. A command keeps the terminal, which is where a + // confirmation prompt reads its answer. + let input = match (stdin, &step.action) { + (Some(_), _) => Stdio::piped(), + (None, Action::Script { .. }) => Stdio::null(), + (None, Action::Command { .. }) => Stdio::inherit(), + }; + let mut child = process + .stdin(input) + .stdout(Stdio::piped()) + .stderr(Stdio::inherit()) + .spawn() + .map_err(|e| { + let program = process.get_program().to_string_lossy().into_owned(); + let message = if e.kind() == std::io::ErrorKind::NotFound { + format!("`{program}` is not installed or not on PATH") + } else { + format!("could not start `{program}`: {e}") + }; + step_failure(step, anyhow!(message)) + })?; + + // Written from a thread: a child that fills its stdout pipe before it + // has read all of stdin would otherwise wait on us while we wait on it. + let writer = stdin.map(|value| { + let body = match value { + Value::String(text) => text.clone(), + other => other.to_string(), + }; + let mut pipe = child.stdin.take().expect("stdin was piped"); + std::thread::spawn(move || { + // A child that exits without reading is reported by its status. + let _ = pipe.write_all(body.as_bytes()); + }) + }); + let finished = child + .wait_with_output() + .map_err(|e| step_failure(step, anyhow!("could not wait for it: {e}")))?; + if let Some(writer) = writer { + let _ = writer.join(); + } + + if !finished.status.success() { + let how = match finished.status.code() { + Some(code) => format!("exited with {code}"), + None => "was stopped by a signal".to_string(), + }; + return Err(step_failure( + step, + anyhow!("{how}; the workflow stopped here"), + )); + } + + let text = String::from_utf8(finished.stdout).map_err(|_| { + step_failure( + step, + anyhow!("its output is binary, which cannot be passed to another step"), + ) + })?; + let text = text.trim(); + if text.is_empty() { + return Ok(Value::Null); + } + Ok(serde_json::from_str(text).unwrap_or_else(|_| Value::String(text.to_string()))) +} + +enum ArgShape { + Flag, + Option { repeatable: bool }, + Positional { repeatable: bool }, +} + +/// The leaf command a step names, when it names one a workflow may run. +fn leaf<'a>(app: &'a Command, path: &[String]) -> Result<&'a Command, String> { + let named = format!("mapbox {}", path.join(" ")); + if path.first().map(String::as_str) == Some(super::COMMAND) { + return Err(format!("`{named}`: a workflow cannot run another workflow")); + } + let mut current = app; + for segment in path { + current = current + .find_subcommand(segment) + .ok_or_else(|| format!("`{named}` is not a command"))?; + } + if current.has_subcommands() { + return Err(format!("`{named}` is a group; name one of its commands")); + } + Ok(current) +} + +/// The argument a step's key names: one of the command's own, by its long +/// flag or, for a positional, its name — the same names `mapbox --schema` +/// lists — or a global option. +fn find_arg<'a>( + app: &'a Command, + command: &'a Command, + key: &str, +) -> Result<(&'a Arg, ArgShape), String> { + if let Some((_, why)) = RESERVED.iter().find(|(name, _)| *name == key) { + return Err(format!("`{key}` cannot be set by a step: {why}")); + } + let own = command + .get_arguments() + .find(|arg| arg.get_long() == Some(key) || (arg.is_positional() && arg.get_id() == key)); + let arg = own + .or_else(|| { + app.get_arguments() + .find(|arg| arg.is_global_set() && arg.get_long() == Some(key)) + }) + .ok_or_else(|| format!("`{key}` is not an argument of this command"))?; + let repeatable = matches!(arg.get_action(), ArgAction::Append); + let shape = if matches!(arg.get_action(), ArgAction::SetTrue | ArgAction::SetFalse) { + ArgShape::Flag + } else if arg.is_positional() { + ArgShape::Positional { repeatable } + } else { + ArgShape::Option { repeatable } + }; + Ok((arg, shape)) +} + +/// Every `command` step's problems against this binary's command tree — +/// checked at install and again before a run, since the CLI may have been +/// upgraded underneath an installed workflow. +pub fn command_problems(app: &Command, workflow: &Workflow) -> Vec { + let mut problems = vec![]; + for step in &workflow.steps { + let Action::Command { path, args } = &step.action else { + continue; + }; + let command = match leaf(app, path) { + Ok(command) => command, + Err(e) => { + problems.push(format!("step `{}`: {e}", step.id)); + continue; + } + }; + for (key, value) in args { + match find_arg(app, command, key) { + Err(e) => problems.push(format!("step `{}`: {e}", step.id)), + Ok((_, ArgShape::Flag)) if !(value.is_boolean() || value.is_string()) => problems + .push(format!( + "step `{}`: `{key}` is a flag; give it true or false", + step.id + )), + Ok(_) => {} + } + } + } + problems +} + +/// A step's arguments as the child's command line. +/// +/// Options are spelled `--name=value`, so a value that starts with a dash — +/// a western longitude — is never read as a flag. Positionals come last, +/// after `--`, for the same reason. +fn command_argv( + app: &Command, + path: &[String], + args: &Map, + inherited: &Inherited, +) -> Result> { + let command = leaf(app, path).map_err(|e| anyhow!(e))?; + let mut argv: Vec = path.iter().map(OsString::from).collect(); + let mut positionals: Vec<(usize, OsString)> = vec![]; + + for (key, value) in args { + let (arg, shape) = find_arg(app, command, key).map_err(|e| anyhow!(e))?; + let values: Vec<&Value> = match value { + Value::Array(items) if !matches!(shape, ArgShape::Flag) => items.iter().collect(), + Value::Null => continue, + other => vec![other], + }; + match shape { + ArgShape::Flag => match value { + Value::Bool(true) => argv.push(format!("--{key}").into()), + Value::Bool(false) => {} + Value::String(text) if text == "true" => argv.push(format!("--{key}").into()), + Value::String(text) if text == "false" => {} + _ => return Err(anyhow!("`{key}` is a flag; give it true or false")), + }, + ArgShape::Option { repeatable } | ArgShape::Positional { repeatable } + if values.len() > 1 && !repeatable => + { + return Err(anyhow!("`{key}` takes one value, not a list")); + } + ArgShape::Option { .. } => { + for value in values { + let text = template::as_text(value).unwrap_or_default(); + argv.push(format!("--{key}={text}").into()); + } + } + ArgShape::Positional { .. } => { + let index = arg.get_index().unwrap_or(usize::MAX); + for value in values { + positionals.push((index, template::as_text(value).unwrap_or_default().into())); + } + } + } + } + + for (option, value) in &inherited.options { + if !args.contains_key(*option) { + argv.push(format!("--{option}={value}").into()); + } + } + for flag in &inherited.flags { + if !args.contains_key(*flag) { + argv.push(format!("--{flag}").into()); + } + } + argv.push(format!("--{}={}", output::ARG, output::JSON).into()); + argv.push(format!("--{}", output::banner::ARG).into()); + + if !positionals.is_empty() { + positionals.sort_by_key(|(index, _)| *index); + argv.push("--".into()); + argv.extend(positionals.into_iter().map(|(_, value)| value)); + } + Ok(argv) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::workflow::definition::{parse, Files, DEFINITION_FILE}; + use std::path::PathBuf; + + fn app() -> Command { + let specs = crate::spec::effective_services().expect("bundled specs"); + crate::build_app(&specs) + } + + fn workflow(steps: &str) -> Workflow { + let yaml = format!( + "version: 1\nname: t\nsummary: t\ninputs:\n n: {{ type: number, default: 2 }}\n flag: {{ type: boolean, default: false }}\nsteps:\n{steps}" + ); + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.into_bytes())].into(); + parse("t", &files).expect("valid") + } + + #[test] + fn a_step_is_checked_against_the_real_command_tree() { + let found = command_problems( + &app(), + &workflow( + " - id: a\n command: styles nope\n\ + \x20 - id: b\n command: styles\n\ + \x20 - id: c\n command: styles get\n args: { bogus: 1, token: x }\n\ + \x20 - id: d\n command: workflow list\n", + ), + ); + let joined = found.join("\n"); + assert!( + joined.contains("`mapbox styles nope` is not a command"), + "{joined}" + ); + assert!(joined.contains("is a group"), "{joined}"); + assert!(joined.contains("`bogus` is not an argument"), "{joined}"); + assert!(joined.contains("a secret in a file"), "{joined}"); + assert!(joined.contains("cannot run another workflow"), "{joined}"); + } + + #[test] + fn arguments_become_flags_and_positionals_come_last() { + let app = app(); + let args = serde_json::json!({ + "style-id": "-abc", + "optimize": true, + "download": false, + "profile": "work", + }); + let inherited = Inherited { + options: vec![("profile", "ignored".into()), ("timeout", "30".into())], + flags: vec!["yes"], + ..Default::default() + }; + let argv = command_argv( + &app, + &["styles".into(), "get".into()], + args.as_object().unwrap(), + &inherited, + ) + .unwrap(); + let argv: Vec = argv + .iter() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + argv, + [ + "styles", + "get", + "--optimize", + "--profile=work", + "--timeout=30", + "--yes", + "--output=json", + "--quiet", + "--", + "-abc" + ] + ); + } + + #[test] + fn inputs_are_typed_and_defaulted() { + let wf = workflow(" - id: a\n command: styles list\n"); + let inputs = read_inputs(&wf, &["n=5".into()]).unwrap(); + assert_eq!(inputs["n"], serde_json::json!(5)); + assert_eq!(inputs["flag"], serde_json::json!(false)); + assert!(read_inputs(&wf, &["n=five".into()]).is_err()); + assert!(read_inputs(&wf, &["nope=1".into()]).is_err()); + assert!(read_inputs(&wf, &["n=1".into(), "n=2".into()]).is_err()); + assert!(read_inputs(&wf, &["n".into()]).is_err()); + } +} diff --git a/src/workflow/store.rs b/src/workflow/store.rs new file mode 100644 index 0000000..03d8d05 --- /dev/null +++ b/src/workflow/store.rs @@ -0,0 +1,644 @@ +//! Where installed workflows live, and how they get there. +//! +//! Nothing is bundled into the binary: a workflow runs only once it has been +//! installed, from a local directory or from a GitHub repository laid out as +//! `workflow///`, into `/workflows//`. A copy +//! is taken rather than a link kept, so what runs is what was checked at +//! install time and nothing edited since. +//! +//! The same rules as `agent-skills` hold for the write, for the same +//! reasons: everything is read and checked in memory first, an existing +//! workflow stops the install unless `--force`, only regular files are +//! taken, and a path that would leave the workflow's directory stops the +//! read altogether. The new directory is staged and renamed into place. + +use std::io::Read; +use std::path::{Component, Path, PathBuf}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{anyhow, Context, Result}; +use serde::{Deserialize, Serialize}; + +use super::definition::{self, is_contained, is_workflow_name, Files, Workflow}; +use crate::auth; +use crate::executor; +use crate::http; +use crate::output::CliError; +use crate::remedy::Remedy; + +const DIR_NAME: &str = "workflows"; + +/// Written beside an installed workflow's files, and what marks a directory +/// as one this command installed — `uninstall` removes nothing without it. +const META_FILE: &str = ".install.json"; + +/// Where the tarball comes from. The API rather than codeload directly, +/// because the API is what honors a token for a private repository; it +/// answers with a redirect to a signed codeload URL, and `reqwest` drops the +/// `Authorization` header when it follows a redirect to another host. +const GITHUB_API: &str = "https://api.github.com"; +pub const DEFAULT_REPO: &str = "mapbox/cli"; +pub const DEFAULT_REF: &str = "main"; + +/// The directory inside a repository that holds workflows, by stage. +const REPO_PREFIX: &str = "workflow"; + +/// The stages a workflow may be published under. `beta` is the only one so +/// far; a `stable/` beside it needs nothing but an entry here. +pub const STAGES: &[&str] = &["beta"]; + +/// A ceiling on what is read, from a tarball or a local directory. Far above +/// any real workflow, and there so a hostile archive cannot make this +/// allocate without bound. +const MAX_BYTES: u64 = 64 * 1024 * 1024; + +/// The files `read_local` passes over rather than refusing. Finder writes +/// `.DS_Store` into any directory it opens. +const IGNORED: &[&str] = &[".DS_Store"]; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Meta { + /// Where it came from, as a person would write it: a directory, or + /// `github:owner/repo`. + pub source: String, + #[serde(default, rename = "ref")] + pub git_ref: Option, + #[serde(default)] + pub stage: Option, + pub installed_at: u64, +} + +/// A workflow read and checked, not yet written anywhere. +pub struct Package { + pub workflow: Workflow, + pub files: Files, + pub meta: Meta, +} + +/// A workflow on disk. +pub struct Installed { + pub root: PathBuf, + pub workflow: Workflow, + pub meta: Meta, +} + +/// `/workflows`, resolved and not created. +pub fn root_path() -> Result { + auth::config_dir_path() + .map(|dir| dir.join(DIR_NAME)) + .ok_or_else(|| anyhow!("Could not determine home directory")) +} + +pub fn invalid_workflow(name: &str, problems: &[String]) -> anyhow::Error { + let listed = problems + .iter() + .map(|problem| format!(" - {problem}")) + .collect::>() + .join("\n"); + CliError::new( + "invalid_workflow", + format!("`{name}` is not a valid workflow:\n{listed}"), + ) + .into() +} + +/// A workflow name from the command line, checked before it is ever joined +/// to a path. +pub fn check_name(name: &str) -> Result<()> { + if is_workflow_name(name) { + return Ok(()); + } + Err(CliError::new( + "invalid_name", + format!("`{name}` is not a workflow name: lower-case letters, digits and dashes"), + ) + .with_remedy(Remedy::default().with_action(Some("mapbox workflow list".to_string()))) + .into()) +} + +fn now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|elapsed| elapsed.as_secs()) + .unwrap_or(0) +} + +/// Every regular file under `dir`, by path relative to it. +/// +/// A symlink is refused rather than followed: it is a way to install +/// something from outside the directory that was named. +fn read_tree(dir: &Path, skip: &[&str]) -> Result { + let mut files = Files::new(); + let mut total: u64 = 0; + let mut pending = vec![dir.to_path_buf()]; + while let Some(current) = pending.pop() { + let entries = std::fs::read_dir(¤t) + .with_context(|| format!("Could not read {}", current.display()))?; + for entry in entries { + let entry = entry.with_context(|| format!("Could not read {}", current.display()))?; + let path = entry.path(); + let relative = path.strip_prefix(dir).expect("under the directory walked"); + let name = entry.file_name(); + if current == dir && skip.iter().any(|s| name == *s) { + continue; + } + if IGNORED.iter().any(|ignored| name == *ignored) { + continue; + } + let kind = entry + .file_type() + .with_context(|| format!("Could not read {}", path.display()))?; + if kind.is_dir() { + pending.push(path); + } else if kind.is_file() { + let bytes = std::fs::read(&path) + .with_context(|| format!("Could not read {}", path.display()))?; + total += bytes.len() as u64; + if total > MAX_BYTES { + return Err(anyhow!( + "{} holds more than a workflow should", + dir.display() + )); + } + files.insert(relative.to_path_buf(), bytes); + } else { + return Err(CliError::new( + "invalid_workflow", + format!( + "{} is not a regular file; a workflow holds only files and directories", + path.display() + ), + ) + .into()); + } + } + } + Ok(files) +} + +fn package(name: &str, files: Files, meta: Meta) -> Result { + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Package { + workflow, + files, + meta, + }) +} + +/// A workflow from a directory on this machine. The directory's own name is +/// the workflow's, as it is in a repository. +pub fn read_local(dir: &Path) -> Result { + let dir = dir + .canonicalize() + .with_context(|| format!("There is no directory {}", dir.display()))?; + if !dir.is_dir() { + return Err(anyhow!("{} is not a directory", dir.display())); + } + let name = dir + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_default(); + let stage = dir + .parent() + .and_then(Path::file_name) + .map(|n| n.to_string_lossy().into_owned()) + .filter(|parent| STAGES.contains(&parent.as_str())); + let files = read_tree(&dir, &[])?; + package( + &name, + files, + Meta { + source: dir.display().to_string(), + git_ref: None, + stage, + installed_at: now(), + }, + ) +} + +/// `owner/repo`, and nothing that could move the request elsewhere. +pub fn check_repo(repo: &str) -> Result<()> { + let part = |p: &str| { + !p.is_empty() + && p != "." + && p != ".." + && p.chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.')) + }; + match repo.split_once('/') { + Some((owner, name)) if part(owner) && part(name) => Ok(()), + _ => Err(CliError::new( + "invalid_repo", + format!("`{repo}` is not a GitHub repository; write it as OWNER/REPO"), + ) + .into()), + } +} + +/// A branch, tag or commit. `/` is allowed, since a branch may hold one; +/// URL syntax and `..` are not, since the ref goes into a URL path. +pub fn check_ref(git_ref: &str) -> Result<()> { + let ok = !git_ref.is_empty() + && !git_ref + .split('/') + .any(|part| part.is_empty() || part == "." || part == "..") + && git_ref + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.' | '/')); + if ok { + return Ok(()); + } + Err(CliError::new( + "invalid_ref", + format!("`{git_ref}` is not a branch, tag or commit this can fetch"), + ) + .into()) +} + +fn github_token() -> Option { + ["GH_TOKEN", "GITHUB_TOKEN"] + .iter() + .filter_map(|name| std::env::var(name).ok()) + .find(|value| !value.trim().is_empty()) +} + +/// A workflow out of a GitHub repository's tarball. +pub fn fetch_github(name: &str, repo: &str, git_ref: &str, debug: bool) -> Result { + fetch_github_from(GITHUB_API, name, repo, git_ref, debug) +} + +fn fetch_github_from( + base: &str, + name: &str, + repo: &str, + git_ref: &str, + debug: bool, +) -> Result { + check_name(name)?; + check_repo(repo)?; + check_ref(git_ref)?; + + let url = format!("{base}/repos/{repo}/tarball/{git_ref}"); + if debug { + eprintln!("[debug] GET {url}"); + } + let token = github_token(); + let mut request = http::client()? + .get(&url) + .header("Accept", "application/vnd.github+json"); + if let Some(token) = &token { + request = request.bearer_auth(token); + } + let response = http::send(request) + .map_err(|e| executor::transport_failure("Could not reach GitHub", e))?; + + let status = response.status(); + if !status.is_success() { + // A private repository answers 404 to a request without a token, the + // same as a ref that does not exist, so both are named. + let message = match (status.as_u16(), &token) { + (404, None) => format!( + "GitHub found no `{git_ref}` in {repo}. If the repository is private, set \ + GITHUB_TOKEN to a token that can read it." + ), + (404, Some(_)) => format!( + "GitHub found no `{git_ref}` in {repo}, or the token in GH_TOKEN/GITHUB_TOKEN \ + cannot read it." + ), + _ => format!("GitHub answered {status} for {url}."), + }; + let mut remedy = Remedy::default(); + if token.is_none() { + remedy = remedy.with_fix("export GITHUB_TOKEN=\"$(gh auth token)\""); + } + return Err(CliError::http(status.as_u16(), &message) + .with_remedy(remedy) + .into()); + } + + let mut bytes = vec![]; + response + .take(MAX_BYTES) + .read_to_end(&mut bytes) + .map_err(|e| anyhow!("Could not read the archive from GitHub: {e}"))?; + let (stage, files) = extract(&bytes, name, repo)?; + package( + name, + files, + Meta { + source: format!("github:{repo}"), + git_ref: Some(git_ref.to_string()), + stage: Some(stage), + installed_at: now(), + }, + ) +} + +fn unsafe_entry(path: &Path) -> anyhow::Error { + CliError::new( + "unsafe_archive", + format!( + "Refusing to read `{}` out of the archive: it points outside the directory it \ + would be written to.", + path.display() + ), + ) + .into() +} + +/// The files of `workflow///` in a repository tarball, and the +/// stage it was found under. +fn extract(archive: &[u8], name: &str, repo: &str) -> Result<(String, Files)> { + let decoder = flate2::read::GzDecoder::new(archive); + let mut tar = tar::Archive::new(decoder.take(MAX_BYTES)); + + let mut found: std::collections::BTreeMap<(String, String), Files> = Default::default(); + for entry in tar + .entries() + .context("The archive from GitHub is not readable as a tarball")? + { + let mut entry = entry.context("The archive from GitHub ended unexpectedly")?; + // Only regular files: a link is a way out of the destination no path + // check would catch. + if entry.header().entry_type() != tar::EntryType::Regular { + continue; + } + let path = entry + .path() + .context("The archive holds an entry whose path is not valid UTF-8")? + .into_owned(); + // GitHub wraps every archive in one directory named after the + // repository and the commit, so it is stripped by position. + let mut components = path.components(); + let Some(Component::Normal(_)) = components.next() else { + return Err(unsafe_entry(&path)); + }; + let inner: PathBuf = components.collect(); + if !is_contained(&inner) { + return Err(unsafe_entry(&path)); + } + let Ok(within) = inner.strip_prefix(REPO_PREFIX) else { + continue; + }; + let mut parts = within.components(); + let (Some(stage), Some(workflow)) = (parts.next(), parts.next()) else { + continue; + }; + let relative: PathBuf = parts.collect(); + if relative.as_os_str().is_empty() { + continue; + } + let stage = stage.as_os_str().to_string_lossy().into_owned(); + let workflow = workflow.as_os_str().to_string_lossy().into_owned(); + if !STAGES.contains(&stage.as_str()) { + continue; + } + let key = (stage, workflow); + if key.1 != name { + found.entry(key).or_default(); + continue; + } + let mut bytes = vec![]; + entry + .read_to_end(&mut bytes) + .with_context(|| format!("Could not read {} out of the archive", path.display()))?; + found.entry(key).or_default().insert(relative, bytes); + } + + let mut matching: Vec<(String, Files)> = vec![]; + let mut available: Vec = vec![]; + for ((stage, workflow), files) in found { + if workflow == name { + matching.push((stage, files)); + } else { + available.push(format!("{workflow} ({stage})")); + } + } + match matching.len() { + 1 => Ok(matching.pop().expect("one")), + 0 => { + let listed = if available.is_empty() { + format!("{repo} publishes no workflows under `{REPO_PREFIX}/`.") + } else { + format!("It publishes: {}.", available.join(", ")) + }; + Err(CliError::new( + "workflow_not_found", + format!("{repo} has no workflow called `{name}`. {listed}"), + ) + .into()) + } + _ => { + let stages: Vec<&str> = matching.iter().map(|(stage, _)| stage.as_str()).collect(); + Err(CliError::new( + "ambiguous_workflow", + format!( + "{repo} publishes `{name}` under more than one stage ({}), which it \ + should not.", + stages.join(", ") + ), + ) + .into()) + } + } +} + +/// Writes `package` into place, replacing an existing install only when +/// `force` says so. Returns where it went. +pub fn install(package: &Package, force: bool) -> Result { + let name = &package.workflow.name; + check_name(name)?; + let root = root_path()?; + std::fs::create_dir_all(&root) + .with_context(|| format!("Could not create {}", root.display()))?; + let target = root.join(name); + if target.exists() && !force { + return Err(already_installed(name, &target)); + } + + let staging = root.join(format!(".{name}.staging")); + let _ = std::fs::remove_dir_all(&staging); + let result = (|| -> Result<()> { + for (relative, bytes) in &package.files { + let path = staging.join(relative); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("Could not create {}", parent.display()))?; + } + std::fs::write(&path, bytes) + .with_context(|| format!("Could not write {}", path.display()))?; + } + std::fs::write( + staging.join(META_FILE), + serde_json::to_string_pretty(&package.meta)?, + ) + .context("Could not record where the workflow came from")?; + + if target.exists() { + std::fs::remove_dir_all(&target) + .with_context(|| format!("Could not replace {}", target.display()))?; + } + std::fs::rename(&staging, &target) + .with_context(|| format!("Could not move the workflow into {}", target.display())) + })(); + let _ = std::fs::remove_dir_all(&staging); + result.map(|()| target) +} + +pub fn already_installed(name: &str, target: &Path) -> anyhow::Error { + CliError::new( + "already_installed", + format!("`{name}` is already installed at {}", target.display()), + ) + .with_remedy(Remedy::default().with_fix("Pass --force to replace it.")) + .into() +} + +fn not_installed(name: &str) -> anyhow::Error { + CliError::new( + "workflow_not_installed", + format!("No workflow called `{name}` is installed."), + ) + .with_remedy( + Remedy::default() + .with_action(Some(format!("mapbox workflow install {name}"))) + .with_action(Some("mapbox workflow list".to_string())), + ) + .into() +} + +/// An installed workflow's directory, if there is one. Only a directory +/// holding [`META_FILE`] counts. +pub fn installed_dir(name: &str) -> Result> { + check_name(name)?; + let dir = root_path()?.join(name); + Ok(dir.join(META_FILE).is_file().then_some(dir)) +} + +/// Reads an installed workflow back and checks it again: a file edited by +/// hand since the install is refused here rather than failing mid-run. +pub fn load(name: &str) -> Result { + let root = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + load_dir(name, root) +} + +fn load_dir(name: &str, root: PathBuf) -> Result { + let meta: Meta = serde_json::from_slice( + &std::fs::read(root.join(META_FILE)) + .with_context(|| format!("Could not read {}", root.join(META_FILE).display()))?, + ) + .with_context(|| format!("{} is not readable", root.join(META_FILE).display()))?; + let files = read_tree(&root, &[META_FILE])?; + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Installed { + root, + workflow, + meta, + }) +} + +/// Every installed workflow, by name, with the ones that no longer load +/// kept as their error. +pub fn list() -> Result)>> { + let root = root_path()?; + let Ok(entries) = std::fs::read_dir(&root) else { + return Ok(vec![]); + }; + let mut out = vec![]; + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if !is_workflow_name(&name) || !entry.path().join(META_FILE).is_file() { + continue; + } + let loaded = load_dir(&name, entry.path()); + out.push((name, loaded)); + } + out.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(out) +} + +/// Removes an installed workflow. The name is checked and the directory +/// must carry [`META_FILE`], so nothing but a workflow this command put +/// there is ever removed. +pub fn uninstall(name: &str) -> Result { + let dir = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + std::fs::remove_dir_all(&dir).with_context(|| format!("Could not remove {}", dir.display()))?; + Ok(dir) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn archive(entries: &[(&str, &[u8])]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + vec![], + flate2::Compression::fast(), + )); + for (path, bytes) in entries { + let mut header = tar::Header::new_gnu(); + header.set_size(bytes.len() as u64); + header.set_mode(0o644); + header.set_entry_type(tar::EntryType::Regular); + builder.append_data(&mut header, path, *bytes).unwrap(); + } + builder.into_inner().unwrap().finish().unwrap() + } + + #[test] + fn only_the_named_workflow_comes_out() { + let bytes = archive(&[ + ("mapbox-cli-abc/README.md", b"x"), + ( + "mapbox-cli-abc/workflow/beta/copy-style/workflow.yaml", + b"a", + ), + ("mapbox-cli-abc/workflow/beta/copy-style/scripts/p.py", b"b"), + ("mapbox-cli-abc/workflow/beta/other/workflow.yaml", b"c"), + ]); + let (stage, files) = extract(&bytes, "copy-style", "mapbox/cli").unwrap(); + assert_eq!(stage, "beta"); + assert_eq!( + files.keys().cloned().collect::>(), + [ + PathBuf::from("scripts/p.py"), + PathBuf::from("workflow.yaml") + ] + ); + } + + #[test] + fn a_missing_workflow_lists_what_is_there() { + let bytes = archive(&[("r-abc/workflow/beta/other/workflow.yaml", b"c")]); + let err = extract(&bytes, "copy-style", "mapbox/cli").unwrap_err(); + assert!(err.to_string().contains("other (beta)"), "{err}"); + } + + #[test] + fn an_unknown_stage_is_not_published() { + let bytes = archive(&[("r-abc/workflow/experimental/copy-style/workflow.yaml", b"c")]); + assert!(extract(&bytes, "copy-style", "mapbox/cli").is_err()); + } + + #[test] + fn repos_and_refs_cannot_carry_url_syntax() { + assert!(check_repo("mapbox/cli").is_ok()); + for bad in [ + "mapbox", + "mapbox/cli/x", + "../x", + "a/b?c", + "a/b#c", + "/cli", + "a/..", + ] { + assert!(check_repo(bad).is_err(), "{bad}"); + } + for good in ["main", "v0.3.0", "feat/copy-style", "063415e"] { + assert!(check_ref(good).is_ok(), "{good}"); + } + for bad in ["", "a?b", "a#b", "../main", "a//b", "a b", "a%2e"] { + assert!(check_ref(bad).is_err(), "{bad}"); + } + } +} diff --git a/src/workflow/template.rs b/src/workflow/template.rs new file mode 100644 index 0000000..51ce29d --- /dev/null +++ b/src/workflow/template.rs @@ -0,0 +1,351 @@ +//! `${{ … }}` expressions: how one step's values reach the next. +//! +//! Deliberately a lookup language and nothing more — no operators, no +//! functions, no conditionals. An expression names a value: an input, or a +//! path into an earlier step's output. Anything that needs logic belongs in a +//! script step, which is a real language and is tested as one. +//! +//! A string that is exactly one expression takes the value's own JSON type, +//! so an object can be handed to `stdin` whole and a number stays a number. +//! An expression inside a longer string is interpolated as text. + +use std::collections::BTreeMap; + +use anyhow::{anyhow, Result}; +use serde_json::{Map, Value}; + +const OPEN: &str = "${{"; +const CLOSE: &str = "}}"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Segment { + Key(String), + Index(usize), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Reference { + Input(String), + Step { id: String, path: Vec }, +} + +/// What an expression can see while a workflow runs. +pub struct Context<'a> { + pub inputs: &'a Map, + /// The outputs of the steps that have finished, by id. + pub steps: &'a BTreeMap, +} + +/// Every reference in `value`, for checking before anything runs. +pub fn references(value: &Value) -> Result, String> { + let mut found = vec![]; + walk_strings(value, &mut |text| { + for piece in pieces(text)? { + if let Piece::Expression(reference) = piece { + found.push(reference); + } + } + Ok(()) + })?; + Ok(found) +} + +/// `value` with every expression replaced by what it names. +pub fn resolve(value: &Value, context: &Context) -> Result { + match value { + Value::String(text) => resolve_string(text, context), + Value::Array(items) => items + .iter() + .map(|item| resolve(item, context)) + .collect::>() + .map(Value::Array), + Value::Object(fields) => fields + .iter() + .map(|(key, item)| Ok((key.clone(), resolve(item, context)?))) + .collect::>() + .map(Value::Object), + other => Ok(other.clone()), + } +} + +/// How a resolved value is spelled where only text fits: an argv entry, an +/// interpolated string. +pub fn as_text(value: &Value) -> Option { + match value { + Value::Null => None, + Value::String(text) => Some(text.clone()), + Value::Bool(_) | Value::Number(_) => Some(value.to_string()), + Value::Array(_) | Value::Object(_) => Some(value.to_string()), + } +} + +fn resolve_string(text: &str, context: &Context) -> Result { + let pieces = pieces(text).map_err(|e| anyhow!(e))?; + if let [Piece::Expression(reference)] = pieces.as_slice() { + return lookup(reference, context); + } + let mut out = String::new(); + for piece in pieces { + match piece { + Piece::Text(literal) => out.push_str(&literal), + Piece::Expression(reference) => { + let value = lookup(&reference, context)?; + // Interpolating a null would write "null" or nothing into the + // middle of a URL or a name, and neither is what was meant. + let text = as_text(&value).ok_or_else(|| { + anyhow!( + "`{}` is empty, so it cannot be written into \"{text}\"", + display(&reference) + ) + })?; + out.push_str(&text); + } + } + } + Ok(Value::String(out)) +} + +fn lookup(reference: &Reference, context: &Context) -> Result { + match reference { + Reference::Input(name) => Ok(context.inputs.get(name).cloned().unwrap_or(Value::Null)), + Reference::Step { id, path } => { + let mut current = context + .steps + .get(id) + .ok_or_else(|| anyhow!("`{}`: step `{id}` has not run", display(reference)))?; + for segment in path { + let next = match segment { + Segment::Key(key) => current.get(key.as_str()), + Segment::Index(index) => current.get(*index), + }; + current = next.ok_or_else(|| { + anyhow!( + "`{}`: step `{id}`'s output has nothing at that path", + display(reference) + ) + })?; + } + Ok(current.clone()) + } + } +} + +/// An expression as the author wrote it, for error messages. +pub fn display(reference: &Reference) -> String { + match reference { + Reference::Input(name) => format!("inputs.{name}"), + Reference::Step { id, path } => { + let mut out = format!("steps.{id}.output"); + for segment in path { + match segment { + Segment::Key(key) => { + out.push('.'); + out.push_str(key); + } + Segment::Index(index) => out.push_str(&format!("[{index}]")), + } + } + out + } + } +} + +fn walk_strings( + value: &Value, + visit: &mut dyn FnMut(&str) -> Result<(), String>, +) -> Result<(), String> { + match value { + Value::String(text) => visit(text), + Value::Array(items) => items.iter().try_for_each(|item| walk_strings(item, visit)), + Value::Object(fields) => fields + .values() + .try_for_each(|item| walk_strings(item, visit)), + _ => Ok(()), + } +} + +#[derive(Debug, PartialEq)] +enum Piece { + Text(String), + Expression(Reference), +} + +fn pieces(text: &str) -> Result, String> { + let mut out = vec![]; + let mut rest = text; + while let Some(start) = rest.find(OPEN) { + if start > 0 { + out.push(Piece::Text(rest[..start].to_string())); + } + let after = &rest[start + OPEN.len()..]; + let end = after + .find(CLOSE) + .ok_or_else(|| format!("unclosed `{OPEN}` in \"{text}\""))?; + out.push(Piece::Expression(parse(after[..end].trim())?)); + rest = &after[end + CLOSE.len()..]; + } + if !rest.is_empty() { + out.push(Piece::Text(rest.to_string())); + } + Ok(out) +} + +/// `inputs.` or `steps..output` followed by `.key` and `[n]`. +fn parse(expression: &str) -> Result { + let invalid = || { + format!( + "`{expression}` is not an expression this understands. Use \ + `inputs.` or `steps..output`, optionally followed by \ + `.key` or `[index]`" + ) + }; + + let mut segments = vec![]; + let mut chars = expression.chars().peekable(); + let mut first = true; + while chars.peek().is_some() { + if !first { + match chars.next() { + Some('.') => {} + Some('[') => { + let digits: String = chars.by_ref().take_while(|c| *c != ']').collect(); + let index = digits.parse::().map_err(|_| invalid())?; + segments.push(Segment::Index(index)); + continue; + } + _ => return Err(invalid()), + } + } + first = false; + let mut key = String::new(); + while let Some(c) = chars.peek() { + if c.is_ascii_alphanumeric() || *c == '_' || *c == '-' { + key.push(*c); + chars.next(); + } else { + break; + } + } + if key.is_empty() { + return Err(invalid()); + } + segments.push(Segment::Key(key)); + } + + let mut segments = segments.into_iter(); + match (segments.next(), segments.next(), segments.next()) { + (Some(Segment::Key(root)), Some(Segment::Key(name)), None) if root == "inputs" => { + Ok(Reference::Input(name)) + } + (Some(Segment::Key(root)), Some(Segment::Key(id)), Some(Segment::Key(output))) + if root == "steps" && output == "output" => + { + Ok(Reference::Step { + id, + path: segments.collect(), + }) + } + _ => Err(invalid()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn context_resolve(value: Value) -> Result { + let inputs = json!({ "style_id": "abc", "zoom": 12, "empty": null }); + let steps = BTreeMap::from([( + "source".to_string(), + json!({ "owner": "alice", "layers": [{ "id": "water" }] }), + )]); + resolve( + &value, + &Context { + inputs: inputs.as_object().unwrap(), + steps: &steps, + }, + ) + } + + #[test] + fn a_whole_expression_keeps_its_type() { + assert_eq!( + context_resolve(json!("${{ inputs.zoom }}")).unwrap(), + json!(12) + ); + assert_eq!( + context_resolve(json!("${{ steps.source.output.layers[0] }}")).unwrap(), + json!({ "id": "water" }) + ); + } + + #[test] + fn an_embedded_expression_is_interpolated_as_text() { + assert_eq!( + context_resolve(json!( + "styles/${{ steps.source.output.owner }}/${{inputs.style_id}}" + )) + .unwrap(), + json!("styles/alice/abc") + ); + } + + #[test] + fn objects_and_arrays_are_resolved_inside() { + assert_eq!( + context_resolve(json!({ "a": ["${{ inputs.style_id }}", 1] })).unwrap(), + json!({ "a": ["abc", 1] }) + ); + } + + #[test] + fn an_empty_value_cannot_be_interpolated() { + assert!(context_resolve(json!("x-${{ inputs.empty }}")).is_err()); + assert_eq!( + context_resolve(json!("${{ inputs.empty }}")).unwrap(), + Value::Null + ); + } + + #[test] + fn a_missing_path_names_the_expression() { + let err = context_resolve(json!("${{ steps.source.output.nope }}")).unwrap_err(); + assert!( + err.to_string().contains("steps.source.output.nope"), + "{err}" + ); + } + + #[test] + fn references_are_found_for_checking() { + assert_eq!( + references(&json!({ "a": "${{ inputs.x }}", "b": ["${{ steps.s.output.k[2] }}"] })) + .unwrap(), + vec![ + Reference::Input("x".into()), + Reference::Step { + id: "s".into(), + path: vec![Segment::Key("k".into()), Segment::Index(2)] + } + ] + ); + } + + #[test] + fn anything_else_is_refused() { + for bad in [ + "${{ inputs }}", + "${{ inputs.a.b }}", + "${{ steps.s }}", + "${{ steps.s.result }}", + "${{ env.HOME }}", + "${{ inputs.a + 1 }}", + "${{ steps.s.output[x] }}", + "${{ inputs.a", + ] { + assert!(references(&json!(bad)).is_err(), "{bad} was accepted"); + } + } +} diff --git a/tests/source_guards.rs b/tests/source_guards.rs index eb6d4e6..d52fda0 100644 --- a/tests/source_guards.rs +++ b/tests/source_guards.rs @@ -48,6 +48,10 @@ fn sources() -> Vec<(String, String)> { /// - `generate_skills` — the staged skill directory it renames into place. /// - `skill_dest` — a test scratch directory. /// - `uninstall` — the binary itself, which is the whole command. +/// - `workflow/store.rs` — its staging directory, the installed workflow +/// `install --force` replaces, and `workflow uninstall`. The name is +/// checked to be one plain directory name first, and only a directory +/// holding its `.install.json` marker is removed. const MAY_DELETE: &[&str] = &[ "agent_skills.rs", "auth.rs", @@ -56,6 +60,7 @@ const MAY_DELETE: &[&str] = &[ "generate_skills.rs", "skill_dest.rs", "uninstall.rs", + "workflow/store.rs", ]; /// A new module that deletes files has to say so here first. @@ -238,7 +243,10 @@ const CARRIES_A_REQUEST_ID: &[&str] = &["account_usage.rs", "auth.rs", "executor /// `agent_skills.rs` talks to GitHub codeload, which identifies requests with /// `x-github-request-id`. That is not something Mapbox support can look up, /// so an id there would point at the wrong company — worse than none. -const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs"]; +/// +/// `workflow/store.rs` fetches a tarball from the GitHub API, for the same +/// reason. +const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs", "workflow/store.rs"]; /// `output/error.rs` defines `CliError::http` rather than calling it over a /// wire. diff --git a/tests/workflow.rs b/tests/workflow.rs new file mode 100644 index 0000000..c8eb052 --- /dev/null +++ b/tests/workflow.rs @@ -0,0 +1,295 @@ +//! End-to-end tests for `mapbox workflow`. +//! +//! The unit tests in `src/workflow/` cover the schema, the expressions and +//! how a step's arguments become a command line. What they cannot show is +//! what a real run does: that a command step is a real child `mapbox` whose +//! JSON reaches the next step, that a script's stdin and stdout carry values +//! between steps, that only the workflow's result lands on stdout, and what +//! `install` and `uninstall` leave on disk. +//! +//! Nothing here reaches the network. The command steps run `config list` and +//! `auth whoami`, which make no request, and every install is from a local +//! directory. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use serde_json::{json, Value}; + +/// A token-shaped fake for account `example-user`. +const TOKEN: &str = "pk.eyJ1IjoiZXhhbXBsZS11c2VyIiwiYSI6IngifQ.SIGNATURE"; + +fn scratch(name: &str) -> PathBuf { + let home = PathBuf::from(env!("CARGO_TARGET_TMPDIR")).join(format!("workflow-{name}")); + let _ = std::fs::remove_dir_all(&home); + std::fs::create_dir_all(&home).expect("create the scratch home"); + home +} + +fn run(home: &Path, args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_mapbox")) + .env_remove("MAPBOX_ACCESS_TOKEN") + .env_remove("MapboxAccessToken") + .env_remove("MAPBOX_USERNAME") + .env_remove("MAPBOX_OUTPUT") + .env_remove("GH_TOKEN") + .env_remove("GITHUB_TOKEN") + .env("MAPBOX_NO_UPDATE_CHECK", "1") + .env("MAPBOX_QUIET", "1") + .env("HOME", home) + .env("XDG_CONFIG_HOME", home.join(".config")) + .env("MAPBOX_CONFIG_DIR", home.join(".mapbox")) + .args(args) + .output() + .expect("run mapbox") +} + +fn stdout_json(out: &Output) -> Value { + serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "stdout is not one JSON document ({e}):\n{}\nstderr:\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }) +} + +/// Writes a workflow at `/src/beta//` and returns its path. +fn write_workflow(home: &Path, name: &str, yaml: &str, scripts: &[(&str, &str)]) -> PathBuf { + let dir = home.join("src").join("beta").join(name); + std::fs::create_dir_all(dir.join("scripts")).unwrap(); + std::fs::write(dir.join("workflow.yaml"), yaml).unwrap(); + for (file, body) in scripts { + std::fs::write(dir.join("scripts").join(file), body).unwrap(); + } + dir +} + +const DEMO: &str = r#"version: 1 +name: demo +summary: Pass values from a command to a script and back +inputs: + greeting: { type: string, default: hello } +steps: + - id: settings + command: config list + - id: whoami + command: auth whoami + - id: shout + script: shout.sh + args: ["${{ inputs.greeting }}"] + stdin: ${{ steps.settings.output[0] }} +outputs: + account: ${{ steps.whoami.output.account }} + first_key: ${{ steps.settings.output[0].key }} + shouted: ${{ steps.shout.output.shouted }} + seen: ${{ steps.shout.output.seen }} +"#; + +/// Echoes its argument upper-cased beside what it read on stdin, and says +/// something on stderr, which must not reach the result. +const SHOUT: &str = r#"read -r line +echo "shouting" >&2 +printf '{"shouted":"%s","seen":%s}\n' "$(printf %s "$1" | tr a-z A-Z)" "$line" +"#; + +fn install_demo(home: &Path) { + let dir = write_workflow(home, "demo", DEMO, &[("shout.sh", SHOUT)]); + let out = run( + home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); +} + +#[test] +fn a_run_carries_values_between_steps_and_prints_only_the_result() { + let home = scratch("run"); + install_demo(&home); + + let out = run( + &home, + &[ + "--token", + TOKEN, + "workflow", + "run", + "demo", + "-i", + "greeting=hi", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + + let first_key = json!("update-check"); + assert_eq!( + stdout_json(&out), + json!({ + "account": "example-user", + "first_key": first_key, + "shouted": "HI", + "seen": { "key": first_key, "value": true }, + }) + ); + + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("[1/3] settings (mapbox config list)"), + "{stderr}" + ); + assert!( + stderr.contains("[3/3] shout (sh scripts/shout.sh)"), + "{stderr}" + ); + assert!(stderr.contains("shouting"), "{stderr}"); + assert!(stderr.contains("beta workflow"), "{stderr}"); +} + +#[test] +fn a_failing_step_stops_the_run() { + let home = scratch("fail"); + let yaml = "version: 1\nname: broken\nsummary: Fails in the middle\nsteps:\n\ + \x20 - id: first\n script: fail.sh\n\ + \x20 - id: second\n command: config list\n"; + let dir = write_workflow(&home, "broken", yaml, &[("fail.sh", "exit 3\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "broken", "-o", "json"]); + assert_eq!(out.status.code(), Some(1)); + assert!( + out.stdout.is_empty(), + "{}", + String::from_utf8_lossy(&out.stdout) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("exited with 3"), "{stderr}"); + assert!(!stderr.contains("[2/2]"), "the second step ran: {stderr}"); +} + +#[test] +fn dry_run_runs_nothing() { + let home = scratch("dry-run"); + let marker = home.join("ran"); + let yaml = "version: 1\nname: touch\nsummary: Leaves a file behind\n\ + inputs:\n path: { type: string, required: true }\nsteps:\n\ + \x20 - id: touch\n script: touch.sh\n args: ['${{ inputs.path }}']\n"; + let dir = write_workflow(&home, "touch", yaml, &[("touch.sh", "touch \"$1\"\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let path = format!("path={}", marker.display()); + let out = run( + &home, + &[ + "workflow", + "run", + "touch", + "-i", + &path, + "--dry-run", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out)["steps"][0]["run"], "sh scripts/touch.sh"); + assert!(!marker.exists(), "--dry-run ran the step"); + + let out = run(&home, &["workflow", "run", "touch", "-i", &path]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(marker.exists(), "the real run did not run the step"); +} + +#[test] +fn install_refuses_an_invalid_workflow_and_writes_nothing() { + let home = scratch("invalid"); + let yaml = "version: 1\nname: bad\nsummary: Names a command that is not one\nsteps:\n\ + \x20 - id: a\n command: styles nope\n"; + let dir = write_workflow(&home, "bad", yaml, &[("unused.sh", "")]); + + let out = run( + &home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert_eq!(out.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("invalid_workflow"), "{stderr}"); + assert!(stderr.contains("scripts/unused.sh"), "{stderr}"); + assert!(!home.join(".mapbox/workflows/bad").exists()); +} + +#[test] +fn install_replaces_only_with_force_and_uninstall_removes() { + let home = scratch("lifecycle"); + install_demo(&home); + let installed = home.join(".mapbox/workflows/demo"); + assert!(installed.join("workflow.yaml").is_file()); + assert!(installed.join(".install.json").is_file()); + + let source = home.join("src/beta/demo"); + let again = run(&home, &["workflow", "install", source.to_str().unwrap()]); + assert_eq!(again.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&again.stderr).contains("already installed")); + + let forced = run( + &home, + &["workflow", "install", source.to_str().unwrap(), "--force"], + ); + assert!( + forced.status.success(), + "{}", + String::from_utf8_lossy(&forced.stderr) + ); + + let listed = stdout_json(&run(&home, &["workflow", "list", "-o", "json"])); + assert_eq!(listed[0]["name"], "demo"); + assert_eq!(listed[0]["stage"], "beta"); + + let removed = run(&home, &["workflow", "uninstall", "demo", "-o", "json"]); + assert!( + removed.status.success(), + "{}", + String::from_utf8_lossy(&removed.stderr) + ); + assert!(!installed.exists()); + + let missing = run(&home, &["workflow", "run", "demo"]); + assert_eq!(missing.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&missing.stderr).contains("mapbox workflow install demo")); +} + +#[test] +fn uninstall_takes_only_a_workflow_name() { + let home = scratch("escape"); + std::fs::create_dir_all(home.join("keep")).unwrap(); + for name in ["../keep", "/tmp", ".."] { + let out = run(&home, &["workflow", "uninstall", name]); + assert_eq!(out.status.code(), Some(1), "{name}"); + assert!( + String::from_utf8_lossy(&out.stderr).contains("not a workflow name"), + "{name}" + ); + } + assert!(home.join("keep").exists()); +} diff --git a/workflow/README.md b/workflow/README.md new file mode 100644 index 0000000..913a767 --- /dev/null +++ b/workflow/README.md @@ -0,0 +1,94 @@ +# Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts. None ships inside the binary: `mapbox workflow install` copies one in, and `mapbox workflow run` runs only what is installed. See [docs/commands.md](../docs/commands.md#workflows) for the commands. + +Workflows and the `workflow` command are beta. The format below is `version: 1`, and may change before it is stable. + +## Layout + +``` +workflow/ + beta/ # the stage; the only one so far + copy-style/ # the workflow's name + workflow.yaml # required + scripts/ # the scripts its steps run + prepare.py + README.md # optional +``` + +- The directory name is the workflow's name: lower-case letters, digits and dashes, and the same as `name` in `workflow.yaml`. +- The stage is the parent directory. A `beta` workflow says so on stderr when it is installed or run. +- A workflow holds `workflow.yaml`, `README.md` and `scripts/`, and nothing else. +- Every file in `scripts/` is run by some step. A workflow ships only the scripts it runs. + +The same rules apply to any workflow `install` reads, from this repository, another one (`--repo OWNER/REPO`, laid out the same way) or a local directory. `every_published_workflow_is_valid` in `src/workflow/mod.rs` holds this directory to them in `cargo test`. + +## `workflow.yaml` + +```yaml +version: 1 # required; the schema version +name: copy-style # required; the directory's name +summary: Copy a style from one account to another # required; one line +description: | # optional; shown by `workflow show` + Longer text. + +inputs: # optional; given as --input KEY=VALUE + style_id: + type: string # string | number | boolean + required: true + description: ID of the style to copy + name: + type: string # not required and no default: null + zoom: + type: number + default: 12 + +steps: # required; run in order, first failure stops + - id: source # required; lower-case, digits, underscores + name: Read the style # optional; the progress line + command: styles get # a mapbox command + args: # names as `mapbox --schema` lists them + style-id: ${{ inputs.style_id }} + profile: work # global options too + use-login: true # a flag takes true or false + + - id: body + script: prepare.py # a file in scripts/ + interpreter: python3 # optional for .sh, .py and .js + args: ["--zoom", "${{ inputs.zoom }}"] + stdin: # optional; sent to the step as JSON + style: ${{ steps.source.output }} + +outputs: # optional; the default is the last step's output + id: ${{ steps.source.output.id }} +``` + +A step is a `command` or a `script`, never both. + +### Expressions + +`${{ inputs. }}` and `${{ steps..output }}`, followed by any number of `.key` and `[index]`. Nothing else: no operators and no functions. Logic belongs in a script. + +- A value that is exactly one expression keeps its JSON type, so a whole object can go to `stdin` and a number stays a number. +- An expression inside a longer string is written in as text. One that is null there is an error rather than an empty string. +- An expression may name only a declared input or an earlier step. That is checked before any step runs. + +### Command steps + +Each command step runs this `mapbox` binary again, with `--output json`, and its JSON result becomes the step's output. It resolves its token, timeouts and path encoding exactly as the same command typed by hand would, and appears in `mapbox history` on its own. + +- `args` keys are the argument names `mapbox --schema ` lists: a flag's long name, or a positional's name. Global options such as `profile`, `username` and `use-login` are accepted too. +- `output`, `quiet`, `schema` and `dry-run` belong to the runner. `token` is refused, because a token written into a workflow is a secret in a file: log in under a profile and name the `profile` instead. +- Global options given to `mapbox workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, `--yes`, `--debug`, `--token`) reach every command step that does not set its own. +- `stdin` is sent to the command, for `--data @-`. Without it, the command reads the terminal, which is where a confirmation prompt gets its answer. +- A step cannot run `mapbox workflow`. + +### Script steps + +A script runs from the installed copy of `scripts/`, in the directory `mapbox workflow run` was started from. + +- It gets `args` as its arguments and `stdin` on standard input, as JSON unless the value is a string. With no `stdin` it reads nothing. +- Whatever it writes to stdout is its output: JSON when it parses as JSON, the text otherwise. It writes progress to stderr. +- A non-zero exit stops the workflow. +- `MAPBOX_CLI` is the path to this `mapbox` binary, for a script that runs commands of its own. `MAPBOX_WORKFLOW_ROOT` is the installed workflow's directory. +- The interpreter must be installed on the machine. `.sh` runs under `sh`, `.py` under `python3` and `.js` under `node`, and any other extension needs `interpreter`. diff --git a/workflow/beta/copy-style/README.md b/workflow/beta/copy-style/README.md new file mode 100644 index 0000000..f3f22da --- /dev/null +++ b/workflow/beta/copy-style/README.md @@ -0,0 +1,21 @@ +# copy-style + +Copies a style from one Mapbox account to another. + +```sh +mapbox auth login --profile source +mapbox auth login --profile target +mapbox workflow install copy-style +mapbox workflow run copy-style \ + --input style_id=