diff --git a/CHANGELOG.md b/CHANGELOG.md index 40c6a8c..d48f1eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,34 @@ that may never merge. They are not releases and are not listed here. ## Unreleased +- `mapbox workflow`, beta and in development, not recommended for use: + install and run workflows, which are named, multi-step recipes of + `mapbox` commands and scripts defined in a `workflow.yaml`. `install` copies one from a local directory or from a + GitHub repository's `workflow//` (by default `mapbox/mapbox-cli`; + `GH_TOKEN`/`GITHUB_TOKEN` for a private one) into `~/.mapbox/workflows/`, + and `list`, `show`, `run` and `uninstall` work on what is installed. + `run` takes each of the workflow's inputs as a flag, as in + `mapbox workflow run copy-style --style-id --from-profile source + --to-profile target`. `run --dry-run` runs only the steps a workflow + marks `dry_run`, which write nothing, so the plan can show real data. At a + terminal each step shows a spinner, the details its script reports on + stderr (a `::progress ` line updates the spinner's text, a `::warn ` line + is a warning, listed again at the end), and `✓` or `✗` with the time it + took; off a terminal, each step is a plain `[n/total]` line followed by + its details. `--quiet` drops the details but keeps warnings. A workflow's + optional `result` template is what text mode prints; `-o json` prints its + outputs. A command step's `save: ` keeps its stdout, such as a + style's ZIP, as a file for later steps, in a working directory removed + when the run ends; a command step may be marked `dry_run` when its command + changes nothing. The + generated agent skill leaves `workflow` out. The first one published is + `copy-style`, which copies a style between accounts with its custom fonts + and icons, and on a partial failure lists what it created and the + commands that remove it. The command, the `version: 1` format and the published + workflows may change or be removed without notice, and every `workflow` + subcommand says so on stderr. Nothing changes for a script that does not + use them. + - New command: `mapbox styles download > style.zip` saves a style as a ZIP with its sprite icons and custom fonts. `mapbox auth login` now also asks for the `styles:download` scope it needs, so log in again diff --git a/README.md b/README.md index d8c00a3..ce07cbe 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,7 @@ mapbox styles list - [Diagnostics and settings](#diagnostics-and-settings) - [Shell completion](#shell-completion) - [Tileset CLI](#tileset-cli) + - [Workflows (beta)](#workflows-beta) - [For AI agents](#for-ai-agents) - [Agent skills](#agent-skills) - [Generate skills](#generate-skills) @@ -289,6 +290,17 @@ apply here; `tilesets` has its own flags, so use `--force`/`-f` for its prompts. If a tileset command answers for the wrong account, `mapbox auth whoami` shows which token is in play. +### Workflows (beta) + +```sh +mapbox workflow install copy-style +mapbox workflow run copy-style --style-id --from-profile source --to-profile target +``` + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts. +`mapbox workflow` is in development and not recommended for use yet. See +[Workflows](./docs/commands.md#workflows). + ## For AI agents Two commands, for two different jobs: `agent-skills` installs guidance on diff --git a/docs/commands.md b/docs/commands.md index c0fd1c1..c2ef019 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -111,6 +111,12 @@ nests, and is typed `mapbox styles draft get`. [tilesets.get-mvt](#mapbox-tilesets-get-mvt) · [tilesets.query](#mapbox-tilesets-query) +**[Workflows](#workflows)** — [workflow.list](#mapbox-workflow-list) · +[workflow.show](#mapbox-workflow-show) · +[workflow.install](#mapbox-workflow-install) · +[workflow.uninstall](#mapbox-workflow-uninstall) · +[workflow.run](#mapbox-workflow-run) + **[Tilesets CLI](#tilesets-cli)** — [tilesets-cli](#mapbox-tilesets-cli-args) Then [Errors](#errors) — the shape a failure takes in each mode. @@ -527,7 +533,7 @@ either. | `--profile ` | Which stored credentials to use. | | `--output`, `-o` | `auto` \| `text` \| `json`. | | `--id ` | On a command that returns a list, print just the row with that `id` or `name`. | -| `--quiet`, `-q` | Don't print the `mapbox · v` banner, which goes to stderr and only when stderr is a terminal. Also `MAPBOX_QUIET`. | +| `--quiet`, `-q` | Don't print the `mapbox · v` banner, which goes to stderr and only when stderr is a terminal, the note after a download, or a workflow step's details. Also `MAPBOX_QUIET`. | | `--timeout ` | How long one request may take, connection included. Defaults to 60 seconds, or 900 for a body read from `--file` or from a `--data @`/`@-`. Also `MAPBOX_TIMEOUT`. | An operation with a request body takes `--data`/`-d` when that body is text @@ -3852,6 +3858,290 @@ list drops the `--daily` suggestion once it's already in effect. --- +## Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts, +defined in a `workflow.yaml`. The format and the rules a workflow directory +follows are in [workflow/README.md](../workflow/README.md). + +**Beta and in development. Not recommended for use.** The commands, the +`version: 1` format and the published workflows may change or be removed +without notice. Every `mapbox workflow` subcommand opens with this line on +stderr: + +``` +Beta: `mapbox workflow` is in development and not recommended for use. +``` + +In text mode, `list`, `show` and `install` end with tips on stderr, naming +the command to run next. `mapbox generate-skills` leaves the `workflow` +commands out of the skill it writes, so that an agent is not taught a command +nobody should rely on yet. + +None ships inside the binary. `install` copies one into +`~/.mapbox/workflows//`, and every other subcommand works on what is +installed there. + +--- + +### `mapbox workflow list` + +Lists the installed workflows. One that no longer loads, because a file was +edited by hand or this CLI no longer reads its format, is listed with its +error instead of a summary. + +#### Outputs + + + + +
textjson
+ +``` +NAME SUMMARY +copy-style Copy a style from one account to another +``` + + + +```json +[ + { + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "github:mapbox/mapbox-cli@main", + "summary": "Copy a style from one account to another" + } +] +``` + +
+ +--- + +### `mapbox workflow show` + +Describes an installed workflow, in the same layout for every one: its name +and summary, its description, its inputs with their types and defaults, its +steps, and where the installed copy came from. When a step names a command or +an argument this build no longer has, the problems are listed after the steps +and under `problems`. How a description is written is in +[workflow/README.md](../workflow/README.md#description). + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | + +--- + +### `mapbox workflow install` + +Installs a workflow from GitHub or from a local directory. Everything is read +and checked before anything is written: the layout, the `workflow.yaml`, and +every command step against this build's commands. A workflow that fails any +of these is `invalid_workflow`, with every problem listed at once. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `SOURCE` | A workflow name, looked up in the repository's `workflow//`, or a path to a local workflow directory: anything with a `/` or starting with `.`. | +| `--repo` | GitHub repository to install from, as `OWNER/REPO`. Defaults to `mapbox/mapbox-cli`. | +| `--ref` | Branch, tag or commit to install from. Defaults to `main`. | +| `--force` | Replace a workflow that is already installed. | +| `--dry-run` | Check the workflow and list the files it would write, then exit. | + +The repository is read as one tarball through the GitHub API. +`mapbox/mapbox-cli` is public and needs no token. For a private repository +named with `--repo`, set `GH_TOKEN` or `GITHUB_TOKEN`. It is sent only to +`api.github.com`. Without one, a private repository answers 404, +exactly as a ref that does not exist does, and the error says both. + +**An installed workflow stops the install** unless `--force` is given. The new +copy is staged and renamed into place, so an interrupted install leaves the +old copy or the new one. Only regular files are taken. A symlink in a local +directory is refused, and an archive entry whose path would leave the +workflow's directory stops the read. + +#### Examples + +```sh +mapbox workflow install copy-style + +mapbox workflow install copy-style --ref v0.4.0 + +export GITHUB_TOKEN="$(gh auth token)" +mapbox workflow install sync-tilesets --repo my-org/private-workflows + +mapbox workflow install ./workflow/copy-style --force +``` + +#### Outputs + + + + +
textjson
+ +``` +Installed copy-style + +Source ~/dev/cli/workflow/copy-style +Path ~/.mapbox/workflows/copy-style +Files README.md, scripts/copy_style.py, workflow.yaml +``` + + + +```json +{ + "dry_run": false, + "files": [ + "README.md", + "scripts/copy_style.py", + "workflow.yaml" + ], + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "/Users/me/dev/cli/workflow/copy-style" +} +``` + +
+ +--- + +### `mapbox workflow uninstall` + +Removes an installed workflow's directory. At a terminal it asks first, and +`--yes` skips the question. Only a directory `install` wrote under +`~/.mapbox/workflows/` is removed. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow, or the directory it was installed from, which names it by its directory name. The directory itself is not touched. | +| `--dry-run` | Say what it would remove, then exit without removing it. | + +--- + +### `mapbox workflow run` + +Runs an installed workflow's steps in order. The first step that fails stops +the run with `workflow_failed`, naming the step, and exit code 1. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | +| `--` | One flag per input the workflow declares, spelled with dashes (`style_id` is `--style-id`). Required, typed and defaulted as its `workflow.yaml` says. `mapbox workflow run --help` lists them. | +| `--dry-run` | Check the workflow and its inputs and print the plan. Runs only the steps marked `dry_run`, which write nothing; skips the rest. | + +The flags come from the installed workflow's definition, read before the +command line is parsed, so a missing input, a misspelled flag or a value of the +wrong type is a usage error (exit 2), as on any other command. They are not in +`--schema` or in the shell completion, which describe the command tree without +reading `~/.mapbox/workflows/`. `workflow show` lists them for any installed +workflow. + +**stdout holds only the result**: the workflow's `outputs`, or the last +step's output if it declares none, rendered like any other result. Each +step's progress and anything a step writes to stderr go to stderr. At a +terminal, each step shows its title, the details its script reports, and a +spinner that becomes `✓` or `✗` with the time it took; anywhere else, each +step is one `[n/total]` line followed by its details, with no escape codes. +`--quiet` keeps each step's title, how it ended and its warnings, and drops +its details. `workflow/README.md` describes how a script reports progress, +details and warnings. + +A workflow that declares `result` prints it in text mode instead of the +outputs as fields; `-o json` always prints the outputs. + +Each command step is this binary run again with `--output json`, so it +resolves its token and applies its timeouts as the same command typed by +hand would, and appears in `mapbox history` as its own run. The globals given +to `workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, +`--yes`, `--debug`, `--token`) reach every command step that does not set its +own. A token typed as `--token` goes to the step's environment, not its +command line. + +#### Examples + +```sh +mapbox workflow run copy-style \ + --style-id cmm28c5rm00bj01qz9hwp69qc \ + --from-profile source \ + --to-profile target + +mapbox workflow run copy-style --style-id cmm28c5rm00bj01qz9hwp69qc \ + --from-profile source --to-profile target --dry-run +``` + +`--dry-run` prints the plan: the inputs as they were read, and each step with +its arguments as written. A step marked `dry_run` in its `workflow.yaml` runs +for real, so the plan can show what the rest would do with real data: a +command step only if its command changes nothing, and a script step with +`MAPBOX_WORKFLOW_DRY_RUN=1` in its environment and the promise to write +nothing. Their outputs are under `results` in `-o json`. Every other step is +skipped. `copy-style` marks all of its steps, so its dry run downloads the +style and lists what it would upload: + +``` +$ mapbox workflow run copy-style --style-id cmums8rlh000301s96498hnju \ + --from-profile default --to-profile default --dry-run +🗺️ mapbox · v0.3.0 +──────────────────────────────────────── +Beta: `mapbox workflow` is in development and not recommended for use. + +▸ Check the source login mapbox auth status + ✓ 0.0s + +▸ Check the target login mapbox auth status + ✓ 0.0s + +▸ Download the style mapbox styles download + Saved style.zip (965 KB) + ✓ 0.3s + +▸ List the target account's fonts mapbox fonts list + ✓ 0.1s + +▸ Plan the copy + Found 561 icons and 1 custom font + Font Yellow Banana Regular is already in zhuwenlong; it will be skipped + ✓ 0.2s + +▸ Copy the fonts, the style and its icons + Would copy the style into zhuwenlong as "CLI copy test: Helsinki Evening · CLI Blog Demo": + skip font Yellow Banana Regular (already in zhuwenlong) + create the style + upload 561 icons in 23 batches + point its sprite and glyphs at zhuwenlong + ✓ 0.1s + +Dry run — only the steps that support it ran, and they wrote nothing. Would run copy-style: + +Inputs + from_profile default + name (none) + style_id cmums8rlh000301s96498hnju + to_profile default + +Steps + 1. Check the source login mapbox auth status (ran in this dry run) + 2. Check the target login mapbox auth status (ran in this dry run) + 3. Download the style mapbox styles download (ran in this dry run) + 4. List the target account's fonts mapbox fonts list (ran in this dry run) + 5. Plan the copy python3 scripts/copy_style.py (ran in this dry run) + 6. Copy the fonts, the style and its icons python3 scripts/copy_style.py (ran in this dry run) +``` + +--- + ## Tilesets CLI ### `mapbox tilesets-cli ` diff --git a/src/generate_skills.rs b/src/generate_skills.rs index fe3e0e2..6783c09 100644 --- a/src/generate_skills.rs +++ b/src/generate_skills.rs @@ -135,8 +135,21 @@ pub fn command() -> Command { .arg(executor::dry_run_arg(DRY_RUN_HELP)) } +/// `--schema`'s walk, without the commands a skill must not teach. +/// +/// `workflow` is beta, in development and not recommended for use, and an +/// agent that reads about a command in its skill takes it as one to reach +/// for. `--schema` still lists it, with the notice in its description. +fn skill_schema(app: &Command, specs: &[ServiceSpec]) -> schema::Schema { + let mut schema = schema::build(app, specs, &[]); + schema + .commands + .retain(|entry| entry.service != crate::workflow::COMMAND); + schema +} + pub fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) -> Result<()> { - let schema = schema::build(app, specs, &[]); + let schema = skill_schema(app, specs); let header = Header::new(requested_services(&schema, matches)?); let files = render(app, &schema, &header); @@ -1475,7 +1488,7 @@ mod tests { fn generated() -> (Command, Vec, Vec) { let specs = specs(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let header = Header::new(None); let files = render(&app, &schema, &header); (app, specs, files) @@ -1574,10 +1587,18 @@ mod tests { // The same walk `schema::every_command_in_the_tree_is_described` // does, for the same reason: the command tree is the authority on // what a command is. - let mut runnable = crate::runnable_commands(&app); + // Less `workflow`, which `skill_schema` leaves out on purpose. + let mut runnable: Vec = crate::runnable_commands(&app) + .into_iter() + .filter(|command| !command.starts_with("mapbox workflow ")) + .collect(); runnable.sort(); assert_eq!(described, runnable); + assert!( + !described.iter().any(|command| command.contains("workflow")), + "the skill teaches `workflow`, which is not recommended for use" + ); let mut unique = described.clone(); unique.dedup(); @@ -1731,7 +1752,7 @@ mod tests { // And through the real renderer, where a wrapper that swallowed part // of the body would still pass the two assertions above. let (app, specs, files) = generated(); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let real_body = render_body(&app, &schema, &by_service(&schema, &header), &header); for name in ["SKILL.md", "AGENTS.md"] { let file = files @@ -1844,7 +1865,7 @@ mod tests { fn a_service_filter_narrows_the_skill_and_is_recorded() { let specs = specs(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let header = Header::new(Some(vec!["styles".to_string()])); let files = render(&app, &schema, &header); @@ -1881,7 +1902,7 @@ mod tests { fn an_unknown_service_is_an_error_that_lists_the_real_ones() { let specs = specs(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let matches = app .clone() @@ -2023,7 +2044,7 @@ mod tests { fn positionals_keep_their_order_and_flags_are_sorted() { let (_, specs, _) = generated(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let mut checked = 0usize; for entry in &schema.commands { @@ -2062,7 +2083,7 @@ mod tests { fn a_table_names_a_value_rather_than_suggesting_one() { let (_, specs, files) = generated(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let output = schema .global_options @@ -2110,7 +2131,7 @@ mod tests { fn a_description_is_cut_only_when_it_would_not_fit() { let (_, specs, _) = generated(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let output = schema .global_options diff --git a/src/main.rs b/src/main.rs index de67a96..456f79e 100644 --- a/src/main.rs +++ b/src/main.rs @@ -39,6 +39,7 @@ mod telemetry; mod tilesets_cli; mod uninstall; mod update_check; +mod workflow; use output::{CliError, Mode}; use remedy::Remedy; @@ -465,7 +466,10 @@ fn build_app(specs: &[ServiceSpec]) -> Command { // error on every command. .value_parser(FalseyValueParser::new()) .global(true) - .help("Don't print the name-and-version banner, or the note after a download, to stderr"), + .help( + "Don't print the name-and-version banner, the note after a download, or a \ + workflow step's details to stderr", + ), ) .arg( Arg::new(http::TIMEOUT_ARG) @@ -637,6 +641,7 @@ fn build_app(specs: &[ServiceSpec]) -> Command { app = app.subcommand(account_usage::command()); + app = app.subcommand(workflow::command()); app.subcommand(tilesets_cli::command()) } @@ -744,7 +749,9 @@ fn cli() -> u8 { } }; - let app = build_app(&specs); + // Before the parse, since a workflow's inputs are flags only once its + // definition has been read. Everything else sees the tree unchanged. + let app = workflow::with_run_target(build_app(&specs), &raw_argv); let argv = tilesets_cli::escape_passthrough_args(&app, raw_argv.clone()); // Parsing consumes the tree, and `--schema` still has to read it // afterwards — so the parse gets the copy and `app` stays whole. @@ -1135,6 +1142,19 @@ fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) - Some(("show", show_matches)) => history::show(show_matches, mode)?, _ => unreachable!("`history` sets subcommand_required(true)"), }, + // Ahead of the generic service arm too. Its own requests go to + // GitHub, and each command step is a child `mapbox` that resolves + // its credentials itself — so nothing is loaded here. + Some((workflow::COMMAND, workflow_matches)) => workflow::run( + app, + workflow_matches, + workflow::RunFlags { + globals: matches, + debug, + assume_yes, + }, + mode, + )?, // Also ahead of the generic service arm: read-only except for the // opt-in `--verify` request, and needs no credential load of its own // — it reports what one would resolve to, not what a fresh one diff --git a/src/output/mod.rs b/src/output/mod.rs index ef57d66..55d49dc 100644 --- a/src/output/mod.rs +++ b/src/output/mod.rs @@ -251,6 +251,38 @@ pub fn emit_value( footer: Option<&str>, service: Option<&str>, page: Option<&str>, +) -> Result<()> { + emit_rendered( + mode, + value, + footer, + service, + page, + "`-o json` for the response as the API sent it.", + ) +} + +/// [`emit_value`] for a result this CLI put together rather than an API +/// response, such as a workflow's outputs, so the tip does not claim the +/// JSON is what an API sent. +pub fn emit_result(mode: Mode, value: &Value) -> Result<()> { + emit_rendered( + mode, + value, + None, + None, + None, + "`-o json` prints it as JSON.", + ) +} + +fn emit_rendered( + mode: Mode, + value: &Value, + footer: Option<&str>, + service: Option<&str>, + page: Option<&str>, + json_tip: &str, ) -> Result<()> { if let Mode::Json { pretty } = mode { write_stdout(&encode(value, pretty)?)?; @@ -286,7 +318,7 @@ pub fn emit_value( let mut tips = vec![if rendered.shortened { "Values are shortened to fit; `-o json` prints each row whole.".to_string() } else { - "`-o json` for the response as the API sent it.".to_string() + json_tip.to_string() }]; tips.extend(next); // Last, because it is about the response as a whole rather than @@ -323,7 +355,7 @@ pub fn progress(message: &str) { /// caller needs no guard. A blank line first — these are notes about /// whatever was just printed, not more of it, and butted against the last /// line they'd read as one. -fn print_tips(tips: &[String]) { +pub(crate) fn print_tips(tips: &[String]) { if tips.is_empty() { return; } diff --git a/src/output/style.rs b/src/output/style.rs index f540fbf..08764a4 100644 --- a/src/output/style.rs +++ b/src/output/style.rs @@ -19,6 +19,9 @@ pub const RESET: &str = "\x1b[0m"; pub const BOLD: &str = "\x1b[1m"; pub const DIM: &str = "\x1b[2m"; pub const ACCENT: &str = "\x1b[94m"; +pub const GREEN: &str = "\x1b[32m"; +pub const RED: &str = "\x1b[31m"; +pub const YELLOW: &str = "\x1b[33m"; /// Whether a stream should be written in color. pub fn enabled(stream_is_terminal: bool) -> bool { @@ -32,6 +35,21 @@ pub fn enabled(stream_is_terminal: bool) -> bool { ) } +/// Whether a line on a stream can be redrawn in place, for a spinner. +/// +/// Not a color question, so `NO_COLOR` does not turn it off; a terminal that +/// would print the escapes literally does. +pub fn redraws(stream_is_terminal: bool) -> bool { + stream_is_terminal + && allowed( + false, + env_value("TERM").as_deref(), + !cfg!(windows) + || env_value("WT_SESSION").is_some() + || env_value("TERM_PROGRAM").is_some(), + ) +} + fn allowed(no_color: bool, term: Option<&str>, console_renders_ansi: bool) -> bool { !no_color && term != Some("dumb") && console_renders_ansi } diff --git a/src/run_record.rs b/src/run_record.rs index fbaa914..19dae68 100644 --- a/src/run_record.rs +++ b/src/run_record.rs @@ -383,6 +383,11 @@ fn leaf<'a>( // Its forwarded words come back as subcommands of their own. return (path, command.find_subcommand(name).unwrap_or(command), sub); } + // The word after `workflow run` names a workflow on this machine, and + // a record keeps command paths, never what was typed as a value. + if path == [crate::workflow::COMMAND, "run"] { + return (path, command.find_subcommand(name).unwrap_or(command), sub); + } match command.find_subcommand(name) { Some(found) => command = found, None => break, @@ -414,7 +419,8 @@ fn tree_path(app: &Command, words: impl IntoIterator) -> Vec { path.push(found.get_name().to_string()); - if found.get_name() == TILESETS { + // See `leaf`: what follows `workflow run` is a workflow's name. + if found.get_name() == TILESETS || path == [crate::workflow::COMMAND, "run"] { break; } command = found; diff --git a/src/schema.rs b/src/schema.rs index de31d92..1a726c1 100644 --- a/src/schema.rs +++ b/src/schema.rs @@ -387,6 +387,14 @@ fn commands(app: &Command, specs: &[ServiceSpec], path: &[String]) -> Vec>; + +/// A checked workflow. Nothing constructs one except [`parse`]. +#[derive(Debug, Clone)] +pub struct Workflow { + pub name: String, + pub summary: String, + pub description: Option, + pub inputs: BTreeMap, + pub steps: Vec, + pub outputs: Option, + /// What `run` prints in text mode instead of the outputs as fields: a + /// template over the same inputs and steps. `-o json` still prints the + /// outputs. + pub result: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Input { + #[serde(rename = "type")] + pub kind: InputType, + #[serde(default)] + pub description: Option, + #[serde(default)] + pub required: bool, + #[serde(default)] + pub default: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum InputType { + String, + Number, + Boolean, +} + +impl InputType { + pub fn as_str(self) -> &'static str { + match self { + InputType::String => "string", + InputType::Number => "number", + InputType::Boolean => "boolean", + } + } + + pub fn accepts(self, value: &Value) -> bool { + matches!( + (self, value), + (InputType::String, Value::String(_)) + | (InputType::Number, Value::Number(_)) + | (InputType::Boolean, Value::Bool(_)) + ) + } +} + +#[derive(Debug, Clone)] +pub struct Step { + pub id: String, + pub name: Option, + pub action: Action, + pub stdin: Option, + /// Runs under `--dry-run` too. A script is told so through + /// [`DRY_RUN_ENV`] and promises to write nothing; a command step may be + /// marked only if its command changes nothing, which + /// [`super::runner::command_problems`] checks against the command tree. + pub dry_run: bool, + /// A command step's stdout, written as this file in the run's + /// [`super::workdir`] rather than read as its output. How a binary + /// response, such as a style's ZIP, reaches a later step. + pub save: Option, +} + +/// Set to `1` for a `dry_run` step while `--dry-run` runs it. +pub const DRY_RUN_ENV: &str = "MAPBOX_WORKFLOW_DRY_RUN"; + +#[derive(Debug, Clone)] +pub enum Action { + Command { + /// `styles get` as `["styles", "get"]`. + path: Vec, + args: Map, + }, + Script { + /// Relative to `scripts/`. + script: PathBuf, + interpreter: String, + args: Vec, + }, +} + +impl Step { + /// How progress and plans name the step. + pub fn label(&self) -> String { + match &self.action { + Action::Command { path, .. } => format!("mapbox {}", path.join(" ")), + Action::Script { + script, + interpreter, + .. + } => format!("{interpreter} {SCRIPTS_DIR}/{}", slash_path(script)), + } + } +} + +/// The file as written, before anything is checked. +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Raw { + version: u64, + name: String, + summary: String, + #[serde(default)] + description: Option, + #[serde(default)] + inputs: BTreeMap, + steps: Vec, + #[serde(default)] + outputs: Option, + #[serde(default)] + result: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct RawStep { + id: String, + #[serde(default)] + name: Option, + #[serde(default)] + command: Option, + #[serde(default)] + script: Option, + #[serde(default)] + interpreter: Option, + #[serde(default)] + args: Option, + #[serde(default)] + stdin: Option, + #[serde(default)] + dry_run: bool, + #[serde(default)] + save: Option, +} + +/// A workflow name is one directory name, lower-case and dash-separated — +/// `uninstall` turns it into a path it deletes, so nothing else may pass. +pub fn is_workflow_name(name: &str) -> bool { + !name.is_empty() + && name.len() <= 64 + && name.starts_with(|c: char| c.is_ascii_lowercase() || c.is_ascii_digit()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') +} + +/// Input names and step ids, which expressions refer to. +fn is_identifier(name: &str) -> bool { + name.starts_with(|c: char| c.is_ascii_lowercase()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_') +} + +/// A path that stays inside the directory it is relative to. +pub fn is_contained(path: &Path) -> bool { + !path.as_os_str().is_empty() + && path + .components() + .all(|component| matches!(component, Component::Normal(_))) +} + +pub fn slash_path(path: &Path) -> String { + path.components() + .map(|part| part.as_os_str().to_string_lossy()) + .collect::>() + .join("/") +} + +/// The interpreter a script runs under when the step names none. A short, +/// fixed list: guessing further is how a script ends up run by the wrong +/// program. +fn default_interpreter(script: &Path) -> Option<&'static str> { + match script.extension()?.to_str()? { + "sh" => Some("sh"), + "py" => Some("python3"), + "js" | "mjs" => Some("node"), + _ => None, + } +} + +/// Checks `files` as the workflow called `name`, and returns it or every +/// problem found — all of them at once, so fixing a workflow is not one +/// install attempt per mistake. +pub fn parse(name: &str, files: &Files) -> Result> { + let mut problems = vec![]; + + for path in files.keys() { + let allowed = path == Path::new(DEFINITION_FILE) + || path == Path::new(README_FILE) + || path.starts_with(SCRIPTS_DIR); + if !allowed { + problems.push(format!( + "`{}` does not belong in a workflow: only {DEFINITION_FILE}, {README_FILE} \ + and {SCRIPTS_DIR}/ do", + slash_path(path) + )); + } + } + + let Some(bytes) = files.get(Path::new(DEFINITION_FILE)) else { + problems.push(format!("there is no {DEFINITION_FILE}")); + return Err(problems); + }; + let raw: Raw = match serde_yaml::from_slice(bytes) { + Ok(raw) => raw, + Err(e) => { + problems.push(format!("{DEFINITION_FILE} is not valid: {e}")); + return Err(problems); + } + }; + + if raw.version != SCHEMA_VERSION { + problems.push(format!( + "`version: {}` is not one this CLI reads; it reads `version: {SCHEMA_VERSION}`", + raw.version + )); + } + if raw.name != name { + problems.push(format!( + "`name: {}` must match the directory it is in, `{name}`", + raw.name + )); + } + if !is_workflow_name(&raw.name) { + problems.push(format!( + "`name: {}` must be lower-case letters, digits and dashes", + raw.name + )); + } + if raw.summary.trim().is_empty() || raw.summary.contains('\n') { + problems.push("`summary` must be one non-empty line".to_string()); + } + + for (input, spec) in &raw.inputs { + if !is_identifier(input) { + problems.push(format!( + "input `{input}` must be lower-case letters, digits and underscores" + )); + } + if let Some(default) = &spec.default { + if spec.required { + problems.push(format!( + "input `{input}` is required and has a default; it can only be one" + )); + } + if !spec.kind.accepts(default) { + problems.push(format!( + "input `{input}`'s default is not a {}", + spec.kind.as_str() + )); + } + } + } + + if raw.steps.is_empty() { + problems.push("`steps` is empty".to_string()); + } + + let mut steps = vec![]; + let mut seen: BTreeSet = BTreeSet::new(); + let mut runs_in_dry_run: BTreeSet = BTreeSet::new(); + let mut referenced_scripts: BTreeSet = BTreeSet::new(); + for raw_step in raw.steps { + let id = raw_step.id.clone(); + if !is_identifier(&id) { + problems.push(format!( + "step `{id}`: the id must be lower-case letters, digits and underscores" + )); + } + if seen.contains(&id) { + problems.push(format!("step `{id}` appears twice")); + } + + for value in raw_step.args.iter().chain(raw_step.stdin.iter()) { + check_references( + value, + &format!("step `{id}`"), + &raw.inputs, + &seen, + &mut problems, + ); + } + + if raw_step.dry_run { + // A dry run skips every other step, so there is no output of + // theirs for this one to read. + for value in raw_step.args.iter().chain(raw_step.stdin.iter()) { + for reference in template::references(value).unwrap_or_default() { + if let Reference::Step { id: read, .. } = &reference { + if seen.contains(read) && !runs_in_dry_run.contains(read) { + problems.push(format!( + "step `{id}` runs under --dry-run but reads `{}`, from a \ + step that does not", + template::display(&reference) + )); + } + } + } + } + } + + let action = match (raw_step.command, raw_step.script) { + (Some(command), None) => { + if let Some(save) = &raw_step.save { + let path = Path::new(save); + if !is_contained(path) || path.components().count() != 1 { + problems.push(format!( + "step `{id}`: `save: {save}` must be a plain file name" + )); + } + } + if raw_step.interpreter.is_some() { + problems.push(format!( + "step `{id}`: `interpreter` only applies to a `script` step" + )); + } + let path: Vec = command.split_whitespace().map(String::from).collect(); + if path.is_empty() { + problems.push(format!("step `{id}`: `command` is empty")); + } + let args = match raw_step.args { + None => Map::new(), + Some(Value::Object(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a command's `args` is a mapping of argument name to value" + )); + Map::new() + } + }; + Some(Action::Command { path, args }) + } + (None, Some(script)) => { + if raw_step.save.is_some() { + problems.push(format!( + "step `{id}`: `save` only applies to a `command` step; a script \ + writes its own files" + )); + } + let script = PathBuf::from(script); + let under_scripts = Path::new(SCRIPTS_DIR).join(&script); + if !is_contained(&script) { + problems.push(format!( + "step `{id}`: `script: {}` must be a path inside {SCRIPTS_DIR}/", + script.display() + )); + } else if !files.contains_key(&under_scripts) { + problems.push(format!( + "step `{id}`: there is no {SCRIPTS_DIR}/{}", + slash_path(&script) + )); + } + referenced_scripts.insert(under_scripts); + + let interpreter = match raw_step + .interpreter + .or_else(|| default_interpreter(&script).map(String::from)) + { + Some(interpreter) + if !interpreter.is_empty() + && !interpreter.contains(char::is_whitespace) => + { + interpreter + } + Some(interpreter) => { + problems.push(format!( + "step `{id}`: `interpreter: {interpreter}` must be one program name" + )); + String::new() + } + None => { + problems.push(format!( + "step `{id}`: name an `interpreter` for {}; only .sh, .py and .js \ + have a default", + slash_path(&script) + )); + String::new() + } + }; + let args = match raw_step.args { + None => vec![], + Some(Value::Array(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a script's `args` is a list of values" + )); + vec![] + } + }; + Some(Action::Script { + script, + interpreter, + args, + }) + } + (Some(_), Some(_)) => { + problems.push(format!( + "step `{id}` has both `command` and `script`; a step is one or the other" + )); + None + } + (None, None) => { + problems.push(format!("step `{id}` needs a `command` or a `script`")); + None + } + }; + + seen.insert(id.clone()); + if raw_step.dry_run { + runs_in_dry_run.insert(id.clone()); + } + if let Some(action) = action { + steps.push(Step { + id, + name: raw_step.name, + action, + stdin: raw_step.stdin, + dry_run: raw_step.dry_run, + save: raw_step.save, + }); + } + } + + if let Some(outputs) = &raw.outputs { + check_references(outputs, "`outputs`", &raw.inputs, &seen, &mut problems); + } + if let Some(result) = &raw.result { + let result = Value::String(result.clone()); + check_references(&result, "`result`", &raw.inputs, &seen, &mut problems); + } + + for path in files.keys().filter(|path| path.starts_with(SCRIPTS_DIR)) { + if !referenced_scripts.contains(path) { + problems.push(format!( + "`{}` is not used by any step; a workflow ships only the scripts it runs", + slash_path(path) + )); + } + } + + if !problems.is_empty() { + return Err(problems); + } + Ok(Workflow { + name: raw.name, + summary: raw.summary, + description: raw.description, + inputs: raw.inputs, + steps, + outputs: raw.outputs, + result: raw.result, + }) +} + +/// Every expression in `value` must name a declared input or a step that +/// has already run by the time `value` is read. +fn check_references( + value: &Value, + owner: &str, + inputs: &BTreeMap, + earlier_steps: &BTreeSet, + problems: &mut Vec, +) { + let references = match template::references(value) { + Ok(references) => references, + Err(e) => { + problems.push(format!("{owner}: {e}")); + return; + } + }; + for reference in references { + match &reference { + Reference::Input(name) if !inputs.contains_key(name) => problems.push(format!( + "{owner}: `{}` names no declared input", + template::display(&reference) + )), + Reference::Step { id, .. } if !earlier_steps.contains(id) => problems.push(format!( + "{owner}: `{}` names no earlier step", + template::display(&reference) + )), + _ => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const MINIMAL: &str = "\ +version: 1 +name: demo +summary: A demo +inputs: + style_id: { type: string, required: true } +steps: + - id: fetch + command: styles get + args: { style-id: '${{ inputs.style_id }}' } + - id: shape + script: shape.py + stdin: '${{ steps.fetch.output }}' +outputs: + id: '${{ steps.shape.output.id }}' +"; + + fn files(entries: &[(&str, &str)]) -> Files { + entries + .iter() + .map(|(path, body)| (PathBuf::from(path), body.as_bytes().to_vec())) + .collect() + } + + fn problems(entries: &[(&str, &str)]) -> Vec { + parse("demo", &files(entries)).expect_err("should be refused") + } + + #[test] + fn a_minimal_workflow_parses() { + let workflow = parse( + "demo", + &files(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + (README_FILE, ""), + ]), + ) + .unwrap(); + assert_eq!(workflow.steps.len(), 2); + assert_eq!(workflow.steps[0].label(), "mapbox styles get"); + assert_eq!(workflow.steps[1].label(), "python3 scripts/shape.py"); + } + + #[test] + fn the_name_must_match_the_directory() { + let found = parse( + "other", + &files(&[(DEFINITION_FILE, MINIMAL), ("scripts/shape.py", "")]), + ) + .unwrap_err(); + assert!(found.iter().any(|p| p.contains("must match")), "{found:?}"); + } + + #[test] + fn stray_and_unused_files_are_refused() { + let found = problems(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + ("scripts/old.py", ""), + ("notes.txt", ""), + ]); + assert!( + found.iter().any(|p| p.contains("scripts/old.py")), + "{found:?}" + ); + assert!(found.iter().any(|p| p.contains("notes.txt")), "{found:?}"); + } + + #[test] + fn a_missing_script_is_refused() { + let found = problems(&[(DEFINITION_FILE, MINIMAL)]); + assert!( + found.iter().any(|p| p.contains("no scripts/shape.py")), + "{found:?}" + ); + } + + #[test] + fn a_script_path_cannot_leave_scripts() { + let yaml = MINIMAL.replace("script: shape.py", "script: ../workflow.yaml"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("inside scripts/")), + "{found:?}" + ); + } + + #[test] + fn a_reference_must_point_backwards() { + let yaml = MINIMAL.replace( + "args: { style-id: '${{ inputs.style_id }}' }", + "args: { style-id: '${{ steps.shape.output.id }}' }", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no earlier step")), + "{found:?}" + ); + } + + #[test] + fn save_is_a_plain_file_name_on_a_command_step() { + for bad in ["../x.zip", "dir/x.zip", "/tmp/x.zip"] { + let yaml = MINIMAL.replace( + " command: styles get\n", + &format!(" command: styles get\n save: '{bad}'\n"), + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.contains("must be a plain file name")), + "{bad}: {found:?}" + ); + } + + let yaml = MINIMAL.replace( + " script: shape.py\n", + " script: shape.py\n save: out.json\n", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.contains("only applies to a `command` step")), + "{found:?}" + ); + } + + /// A dry run skips the steps not marked `dry_run`, so a step that is + /// marked cannot read their output. + #[test] + fn a_dry_run_step_reads_only_dry_run_steps() { + let yaml = MINIMAL.replace( + " script: shape.py\n", + " script: shape.py\n dry_run: true\n", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.contains("runs under --dry-run but reads `steps.fetch")), + "{found:?}" + ); + } + + #[test] + fn result_may_name_only_earlier_steps_and_declared_inputs() { + let yaml = format!("{MINIMAL}result: 'Made ${{{{ steps.nope.output.id }}}}'\n"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.starts_with("`result`") && p.contains("no earlier step")), + "{found:?}" + ); + } + + #[test] + fn an_undeclared_input_is_refused() { + let yaml = MINIMAL.replace("inputs.style_id }}'", "inputs.nope }}'"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no declared input")), + "{found:?}" + ); + } + + #[test] + fn an_unknown_field_is_refused() { + let yaml = MINIMAL.replace("summary: A demo", "summary: A demo\nretries: 3"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!(found.iter().any(|p| p.contains("retries")), "{found:?}"); + } + + #[test] + fn an_unknown_extension_needs_an_interpreter() { + let yaml = MINIMAL.replace("shape.py", "shape.rb"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.rb", "")]); + assert!(found.iter().any(|p| p.contains("interpreter")), "{found:?}"); + } + + #[test] + fn workflow_names_are_one_plain_directory_name() { + for good in ["copy-style", "a", "v2-sync"] { + assert!(is_workflow_name(good), "{good}"); + } + for bad in ["", "..", "a/b", "/tmp", "Copy", "-x", "a_b", "a.b"] { + assert!(!is_workflow_name(bad), "{bad}"); + } + } +} diff --git a/src/workflow/mod.rs b/src/workflow/mod.rs new file mode 100644 index 0000000..217d200 --- /dev/null +++ b/src/workflow/mod.rs @@ -0,0 +1,878 @@ +//! `mapbox workflow` — install and run workflows: named, multi-step recipes +//! of `mapbox` commands and scripts. Beta, in development, and not +//! recommended for use: every subcommand says so on stderr. +//! +//! A workflow is a directory holding a `workflow.yaml` and the scripts it +//! runs (see [`definition`] for the schema and the layout rules). None ships +//! inside the binary. `install` copies one in, from a local directory or a +//! GitHub repository's `workflow//`, and `run` runs only what +//! is installed. +//! +//! What this refuses to be: a scheduler, a retry engine, or a language. +//! Steps run in order and the first failure stops the run. Logic belongs in +//! a script step. + +pub mod definition; +pub mod progress; +mod prose; +pub mod runner; +pub mod store; +pub mod template; +pub mod workdir; + +use std::ffi::OsString; +use std::io::IsTerminal; +use std::path::Path; + +use anyhow::Result; +use clap::builder::{StringValueParser, TypedValueParser}; +use clap::{Arg, ArgAction, ArgMatches, Command}; +use serde_json::{json, Value}; + +use crate::confirm; +use crate::executor; +use crate::output::{self, field_lines, style, Mode}; + +pub const COMMAND: &str = "workflow"; + +/// Printed by every subcommand. Nothing here is a promise yet, and a +/// person who found the command in `--help` should not build on it. One +/// line on purpose: `--help` and the docs carry the rest. +const NOTICE: &str = "`mapbox workflow` is in development and not recommended for use."; + +const NAME_ARG: &str = "name"; +const SOURCE_ARG: &str = "source"; +const REPO_ARG: &str = "repo"; +const REF_ARG: &str = "ref"; +const FORCE_ARG: &str = "force"; +const RUN: &str = "run"; + +pub fn command() -> Command { + let name = || { + Arg::new(NAME_ARG) + .value_name("NAME") + .required(true) + .help("Name of an installed workflow") + }; + Command::new(COMMAND) + .about( + "Install and run multi-step workflows of mapbox commands and scripts \ + (beta, in development, not recommended for use)", + ) + .long_about( + "Install and run workflows: named, multi-step recipes of mapbox commands and \ + scripts, defined in a workflow.yaml.\n\n\ + A workflow runs only once it is installed, from a local directory or from a \ + GitHub repository's workflow// directory.\n\n\ + Beta and in development, and not recommended for use: the commands, the \ + workflow format and the published workflows may change or be removed \ + without notice.", + ) + .subcommand_required(true) + .subcommand(Command::new("list").about("List installed workflows")) + .subcommand( + Command::new("show") + .about("Describe an installed workflow: its inputs and steps") + .arg(name()), + ) + .subcommand( + Command::new("install") + .about("Install a workflow from a local directory or from GitHub") + .arg( + Arg::new(SOURCE_ARG) + .value_name("SOURCE") + .required(true) + .help( + "A workflow name, fetched from GitHub, or a path to a local \ + workflow directory (anything with a `/` or starting with `.`)", + ), + ) + .arg( + Arg::new(REPO_ARG) + .long(REPO_ARG) + .value_name("OWNER/REPO") + .help(format!( + "GitHub repository to install from [default: {}]. Reads \ + GH_TOKEN or GITHUB_TOKEN for a private one", + store::DEFAULT_REPO + )), + ) + .arg( + Arg::new(REF_ARG) + .long(REF_ARG) + .value_name("REF") + .help(format!( + "Branch, tag or commit to install from [default: {}]", + store::DEFAULT_REF + )), + ) + .arg( + Arg::new(FORCE_ARG) + .long(FORCE_ARG) + .action(ArgAction::SetTrue) + .help("Replace a workflow that is already installed"), + ) + .arg(executor::dry_run_arg( + "Check the workflow and list the files it would write, then exit", + )), + ) + .subcommand( + Command::new("uninstall") + .about("Remove an installed workflow") + .arg(Arg::new(NAME_ARG).value_name("NAME").required(true).help( + "Name of an installed workflow, or the directory it was installed \ + from", + )) + .arg(executor::dry_run_arg( + "Say what it would remove, then exit without removing it", + )), + ) + .subcommand( + Command::new(RUN) + .about("Run an installed workflow") + .long_about( + "Run an installed workflow, giving each of its inputs as a flag: \ + `mapbox workflow run copy-style --style-id --from-profile source \ + --to-profile target`.\n\n\ + `mapbox workflow run --help` lists that workflow's flags, and \ + `--dry-run` prints its plan without running a step.", + ) + .subcommand_value_name("NAME") + .subcommand_help_heading("Workflows") + .subcommand_required(true) + // A workflow is a subcommand only once `with_run_target` has + // read it from disk. Any other name still parses, so that + // `run` can say it is not installed, or why it cannot load. + .allow_external_subcommands(true), + ) +} + +/// The `run` subcommand for one installed workflow: its inputs as flags, +/// typed and required as the definition says, so that `--help`, a missing +/// input and a misspelled flag read as they do for every other command. +fn run_command(workflow: &definition::Workflow) -> Command { + let mut command = Command::new(workflow.name.clone()) + .about(workflow.summary.clone()) + .arg(executor::dry_run_arg( + "Check the workflow and its inputs and print the plan, then exit without \ + running a step", + )); + if let Some(description) = &workflow.description { + command = command.long_about(format!( + "{}\n\n{}", + workflow.summary, + prose::render(description, false) + )); + } + for (name, input) in &workflow.inputs { + let flag = runner::input_flag(name); + let mut help = input.description.clone().unwrap_or_default(); + if let Some(default) = &input.default { + help.push_str(&format!( + " [default: {}]", + template::as_text(default).unwrap_or_default() + )); + } + let mut arg = Arg::new(name.clone()) + .long(flag.clone()) + .value_name(flag.to_uppercase().replace('-', "_")) + .required(input.required) + .help(help.trim().to_string()) + // Apart from the globals clap lists beside them, and required + // ones first, as `workflow show` orders them. + .help_heading("Inputs") + .display_order(usize::from(!input.required)); + if flag != *name { + arg = arg.alias(name.clone()); + } + arg = match input.kind { + definition::InputType::String => arg, + definition::InputType::Number => { + arg.value_parser(StringValueParser::new().try_map(|raw: String| { + runner::parse_number(&raw) + .map(|_| raw.clone()) + .ok_or_else(|| format!("`{raw}` is not a number")) + })) + } + // `--flag` alone means true, so a boolean reads like any flag. + definition::InputType::Boolean => arg + .value_parser(["true", "false"]) + .num_args(0..=1) + .default_missing_value("true"), + }; + command = command.arg(arg); + } + command +} + +/// The workflow a command line runs, when it is `… workflow run …`. +/// +/// Read off argv because it has to be known before clap parses: the +/// workflow's flags come from its `workflow.yaml`, and parsing is what they +/// are needed for. +fn run_target(argv: &[OsString]) -> Option { + let words: Vec<&str> = argv.iter().skip(1).filter_map(|arg| arg.to_str()).collect(); + let at = words.windows(2).position(|pair| pair == [COMMAND, RUN])?; + let name = *words.get(at + 2)?; + definition::is_workflow_name(name).then(|| name.to_string()) +} + +/// `app` with the workflow this command line runs attached under `run`, so +/// that clap parses its inputs as flags. Anything that stops it loading is +/// left for `run` to report, with the command tree unchanged: every other +/// command pays nothing for this, and a broken workflow cannot break the +/// parse. +pub fn with_run_target(app: Command, argv: &[OsString]) -> Command { + let Some(name) = run_target(argv) else { + return app; + }; + let Ok(found) = store::load(&name) else { + return app; + }; + if !runner::command_problems(&app, &found.workflow).is_empty() { + return app; + } + let run = run_command(&found.workflow); + app.mut_subcommand(COMMAND, |workflow| { + workflow.mut_subcommand(RUN, |parent| parent.subcommand(run)) + }) +} + +/// What the globals say, for the subcommands that need them. +pub struct RunFlags<'a> { + pub globals: &'a ArgMatches, + pub debug: bool, + pub assume_yes: bool, +} + +pub fn run(app: &Command, matches: &ArgMatches, flags: RunFlags, mode: Mode) -> Result<()> { + let terminal = std::io::stderr().is_terminal(); + let color = style::enabled(terminal); + output::progress(&format!( + "{} {}", + style::paint("Beta:", style::BOLD, color), + style::dim_prose(NOTICE, color) + )); + // At a terminal the notice sits right above the result; a blank line + // keeps it from reading as the result's first line. + if terminal { + output::progress(""); + } + match matches.subcommand() { + Some(("list", _)) => list(mode), + Some(("show", m)) => show(app, name(m), mode), + Some(("install", m)) => install(app, m, flags.debug, mode), + Some(("uninstall", m)) => { + uninstall(name(m), executor::wants_dry_run(m), flags.assume_yes, mode) + } + Some((RUN, m)) => match m.subcommand() { + Some((name, workflow_matches)) => { + run_workflow(app, name, workflow_matches, flags.globals, mode) + } + None => unreachable!("`run` sets subcommand_required(true)"), + }, + _ => unreachable!("`workflow` sets subcommand_required(true)"), + } +} + +fn name(matches: &ArgMatches) -> &str { + matches.get_one::(NAME_ARG).expect("required") +} + +/// A path as a person reads it: under the home directory as `~/…`. Text +/// output only; JSON keeps the absolute path a program can open. +pub(crate) fn tilde(path: &Path) -> String { + match dirs::home_dir().and_then(|home| path.strip_prefix(home).ok().map(Path::to_path_buf)) { + Some(rest) if rest.as_os_str().is_empty() => "~".to_string(), + Some(rest) => format!("~/{}", rest.display()), + None => path.display().to_string(), + } +} + +fn source_label(meta: &store::Meta) -> String { + match &meta.git_ref { + Some(git_ref) => format!("{}@{git_ref}", meta.source), + None => meta.source.clone(), + } +} + +fn source_text(meta: &store::Meta) -> String { + match &meta.git_ref { + Some(_) => source_label(meta), + None => tilde(Path::new(&meta.source)), + } +} + +/// Rows of cells, each column padded to its widest cell. Padding is +/// measured before painting, so escapes never skew the alignment. +fn columns(rows: &[Vec<(String, &'static str)>], color: bool) -> String { + let widths: Vec = (0..rows.iter().map(Vec::len).max().unwrap_or(0)) + .map(|column| { + rows.iter() + .filter_map(|row| row.get(column)) + .map(|(cell, _)| cell.chars().count()) + .max() + .unwrap_or(0) + }) + .collect(); + rows.iter() + .map(|row| { + let last = row.len().saturating_sub(1); + row.iter() + .enumerate() + .map(|(column, (cell, paint))| { + let pad = if column == last { + String::new() + } else { + " ".repeat(widths[column] - cell.chars().count() + 2) + }; + format!( + "{}{pad}", + style::paint(cell, paint, color && !paint.is_empty()) + ) + }) + .collect::() + }) + .collect::>() + .join("\n") +} + +fn heading(text: &str, color: bool) -> String { + style::paint(text, style::BOLD, color) +} + +/// The `run` line a person would type, with every required input spelled +/// out so the next command is one they only have to fill in. +fn run_example(workflow: &definition::Workflow) -> String { + let mut line = format!("mapbox workflow run {}", workflow.name); + for (name, input) in &workflow.inputs { + if input.required { + line.push_str(&format!(" --{} …", runner::input_flag(name))); + } + } + line +} + +/// Tips go with the text rendering only, as every other command's do. +fn tips(mode: Mode, tips: &[String]) { + if !mode.is_json() { + output::print_tips(tips); + } +} + +fn list(mode: Mode) -> Result<()> { + let installed = store::list()?; + let color = output::result_in_color(); + let mut rows = vec![]; + let mut table = vec![vec![ + ("NAME".to_string(), style::BOLD), + ("SUMMARY".to_string(), style::BOLD), + ]]; + for (name, loaded) in &installed { + match loaded { + Ok(found) => { + rows.push(json!({ + "name": name, + "summary": found.workflow.summary, + "source": source_label(&found.meta), + "path": found.root, + })); + table.push(vec![ + (name.clone(), ""), + (found.workflow.summary.clone(), ""), + ]); + } + Err(e) => { + rows.push(json!({ "name": name, "error": format!("{e:#}") })); + table.push(vec![ + (name.clone(), ""), + (format!("cannot load: {e:#}"), style::DIM), + ]); + } + } + } + + if installed.is_empty() { + output::emit(mode, "No workflows installed.", Value::Array(rows))?; + tips( + mode, + &["Install one with `mapbox workflow install ` or `mapbox workflow install ./`." + .to_string()], + ); + return Ok(()); + } + output::emit(mode, &columns(&table, color), Value::Array(rows))?; + tips( + mode, + &["`mapbox workflow show ` for a workflow's inputs and steps.".to_string()], + ); + Ok(()) +} + +fn show(app: &Command, name: &str, mode: Mode) -> Result<()> { + let found = store::load(name)?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + let color = output::result_in_color(); + + let inputs: Vec = workflow + .inputs + .iter() + .map(|(name, input)| { + json!({ + "name": name, + "type": input.kind.as_str(), + "required": input.required, + "default": input.default, + "description": input.description, + }) + }) + .collect(); + let steps: Vec = workflow + .steps + .iter() + .map(|step| json!({ "id": step.id, "name": step.name, "run": step.label() })) + .collect(); + + // Fixed order for every workflow: what it is, what it needs, what it + // does, then where this copy came from. + let mut text = format!( + "{}\n{}", + heading(&workflow.name, color), + style::paint(&workflow.summary, style::DIM, color) + ); + if let Some(description) = &workflow.description { + text.push_str(&format!("\n\n{}", prose::render(description, color))); + } + + text.push_str(&format!("\n\n{}", heading("Inputs", color))); + if workflow.inputs.is_empty() { + text.push_str("\n none"); + } else { + // Required inputs first: they are the ones a run cannot do without. + let mut ordered: Vec<_> = workflow.inputs.iter().collect(); + ordered.sort_by_key(|(name, input)| (!input.required, name.as_str())); + let rows: Vec> = ordered + .into_iter() + .map(|(name, input)| { + let mut kind = input.kind.as_str().to_string(); + if input.required { + kind.push_str(", required"); + } + if let Some(default) = &input.default { + kind.push_str(&format!(", default {default}")); + } + vec![ + (format!(" {name}"), ""), + (kind, style::DIM), + (input.description.clone().unwrap_or_default(), ""), + ] + }) + .collect(); + text.push('\n'); + text.push_str(&columns(&rows, color)); + } + + text.push_str(&format!("\n\n{}\n", heading("Steps", color))); + let rows: Vec> = workflow + .steps + .iter() + .enumerate() + .map(|(index, step)| { + vec![ + ( + format!( + " {}. {}", + index + 1, + step.name.as_deref().unwrap_or(&step.id) + ), + "", + ), + (step.label(), style::DIM), + ] + }) + .collect(); + text.push_str(&columns(&rows, color)); + + if !problems.is_empty() { + text.push_str(&format!("\n\n{}", heading("This CLI cannot run it", color))); + for problem in &problems { + text.push_str(&format!("\n - {problem}")); + } + } + + text.push_str(&format!("\n\n{}\n", heading("Installed", color))); + text.push_str(&columns( + &[ + vec![ + (" Source".to_string(), ""), + (source_text(&found.meta), style::DIM), + ], + vec![(" Path".to_string(), ""), (tilde(&found.root), style::DIM)], + ], + color, + )); + + output::emit( + mode, + &text, + json!({ + "name": workflow.name, + "summary": workflow.summary, + "description": workflow.description, + "source": source_label(&found.meta), + "path": found.root, + "inputs": inputs, + "steps": steps, + "problems": problems, + }), + )?; + if problems.is_empty() { + tips(mode, &[format!("Run it with `{}`.", run_example(workflow))]); + } + Ok(()) +} + +/// A source with a path separator or a leading `.` is a directory; anything +/// else is a name to look up on GitHub. A workflow name can hold neither, +/// so the two never overlap. +fn is_local_source(source: &str) -> bool { + source.starts_with('.') || source.contains('/') || source.contains('\\') +} + +fn install(app: &Command, matches: &ArgMatches, debug: bool, mode: Mode) -> Result<()> { + let source = matches.get_one::(SOURCE_ARG).expect("required"); + let repo = matches.get_one::(REPO_ARG); + let git_ref = matches.get_one::(REF_ARG); + let force = matches.get_flag(FORCE_ARG); + let dry_run = executor::wants_dry_run(matches); + + let package = if is_local_source(source) { + if repo.is_some() || git_ref.is_some() { + return Err(output::CliError::new( + "invalid_arguments", + "`--repo` and `--ref` name a GitHub source; this one is a local directory.", + ) + .into()); + } + store::read_local(std::path::Path::new(source))? + } else { + store::fetch_github( + source, + repo.map_or(store::DEFAULT_REPO, String::as_str), + git_ref.map_or(store::DEFAULT_REF, String::as_str), + debug, + )? + }; + let workflow = &package.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let files: Vec = package + .files + .keys() + .map(|path| definition::slash_path(path)) + .collect(); + let target = store::root_path()?.join(&workflow.name); + let summary = json!({ + "name": workflow.name, + "source": source_label(&package.meta), + "path": target, + "files": files, + "dry_run": dry_run, + }); + + let color = output::result_in_color(); + let fields = field_lines( + &[ + ("Source", source_text(&package.meta)), + ("Path", tilde(&target)), + ("Files", files.join(", ")), + ], + color, + ); + + // Checked here as well as in `store::install`, so the error can name + // the line to retry with; the store's own check covers a race. + if target.exists() && !force { + let retry = format!("mapbox workflow install {source} --force"); + return Err(store::already_installed( + &workflow.name, + &target, + Some(retry), + )); + } + + if dry_run { + let text = format!( + "Dry run — nothing was written. Would install {}:\n\n{fields}", + heading(&workflow.name, color) + ); + return output::emit(mode, &text, summary); + } + + store::install(&package, force)?; + let text = format!("Installed {}\n\n{fields}", heading(&workflow.name, color)); + output::emit(mode, &text, summary)?; + tips( + mode, + &[ + format!("Run it with `{}`.", run_example(workflow)), + format!( + "`mapbox workflow show {}` describes its inputs.", + workflow.name + ), + ], + ); + Ok(()) +} + +/// The workflow `uninstall` means. A path is accepted, as `install` takes +/// one, and names the workflow by its directory — which must still be a +/// plain workflow name, so nothing but an installed workflow is removed. +fn uninstall_name(given: &str) -> &str { + if !is_local_source(given) { + return given; + } + Path::new(given) + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(given) +} + +fn uninstall(given: &str, dry_run: bool, assume_yes: bool, mode: Mode) -> Result<()> { + let name = uninstall_name(given); + let found = store::installed_dir(name)?; + let Some(dir) = found else { + // `load` words the error, with the commands to try instead. + return store::load(name).map(|_| ()); + }; + if dry_run { + return output::emit( + mode, + &format!( + "Dry run — nothing was removed. Would remove {}.", + tilde(&dir) + ), + json!({ "name": name, "path": dir, "dry_run": true }), + ); + } + confirm::destructive_local_action( + &format!("Remove the workflow at {}?", tilde(&dir)), + assume_yes, + )?; + let dir = store::uninstall(name)?; + output::emit( + mode, + &format!("Removed {name} ({}).", tilde(&dir)), + json!({ "name": name, "path": dir, "dry_run": false }), + ) +} + +fn run_workflow( + app: &Command, + name: &str, + matches: &ArgMatches, + globals: &ArgMatches, + mode: Mode, +) -> Result<()> { + store::check_name(name)?; + let found = store::load(name)?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let attached = app + .find_subcommand(COMMAND) + .and_then(|group| group.find_subcommand(RUN)) + .and_then(|run| run.find_subcommand(name)) + .is_some(); + if !attached { + // Only when argv was spelled in a way `run_target` does not read. + return Err(output::CliError::new( + "invalid_arguments", + format!( + "Could not read `{name}`'s inputs from this command line. Write it as \ + `{}`.", + run_example(workflow) + ), + ) + .into()); + } + + let given: Vec<(String, String)> = workflow + .inputs + .keys() + .filter_map(|input| { + matches + .get_one::(input) + .map(|value| (input.clone(), value.clone())) + }) + .collect(); + let inputs = runner::read_inputs(workflow, &given)?; + + let inherited = runner::Inherited::from_matches(globals); + + if executor::wants_dry_run(matches) { + let mut plan = runner::plan(workflow, &inputs); + // Steps that promise to write nothing run for real, so the plan can + // say what the rest would do with actual data rather than guess. + let rehearsed = workflow.steps.iter().any(|step| step.dry_run); + if rehearsed { + let results = runner::dry_run(app, workflow, &found.root, &inputs, &inherited)?; + plan["results"] = json!(results); + } + let color = output::result_in_color(); + let input_rows: Vec> = inputs + .iter() + .map(|(name, value)| { + let shown = template::as_text(value).unwrap_or_else(|| "(none)".to_string()); + vec![(format!(" {name}"), ""), (shown, "")] + }) + .collect(); + let step_rows: Vec> = workflow + .steps + .iter() + .enumerate() + .map(|(index, step)| { + vec![ + ( + format!( + " {}. {}", + index + 1, + step.name.as_deref().unwrap_or(&step.id) + ), + "", + ), + ( + if step.dry_run { + format!("{} (ran in this dry run)", step.label()) + } else { + step.label() + }, + style::DIM, + ), + ] + }) + .collect(); + let mut text = if rehearsed { + format!( + "Dry run — only the steps that support it ran, and they wrote nothing. \ + Would run {}:", + heading(&workflow.name, color) + ) + } else { + format!( + "Dry run — nothing was run. Would run {}:", + heading(&workflow.name, color) + ) + }; + if !input_rows.is_empty() { + text.push_str(&format!( + "\n\n{}\n{}", + heading("Inputs", color), + columns(&input_rows, color) + )); + } + text.push_str(&format!( + "\n\n{}\n{}", + heading("Steps", color), + columns(&step_rows, color) + )); + return output::emit(mode, &text, plan); + } + + let finished = runner::run(app, workflow, &found.root, &inputs, &inherited)?; + match finished.text { + // The workflow's own wording for a person; `-o json` still gets the + // outputs, which is what a script reads. + Some(text) if !mode.is_json() => output::emit(mode, text.trim_end(), finished.outputs), + _ => output::emit_result(mode, &finished.outputs), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn uninstall_reads_a_path_as_its_directory_name() { + assert_eq!(uninstall_name("copy-style"), "copy-style"); + assert_eq!(uninstall_name("./workflow/copy-style"), "copy-style"); + assert_eq!(uninstall_name("./workflow/copy-style/"), "copy-style"); + // No directory name at all: left as typed, for the name check to refuse. + assert_eq!(uninstall_name(".."), ".."); + } + + #[test] + fn columns_line_up_with_color_on_or_off() { + let rows = vec![ + vec![ + ("NAME".to_string(), style::BOLD), + ("SUMMARY".to_string(), style::BOLD), + ], + vec![ + ("copy-style".to_string(), ""), + ("Copy a style".to_string(), ""), + ], + ]; + let plain = columns(&rows, false); + assert_eq!(plain, "NAME SUMMARY\ncopy-style Copy a style"); + assert_eq!(style::strip(&columns(&rows, true)), plain); + } + + #[test] + fn a_path_under_home_is_shown_with_a_tilde() { + let home = dirs::home_dir().expect("a home directory"); + assert_eq!( + tilde(&home.join(".mapbox/workflows/x")), + "~/.mapbox/workflows/x" + ); + assert_eq!(tilde(Path::new("/opt/x")), "/opt/x"); + } + + #[test] + fn a_name_and_a_path_never_look_alike() { + for local in [ + "./copy-style", + "../x", + "workflow/copy-style", + "/abs", + ".", + "a\\b", + ] { + assert!(is_local_source(local), "{local}"); + } + assert!(!is_local_source("copy-style")); + } + + /// Every workflow this repository publishes is one `install` accepts. + /// The rules are the same as for anybody's, so this is also what + /// catches a stray file or a step naming a command that was renamed. + #[test] + fn every_published_workflow_is_valid() { + let app = crate::build_app(&crate::spec::effective_services().expect("bundled specs")); + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join(store::REPO_PREFIX); + let mut checked = 0; + for dir in std::fs::read_dir(&root) + .expect("workflow/ exists") + .flatten() + .filter(|entry| entry.path().is_dir()) + { + let package = store::read_local(&dir.path()) + .unwrap_or_else(|e| panic!("{}: {e:#}", dir.path().display())); + let problems = runner::command_problems(&app, &package.workflow); + assert!( + problems.is_empty(), + "{}: {problems:#?}", + dir.path().display() + ); + checked += 1; + } + assert!(checked > 0, "no workflows found under workflow/"); + } +} diff --git a/src/workflow/progress.rs b/src/workflow/progress.rs new file mode 100644 index 0000000..79c08dd --- /dev/null +++ b/src/workflow/progress.rs @@ -0,0 +1,332 @@ +//! How a workflow run shows its steps on stderr. +//! +//! Two modes, chosen once per run. Where the line can be redrawn, each step +//! is its title, then what it reports, indented, then one status line: a +//! spinner while it runs, a mark and a duration once it ends. Only that last +//! line is ever redrawn, so a detail that wraps cannot throw the drawing off. Anywhere else +//! — a pipe, a CI log, an agent reading stderr — each step is one plain +//! `[n/total]` line and its reports follow as they arrive, with nothing that +//! only makes sense on a screen. +//! +//! A script reports through its stderr. A line starting with +//! [`PROGRESS_PREFIX`] replaces the text beside the spinner and is dropped +//! where there is none. One starting with [`WARN_PREFIX`] is a warning: shown +//! where it happened and listed again once the run ends, so it is not lost +//! among the details. Any other line is a detail, kept in both modes unless +//! the run is `--quiet`. +//! +//! A command step is never animated. It keeps the terminal, where a +//! confirmation prompt has to be able to reach the person running it, so +//! nothing here may draw over its output. + +use std::io::{IsTerminal, Write}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread::JoinHandle; +use std::time::{Duration, Instant}; + +use crate::output::style; + +/// Marks a script's stderr line as progress rather than a detail. +pub const PROGRESS_PREFIX: &str = "::progress "; +/// Marks a script's stderr line as a warning. +pub const WARN_PREFIX: &str = "::warn "; + +const FRAMES: [&str; 10] = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"]; +const TICK: Duration = Duration::from_millis(80); +const DETAIL_INDENT: &str = " "; + +pub struct Display { + live: bool, + color: bool, + quiet: bool, + total: usize, + warnings: Arc>>, +} + +impl Display { + pub fn for_stderr(total: usize, quiet: bool) -> Self { + let terminal = std::io::stderr().is_terminal(); + Display { + live: style::redraws(terminal), + color: style::enabled(terminal), + quiet, + total, + warnings: Arc::default(), + } + } + + /// Lists every warning a step reported, once the steps are done. + pub fn finish_run(&self) { + let warnings = self.warnings.lock().expect("warnings"); + if warnings.is_empty() { + return; + } + let heading = if warnings.len() == 1 { + "Warning" + } else { + "Warnings" + }; + eprintln!("{}", style::paint(heading, style::BOLD, self.color)); + for warning in warnings.iter() { + eprintln!( + " {} {warning}", + style::paint("!", style::YELLOW, self.color) + ); + } + eprintln!(); + } + + /// A step that will not run, because this is a dry run and it did not + /// promise to write nothing. + pub fn skipped(&self, index: usize, title: &str, label: &str) { + let line = if self.live { + format!( + "{} {} {}", + style::paint("○", style::DIM, self.color), + style::paint(title, style::DIM, self.color), + style::paint("skipped in a dry run", style::DIM, self.color), + ) + } else { + format!( + "{} {} {}", + self.counter(index), + style::paint(title, style::BOLD, self.color), + style::paint( + &format!("({label}, skipped in a dry run)"), + style::DIM, + self.color + ), + ) + }; + eprintln!("{line}"); + } + + pub fn start(&self, index: usize, title: &str, label: &str, animate: bool) -> StepView { + let view = StepView { + state: Arc::new(Mutex::new(State { + progress: String::new(), + frame: 0, + spinning: self.live && animate, + })), + stop: Arc::new(AtomicBool::new(false)), + ticker: None, + started: Instant::now(), + live: self.live, + color: self.color, + quiet: self.quiet, + warnings: Arc::clone(&self.warnings), + }; + + if self.live { + eprintln!("{}", self.header(title, (!animate).then_some(label))); + } + if self.live && animate { + draw_spinner(&view.state.lock().expect("progress state"), self.color); + let state = Arc::clone(&view.state); + let stop = Arc::clone(&view.stop); + let color = self.color; + let ticker = std::thread::spawn(move || { + while !stop.load(Ordering::Relaxed) { + std::thread::sleep(TICK); + let mut state = state.lock().expect("progress state"); + if stop.load(Ordering::Relaxed) { + break; + } + state.frame = (state.frame + 1) % FRAMES.len(); + draw_spinner(&state, color); + } + }); + return StepView { + ticker: Some(ticker), + ..view + }; + } + + if !self.live { + eprintln!( + "{} {} {}", + self.counter(index), + style::paint(title, style::BOLD, self.color), + style::paint(&format!("({label})"), style::DIM, self.color), + ); + } + view + } + + /// `label` only for a step without a spinner, which has nowhere else to + /// say what is running. + fn header(&self, title: &str, label: Option<&str>) -> String { + let mut line = format!( + "{} {}", + style::paint("▸", style::ACCENT, self.color), + style::paint(title, style::BOLD, self.color), + ); + if let Some(label) = label { + line.push_str(&format!( + " {}", + style::paint(label, style::DIM, self.color) + )); + } + line + } + + fn counter(&self, index: usize) -> String { + style::paint( + &format!("[{}/{}]", index + 1, self.total), + style::DIM, + self.color, + ) + } +} + +struct State { + progress: String, + frame: usize, + spinning: bool, +} + +/// One running step's place on stderr. Shared with the thread reading the +/// step's stderr, so every write goes through the one lock. +pub struct StepView { + state: Arc>, + stop: Arc, + ticker: Option>, + started: Instant, + live: bool, + color: bool, + quiet: bool, + warnings: Arc>>, +} + +impl StepView { + /// A handle the stderr reader can own. + pub fn reporter(&self) -> Reporter { + Reporter { + state: Arc::clone(&self.state), + live: self.live, + color: self.color, + quiet: self.quiet, + warnings: Arc::clone(&self.warnings), + } + } + + /// Replaces the status line with how the step ended. Nothing in the plain + /// mode: the next step's line, or the error, already says so. + pub fn finish(mut self, succeeded: bool) { + self.stop.store(true, Ordering::Relaxed); + if let Some(ticker) = self.ticker.take() { + let _ = ticker.join(); + } + if !self.live { + return; + } + let state = self.state.lock().expect("progress state"); + if state.spinning { + clear_line(); + } + let took = elapsed(self.started.elapsed()); + let (mark, paint, text) = if succeeded { + ("✓", style::GREEN, took) + } else { + ("✗", style::RED, format!("failed after {took}")) + }; + // A blank line after, so each step reads as its own block. + eprintln!( + " {} {}\n", + style::paint(mark, paint, self.color), + style::paint(&text, style::DIM, self.color), + ); + let _ = std::io::stderr().flush(); + } +} + +/// What the thread reading a step's stderr holds: the same view, without the +/// right to finish it. +pub struct Reporter { + state: Arc>, + live: bool, + color: bool, + quiet: bool, + warnings: Arc>>, +} + +impl Reporter { + /// One line the step wrote to stderr. + pub fn report(&self, line: &str) { + let mut state = self.state.lock().expect("progress state"); + if let Some(progress) = line.strip_prefix(PROGRESS_PREFIX) { + if state.spinning { + state.progress = progress.trim().to_string(); + draw_spinner(&state, self.color); + } + return; + } + let shown = match line.strip_prefix(WARN_PREFIX) { + Some(warning) => { + let warning = warning.trim().to_string(); + self.warnings + .lock() + .expect("warnings") + .push(warning.clone()); + if self.live { + format!("{} {warning}", style::paint("!", style::YELLOW, self.color)) + } else { + format!("warning: {warning}") + } + } + None if self.quiet => return, + None => line.to_string(), + }; + if state.spinning { + clear_line(); + } + if self.live { + eprintln!("{DETAIL_INDENT}{shown}"); + } else { + eprintln!("{shown}"); + } + if state.spinning { + draw_spinner(&state, self.color); + } + } +} + +fn draw_spinner(state: &State, color: bool) { + let progress = if state.progress.is_empty() { + "working" + } else { + &state.progress + }; + eprint!( + "\r\x1b[2K {} {}", + style::paint(FRAMES[state.frame], style::ACCENT, color), + style::paint(progress, style::DIM, color), + ); + let _ = std::io::stderr().flush(); +} + +fn clear_line() { + eprint!("\r\x1b[2K"); +} + +fn elapsed(duration: Duration) -> String { + let seconds = duration.as_secs_f64(); + if seconds < 60.0 { + format!("{seconds:.1}s") + } else { + let whole = duration.as_secs(); + format!("{}m {:02}s", whole / 60, whole % 60) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_duration_reads_in_seconds_then_minutes() { + assert_eq!(elapsed(Duration::from_millis(2840)), "2.8s"); + assert_eq!(elapsed(Duration::from_secs(72)), "1m 12s"); + } +} diff --git a/src/workflow/prose.rs b/src/workflow/prose.rs new file mode 100644 index 0000000..de90dc2 --- /dev/null +++ b/src/workflow/prose.rs @@ -0,0 +1,237 @@ +//! Renders a workflow's `description` for `workflow show`. +//! +//! A deliberately small subset of Markdown, the parts a description needs +//! and nothing a terminal cannot show: paragraphs separated by a blank line, +//! lines indented by two or more spaces as commands, `- ` items as a list, +//! and `code` spans. Anything else is a paragraph, so no description can +//! fail to render — it can only render plainly. +//! +//! Paragraphs and list items are reflowed to [`WIDTH`], so an author's own +//! line breaks inside a YAML `|` block do not leave ragged lines. Commands +//! are kept exactly as written. + +use crate::output::style; + +/// Columns a paragraph is reflowed to. There is no terminal-size lookup in +/// this crate, and 80 columns is what every terminal shows. +pub const WIDTH: usize = 80; + +const COMMAND_INDENT: &str = " "; + +enum Block { + Paragraph(String), + Commands(Vec), + List(Vec), +} + +fn is_command(line: &str) -> bool { + line.starts_with(" ") +} + +fn list_item(line: &str) -> Option<&str> { + line.strip_prefix("- ").or_else(|| line.strip_prefix("* ")) +} + +fn blocks(text: &str) -> Vec { + let mut out: Vec = vec![]; + for chunk in text.split("\n\n") { + // Only blocks from this chunk may be continued: a blank line always + // ends a list or a run of commands. + let first = out.len(); + let mut paragraph: Vec<&str> = vec![]; + for line in chunk.lines().filter(|line| !line.trim().is_empty()) { + let open = out.len() > first && paragraph.is_empty(); + match out.last_mut() { + // An indented line under a list item continues the item. + Some(Block::List(items)) if open && is_command(line) => { + let last = items.last_mut().expect("a list has an item"); + last.push(' '); + last.push_str(line.trim()); + continue; + } + _ => {} + } + if let Some(item) = list_item(line) { + flush(&mut paragraph, &mut out); + let continues = out.len() > first; + match out.last_mut() { + Some(Block::List(items)) if continues => items.push(item.trim().to_string()), + _ => out.push(Block::List(vec![item.trim().to_string()])), + } + } else if is_command(line) { + flush(&mut paragraph, &mut out); + let continues = out.len() > first; + match out.last_mut() { + Some(Block::Commands(commands)) if continues => { + commands.push(line.trim().to_string()) + } + _ => out.push(Block::Commands(vec![line.trim().to_string()])), + } + } else { + paragraph.push(line.trim()); + } + } + flush(&mut paragraph, &mut out); + } + out +} + +fn flush(paragraph: &mut Vec<&str>, out: &mut Vec) { + if !paragraph.is_empty() { + out.push(Block::Paragraph(paragraph.join(" "))); + paragraph.clear(); + } +} + +/// Words laid into lines of at most `width` columns. A word longer than a +/// line gets a line to itself rather than being broken. +fn wrap(text: &str, width: usize) -> Vec { + let mut lines = vec![]; + let mut line = String::new(); + for word in text.split_whitespace() { + let fits = line.is_empty() || line.chars().count() + 1 + word.chars().count() <= width; + if !fits { + lines.push(std::mem::take(&mut line)); + } + if !line.is_empty() { + line.push(' '); + } + line.push_str(word); + } + if !line.is_empty() { + lines.push(line); + } + lines +} + +/// Paints the `code` spans in one line, backticks included. `inside` +/// carries an open span over to the next line, since wrapping may split one. +fn paint_spans(line: &str, inside: &mut bool, color: bool) -> String { + if !color { + return line.to_string(); + } + let mut out = String::new(); + if *inside { + out.push_str(style::ACCENT); + } + for c in line.chars() { + if c == '`' && !*inside { + out.push_str(style::ACCENT); + out.push(c); + } else if c == '`' { + out.push(c); + out.push_str(style::RESET); + } else { + out.push(c); + } + if c == '`' { + *inside = !*inside; + } + } + if *inside { + out.push_str(style::RESET); + } + out +} + +fn reflowed(text: &str, first: &str, rest: &str, color: bool) -> Vec { + let width = WIDTH.saturating_sub(first.chars().count()).max(20); + let mut inside = false; + wrap(text, width) + .iter() + .enumerate() + .map(|(index, line)| { + let lead = if index == 0 { first } else { rest }; + format!("{lead}{}", paint_spans(line, &mut inside, color)) + }) + .collect() +} + +pub fn render(text: &str, color: bool) -> String { + blocks(text) + .into_iter() + .map(|block| match block { + Block::Paragraph(text) => reflowed(&text, "", "", color).join("\n"), + Block::Commands(commands) => commands + .iter() + .map(|command| { + format!( + "{COMMAND_INDENT}{}", + style::paint(command, style::ACCENT, color) + ) + }) + .collect::>() + .join("\n"), + Block::List(items) => items + .iter() + .flat_map(|item| reflowed(item, " • ", " ", color)) + .collect::>() + .join("\n"), + }) + .collect::>() + .join("\n\n") +} + +#[cfg(test)] +mod tests { + use super::*; + + const DESCRIPTION: &str = "\ +Reads a style with one credential profile and creates a copy of it with +another. Log in to each account first: + + mapbox auth login --profile source + mapbox auth login --profile target + +Not copied: +- tilesets and fonts that belong to the source account, which the target + can read only if they are public +- the sprite +"; + + #[test] + fn paragraphs_reflow_commands_stay_and_lists_hang() { + assert_eq!( + render(DESCRIPTION, false), + "\ +Reads a style with one credential profile and creates a copy of it with another. +Log in to each account first: + + mapbox auth login --profile source + mapbox auth login --profile target + +Not copied: + + • tilesets and fonts that belong to the source account, which the target can + read only if they are public + • the sprite" + ); + } + + #[test] + fn a_command_right_under_a_paragraph_is_still_a_command() { + assert_eq!( + render("Log in first:\n mapbox auth login\nThen run it.", false), + "Log in first:\n\n mapbox auth login\n\nThen run it." + ); + } + + #[test] + fn color_changes_nothing_but_the_escapes() { + let colored = render("See `warnings` and `a b`.\n\n mapbox x", true); + assert!(colored.contains(style::ACCENT), "{colored:?}"); + assert_eq!( + style::strip(&colored), + render("See `warnings` and `a b`.\n\n mapbox x", false) + ); + } + + #[test] + fn a_span_split_by_wrapping_stays_painted_on_both_lines() { + let long = format!("{} `one two three`", "word ".repeat(15).trim_end()); + let colored = render(&long, true); + let lines: Vec<&str> = colored.lines().collect(); + assert_eq!(lines.len(), 2, "{colored:?}"); + assert_eq!(style::strip(&colored), render(&long, false)); + } +} diff --git a/src/workflow/runner.rs b/src/workflow/runner.rs new file mode 100644 index 0000000..6a6fc10 --- /dev/null +++ b/src/workflow/runner.rs @@ -0,0 +1,773 @@ +//! Runs a checked workflow, one step at a time. +//! +//! A `command` step is this binary run again as a child, with `--output json` +//! and the step's arguments turned into flags. That is the point rather than +//! a shortcut: the child resolves its token, applies its timeouts, encodes +//! its path segments and records its run exactly as a command typed by hand +//! would, so a workflow cannot reach the API any way a person could not. +//! +//! Only the child's stdout is captured, as the step's output. Its stderr and +//! stdin are the terminal's, so progress, warnings and a confirmation prompt +//! reach the person running the workflow — unless the step feeds `stdin` +//! itself. +//! +//! A `script` step runs its file from the installed workflow's `scripts/` +//! under the interpreter the definition settled on. It is given `MAPBOX_CLI`, +//! the path to this binary, so it can call commands of its own. + +use std::collections::BTreeMap; +use std::ffi::OsString; +use std::io::{BufRead, BufReader, Write}; +use std::path::Path; +use std::process::{Command as Process, Stdio}; + +use anyhow::{anyhow, Context as _, Result}; +use clap::{Arg, ArgAction, Command}; +use serde_json::{json, Map, Value}; + +use super::definition::{Action, InputType, Step, Workflow, DRY_RUN_ENV, SCRIPTS_DIR}; +use super::progress::{Display, StepView}; +use super::template::{self, Context}; +use super::workdir::{self, Workdir}; +use crate::auth; +use crate::output::{self, CliError}; + +/// Global options a step may not set, because the runner owns them or +/// because they would put a credential in a file. +const RESERVED: &[(&str, &str)] = &[ + ("output", "the runner reads every step as JSON"), + ("quiet", "the runner sets it"), + ( + "schema", + "a workflow runs commands rather than describing them", + ), + ( + "dry-run", + "use `mapbox workflow run --dry-run` for the whole workflow", + ), + ( + "token", + "a token written into a workflow is a secret in a file; use `profile`", + ), +]; + +/// The globals of the `workflow run` line that each command step inherits +/// unless it sets its own. +#[derive(Debug, Default, Clone)] +pub struct Inherited { + /// Only a token typed as `--token`; one from the environment reaches the + /// child by being in its environment already. + pub typed_token: Option, + pub options: Vec<(&'static str, String)>, + pub flags: Vec<&'static str>, + /// `--quiet`: steps show their title and how they ended, and warnings, + /// but not their details. + pub quiet: bool, +} + +impl Inherited { + pub fn from_matches(matches: &clap::ArgMatches) -> Self { + let mut inherited = Inherited { + typed_token: auth::typed_token(matches), + quiet: matches.get_flag(output::banner::ARG), + ..Default::default() + }; + for option in ["profile", "username", crate::http::TIMEOUT_ARG] { + let typed = + matches.value_source(option) == Some(clap::parser::ValueSource::CommandLine); + if let (true, Some(value)) = (typed, matches.get_one::(option)) { + inherited.options.push((option, value.clone())); + } + } + for flag in ["use-login", "debug", crate::confirm::ARG] { + if matches.get_flag(flag) { + inherited.flags.push(flag); + } + } + inherited + } +} + +/// The flag an input is given as: its name, dashed, as every other flag in +/// this CLI is spelled. `--style_id` is accepted too, as an alias. +pub fn input_flag(name: &str) -> String { + name.replace('_', "-") +} + +/// Turns the values given for a workflow's inputs, by input name, into its +/// inputs typed as the definition declares, with defaults filled in. The +/// parser has already checked each value's shape; this still does, so it +/// can be trusted on its own. +pub fn read_inputs(workflow: &Workflow, given: &[(String, String)]) -> Result> { + let mut inputs: Map = Map::new(); + for (key, raw) in given { + let flag = input_flag(key); + let Some(spec) = workflow.inputs.get(key) else { + return Err(invalid_input( + format!("`{}` has no input called `{key}`", workflow.name), + workflow, + )); + }; + let value = match spec.kind { + InputType::String => Value::String(raw.to_string()), + InputType::Number => parse_number(raw).ok_or_else(|| { + invalid_input(format!("`--{flag}` is a number, not `{raw}`"), workflow) + })?, + InputType::Boolean => match raw.as_str() { + "true" => Value::Bool(true), + "false" => Value::Bool(false), + _ => { + return Err(invalid_input( + format!("`--{flag}` is `true` or `false`, not `{raw}`"), + workflow, + )) + } + }, + }; + inputs.insert(key.to_string(), value); + } + + let mut missing = vec![]; + for (name, spec) in &workflow.inputs { + if inputs.contains_key(name) { + continue; + } + match &spec.default { + Some(default) => { + inputs.insert(name.clone(), default.clone()); + } + None if spec.required => missing.push(format!("--{}", input_flag(name))), + None => { + inputs.insert(name.clone(), Value::Null); + } + } + } + if !missing.is_empty() { + return Err(invalid_input( + format!("missing required input: {}", missing.join(" ")), + workflow, + )); + } + Ok(inputs) +} + +pub fn parse_number(raw: &str) -> Option { + raw.parse::() + .map(Value::from) + .ok() + .or_else(|| raw.parse::().ok().map(Value::from)) +} + +fn invalid_input(message: String, workflow: &Workflow) -> anyhow::Error { + CliError::new("invalid_input", message) + .with_remedy( + crate::remedy::Remedy::default() + .with_action(Some(format!("mapbox workflow show {}", workflow.name))), + ) + .into() +} + +/// What `--dry-run` prints: the plan, with the inputs resolved and every +/// later reference left as written, since nothing has run to fill it. +pub fn plan(workflow: &Workflow, inputs: &Map) -> Value { + let steps: Vec = workflow + .steps + .iter() + .map(|step| { + let (kind, args) = match &step.action { + Action::Command { args, .. } => ("command", Value::Object(args.clone())), + Action::Script { args, .. } => ("script", Value::Array(args.clone())), + }; + json!({ + "id": step.id, + "kind": kind, + "run": step.label(), + "args": args, + "stdin": step.stdin, + "dry_run": step.dry_run, + }) + }) + .collect(); + json!({ "workflow": workflow.name, "inputs": inputs, "steps": steps, "outputs": workflow.outputs }) +} + +/// What a run ends with. +pub struct Finished { + /// What `outputs` resolves to, or the last step's output when the + /// workflow declares none. + pub outputs: Value, + /// What `result` resolves to, when the workflow declares one. + pub text: Option, +} + +/// Runs every step. +pub fn run( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, +) -> Result { + let outputs = run_steps(app, workflow, root, inputs, inherited, false)?; + let context = Context { + inputs, + steps: &outputs, + }; + let text = match &workflow.result { + Some(template) => Some( + template::resolve(&Value::String(template.clone()), &context) + .map(|value| template::as_text(&value).unwrap_or_default()) + .map_err(|e| CliError::new("workflow_failed", format!("`result`: {e:#}")))?, + ), + None => None, + }; + + let last = workflow + .steps + .last() + .and_then(|step| outputs.get(&step.id)) + .cloned() + .unwrap_or(Value::Null); + let outputs = match &workflow.outputs { + Some(declared) => template::resolve(declared, &context) + .map_err(|e| CliError::new("workflow_failed", format!("`outputs`: {e:#}")))?, + None => last, + }; + Ok(Finished { outputs, text }) +} + +/// Runs only the steps marked `dry_run`, each with [`DRY_RUN_ENV`] set, and +/// returns their outputs by step id. `outputs` is not resolved: the steps it +/// reads may not have run. +pub fn dry_run( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, +) -> Result> { + run_steps(app, workflow, root, inputs, inherited, true) +} + +fn run_steps( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, + dry_run: bool, +) -> Result> { + let exe = std::env::current_exe().context("Could not find this program's own path")?; + let workdir = Workdir::create()?; + let mut outputs: BTreeMap = BTreeMap::new(); + let total = workflow.steps.len(); + + let display = Display::for_stderr(total, inherited.quiet); + for (index, step) in workflow.steps.iter().enumerate() { + let title = step.name.as_deref().unwrap_or(&step.id); + if dry_run && !step.dry_run { + display.skipped(index, title, &step.label()); + continue; + } + + let context = Context { + inputs, + steps: &outputs, + }; + let stdin = step + .stdin + .as_ref() + .map(|value| template::resolve(value, &context)) + .transpose() + .map_err(|e| step_failure(step, e))?; + + let mut process = match &step.action { + Action::Command { path, args } => { + let args = match template::resolve(&Value::Object(args.clone()), &context) + .map_err(|e| step_failure(step, e))? + { + Value::Object(args) => args, + _ => unreachable!("an object resolves to an object"), + }; + let argv = + command_argv(app, path, &args, inherited).map_err(|e| step_failure(step, e))?; + let mut process = Process::new(&exe); + process.args(argv); + process + } + Action::Script { + script, + interpreter, + args, + } => { + let args = template::resolve(&Value::Array(args.clone()), &context) + .map_err(|e| step_failure(step, e))?; + let mut process = Process::new(interpreter); + process.arg(root.join(SCRIPTS_DIR).join(script)); + for value in args.as_array().into_iter().flatten() { + process.arg(template::as_text(value).unwrap_or_default()); + } + process + .env("MAPBOX_CLI", &exe) + .env("MAPBOX_WORKFLOW_ROOT", root) + .env(workdir::ENV, workdir.path()); + if dry_run { + process.env(DRY_RUN_ENV, "1"); + } + process + } + }; + if let Some(token) = &inherited.typed_token { + process.env(auth::CLAP_TOKEN_ENV, token); + } + + let is_script = matches!(step.action, Action::Script { .. }); + let view = display.start(index, title, &step.label(), is_script); + let result = execute(&mut process, stdin.as_ref(), step, &view, workdir.path()); + view.finish(result.is_ok()); + outputs.insert(step.id.clone(), result?); + } + + display.finish_run(); + Ok(outputs) +} + +fn size(bytes: usize) -> String { + match bytes { + 0..=1023 => format!("{bytes} bytes"), + 1024..=1_048_575 => format!("{} KB", bytes / 1024), + _ => format!("{:.1} MB", bytes as f64 / 1_048_576.0), + } +} + +fn step_failure(step: &Step, err: anyhow::Error) -> anyhow::Error { + CliError::new( + "workflow_failed", + format!("Step `{}` ({}): {err:#}", step.id, step.label()), + ) + .into() +} + +/// Spawns one step and reads its stdout as its output: JSON when it parses, +/// the text otherwise, nothing when there is none. +fn execute( + process: &mut Process, + stdin: Option<&Value>, + step: &Step, + view: &StepView, + workdir: &Path, +) -> Result { + // A script with nothing to read gets nothing, rather than a terminal it + // might block on. A command keeps the terminal, which is where a + // confirmation prompt reads its answer. + let input = match (stdin, &step.action) { + (Some(_), _) => Stdio::piped(), + (None, Action::Script { .. }) => Stdio::null(), + (None, Action::Command { .. }) => Stdio::inherit(), + }; + // A script's stderr is read line by line, for its progress and details. + // A command's is the terminal's, so a confirmation prompt still shows. + let errors = match step.action { + Action::Script { .. } => Stdio::piped(), + Action::Command { .. } => Stdio::inherit(), + }; + let mut child = process + .stdin(input) + .stdout(Stdio::piped()) + .stderr(errors) + .spawn() + .map_err(|e| { + let program = process.get_program().to_string_lossy().into_owned(); + let message = if e.kind() == std::io::ErrorKind::NotFound { + format!("`{program}` is not installed or not on PATH") + } else { + format!("could not start `{program}`: {e}") + }; + step_failure(step, anyhow!(message)) + })?; + + // Written from a thread: a child that fills its stdout pipe before it + // has read all of stdin would otherwise wait on us while we wait on it. + let writer = stdin.map(|value| { + let body = match value { + Value::String(text) => text.clone(), + other => other.to_string(), + }; + let mut pipe = child.stdin.take().expect("stdin was piped"); + std::thread::spawn(move || { + // A child that exits without reading is reported by its status. + let _ = pipe.write_all(body.as_bytes()); + }) + }); + let reader = child.stderr.take().map(|pipe| { + let reporter = view.reporter(); + std::thread::spawn(move || { + for line in BufReader::new(pipe).lines().map_while(Result::ok) { + reporter.report(&line); + } + }) + }); + let finished = child + .wait_with_output() + .map_err(|e| step_failure(step, anyhow!("could not wait for it: {e}")))?; + if let Some(writer) = writer { + let _ = writer.join(); + } + // Every line the step wrote is shown before its outcome is. + if let Some(reader) = reader { + let _ = reader.join(); + } + + if !finished.status.success() { + let how = match finished.status.code() { + Some(code) => format!("exited with {code}"), + None => "was stopped by a signal".to_string(), + }; + return Err(step_failure( + step, + anyhow!("{how}; the workflow stopped here"), + )); + } + + if let Some(save) = &step.save { + let path = workdir.join(save); + std::fs::write(&path, &finished.stdout) + .map_err(|e| step_failure(step, anyhow!("could not save {}: {e}", path.display())))?; + let bytes = finished.stdout.len(); + view.reporter() + .report(&format!("Saved {save} ({})", size(bytes))); + return Ok(json!({ "path": path, "bytes": bytes })); + } + + let text = String::from_utf8(finished.stdout).map_err(|_| { + step_failure( + step, + anyhow!( + "its output is binary, which cannot be passed to another step; give it \ + `save: ` to keep it as a file instead" + ), + ) + })?; + let text = text.trim(); + if text.is_empty() { + return Ok(Value::Null); + } + Ok(serde_json::from_str(text).unwrap_or_else(|_| Value::String(text.to_string()))) +} + +enum ArgShape { + Flag, + Option { repeatable: bool }, + Positional { repeatable: bool }, +} + +/// The leaf command a step names, when it names one a workflow may run. +fn leaf<'a>(app: &'a Command, path: &[String]) -> Result<&'a Command, String> { + let named = format!("mapbox {}", path.join(" ")); + if path.first().map(String::as_str) == Some(super::COMMAND) { + return Err(format!("`{named}`: a workflow cannot run another workflow")); + } + let mut current = app; + for segment in path { + current = current + .find_subcommand(segment) + .ok_or_else(|| format!("`{named}` is not a command"))?; + } + if current.has_subcommands() { + return Err(format!("`{named}` is a group; name one of its commands")); + } + Ok(current) +} + +/// The argument a step's key names: one of the command's own, by its long +/// flag or, for a positional, its name — the same names `mapbox --schema` +/// lists — or a global option. +fn find_arg<'a>( + app: &'a Command, + command: &'a Command, + key: &str, +) -> Result<(&'a Arg, ArgShape), String> { + if let Some((_, why)) = RESERVED.iter().find(|(name, _)| *name == key) { + return Err(format!("`{key}` cannot be set by a step: {why}")); + } + let own = command + .get_arguments() + .find(|arg| arg.get_long() == Some(key) || (arg.is_positional() && arg.get_id() == key)); + let arg = own + .or_else(|| { + app.get_arguments() + .find(|arg| arg.is_global_set() && arg.get_long() == Some(key)) + }) + .ok_or_else(|| format!("`{key}` is not an argument of this command"))?; + let repeatable = matches!(arg.get_action(), ArgAction::Append); + let shape = if matches!(arg.get_action(), ArgAction::SetTrue | ArgAction::SetFalse) { + ArgShape::Flag + } else if arg.is_positional() { + ArgShape::Positional { repeatable } + } else { + ArgShape::Option { repeatable } + }; + Ok((arg, shape)) +} + +/// Every `command` step's problems against this binary's command tree — +/// checked at install and again before a run, since the CLI may have been +/// upgraded underneath an installed workflow. +pub fn command_problems(app: &Command, workflow: &Workflow) -> Vec { + let mut problems = vec![]; + + // An input is given as a flag beside the globals, so it cannot take one + // of their names — clap would refuse to build the command at all. + let taken: Vec<&str> = app + .get_arguments() + .filter(|arg| arg.is_global_set()) + .filter_map(Arg::get_long) + .chain([crate::executor::DRY_RUN_ARG, "help", "version"]) + .collect(); + for name in workflow.inputs.keys() { + let flag = input_flag(name); + if taken.contains(&flag.as_str()) { + problems.push(format!( + "input `{name}` would be given as `--{flag}`, which is already an option \ + of every command; rename it" + )); + } + } + for step in &workflow.steps { + let Action::Command { path, args } = &step.action else { + continue; + }; + let command = match leaf(app, path) { + Ok(command) => command, + Err(e) => { + problems.push(format!("step `{}`: {e}", step.id)); + continue; + } + }; + // A dry run sends every step it runs, so a command step may be in one + // only if its command changes nothing — which is exactly a command + // without a `--dry-run` of its own. + let writes = command + .get_arguments() + .any(|arg| arg.get_long() == Some(crate::executor::DRY_RUN_ARG)); + if step.dry_run && writes { + problems.push(format!( + "step `{}`: `dry_run` needs a command that changes nothing, and `mapbox {}` \ + changes something", + step.id, + path.join(" ") + )); + } + for (key, value) in args { + match find_arg(app, command, key) { + Err(e) => problems.push(format!("step `{}`: {e}", step.id)), + Ok((_, ArgShape::Flag)) if !(value.is_boolean() || value.is_string()) => problems + .push(format!( + "step `{}`: `{key}` is a flag; give it true or false", + step.id + )), + Ok(_) => {} + } + } + } + problems +} + +/// A step's arguments as the child's command line. +/// +/// Options are spelled `--name=value`, so a value that starts with a dash — +/// a western longitude — is never read as a flag. Positionals come last, +/// after `--`, for the same reason. +fn command_argv( + app: &Command, + path: &[String], + args: &Map, + inherited: &Inherited, +) -> Result> { + let command = leaf(app, path).map_err(|e| anyhow!(e))?; + let mut argv: Vec = path.iter().map(OsString::from).collect(); + let mut positionals: Vec<(usize, OsString)> = vec![]; + + for (key, value) in args { + let (arg, shape) = find_arg(app, command, key).map_err(|e| anyhow!(e))?; + let values: Vec<&Value> = match value { + Value::Array(items) if !matches!(shape, ArgShape::Flag) => items.iter().collect(), + Value::Null => continue, + other => vec![other], + }; + match shape { + ArgShape::Flag => match value { + Value::Bool(true) => argv.push(format!("--{key}").into()), + Value::Bool(false) => {} + Value::String(text) if text == "true" => argv.push(format!("--{key}").into()), + Value::String(text) if text == "false" => {} + _ => return Err(anyhow!("`{key}` is a flag; give it true or false")), + }, + ArgShape::Option { repeatable } | ArgShape::Positional { repeatable } + if values.len() > 1 && !repeatable => + { + return Err(anyhow!("`{key}` takes one value, not a list")); + } + ArgShape::Option { .. } => { + for value in values { + let text = template::as_text(value).unwrap_or_default(); + argv.push(format!("--{key}={text}").into()); + } + } + ArgShape::Positional { .. } => { + let index = arg.get_index().unwrap_or(usize::MAX); + for value in values { + positionals.push((index, template::as_text(value).unwrap_or_default().into())); + } + } + } + } + + for (option, value) in &inherited.options { + if !args.contains_key(*option) { + argv.push(format!("--{option}={value}").into()); + } + } + for flag in &inherited.flags { + if !args.contains_key(*flag) { + argv.push(format!("--{flag}").into()); + } + } + argv.push(format!("--{}={}", output::ARG, output::JSON).into()); + argv.push(format!("--{}", output::banner::ARG).into()); + + if !positionals.is_empty() { + positionals.sort_by_key(|(index, _)| *index); + argv.push("--".into()); + argv.extend(positionals.into_iter().map(|(_, value)| value)); + } + Ok(argv) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::workflow::definition::{parse, Files, DEFINITION_FILE}; + use std::path::PathBuf; + + fn app() -> Command { + let specs = crate::spec::effective_services().expect("bundled specs"); + crate::build_app(&specs) + } + + fn workflow(steps: &str) -> Workflow { + let yaml = format!( + "version: 1\nname: t\nsummary: t\ninputs:\n n: {{ type: number, default: 2 }}\n flag: {{ type: boolean, default: false }}\nsteps:\n{steps}" + ); + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.into_bytes())].into(); + parse("t", &files).expect("valid") + } + + /// A dry run sends every step it runs, so a command step can be in one + /// only if its command has no `--dry-run` of its own to hold back. + #[test] + fn only_a_command_that_changes_nothing_runs_in_a_dry_run() { + let found = command_problems( + &app(), + &workflow( + " - id: read\n command: styles get\n args: { style-id: x }\n dry_run: true\n\ + \x20 - id: write\n command: styles create\n dry_run: true\n", + ), + ) + .join("\n"); + assert!( + found.contains("step `write`: `dry_run` needs a command that changes nothing"), + "{found}" + ); + assert!(!found.contains("step `read`"), "{found}"); + } + + #[test] + fn a_step_is_checked_against_the_real_command_tree() { + let found = command_problems( + &app(), + &workflow( + " - id: a\n command: styles nope\n\ + \x20 - id: b\n command: styles\n\ + \x20 - id: c\n command: styles get\n args: { bogus: 1, token: x }\n\ + \x20 - id: d\n command: workflow list\n", + ), + ); + let joined = found.join("\n"); + assert!( + joined.contains("`mapbox styles nope` is not a command"), + "{joined}" + ); + assert!(joined.contains("is a group"), "{joined}"); + assert!(joined.contains("`bogus` is not an argument"), "{joined}"); + assert!(joined.contains("a secret in a file"), "{joined}"); + assert!(joined.contains("cannot run another workflow"), "{joined}"); + } + + #[test] + fn arguments_become_flags_and_positionals_come_last() { + let app = app(); + let args = serde_json::json!({ + "style-id": "-abc", + "optimize": true, + "download": false, + "profile": "work", + }); + let inherited = Inherited { + options: vec![("profile", "ignored".into()), ("timeout", "30".into())], + flags: vec!["yes"], + ..Default::default() + }; + let argv = command_argv( + &app, + &["styles".into(), "get".into()], + args.as_object().unwrap(), + &inherited, + ) + .unwrap(); + let argv: Vec = argv + .iter() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + argv, + [ + "styles", + "get", + "--optimize", + "--profile=work", + "--timeout=30", + "--yes", + "--output=json", + "--quiet", + "--", + "-abc" + ] + ); + } + + #[test] + fn inputs_are_typed_and_defaulted() { + let wf = workflow(" - id: a\n command: styles list\n"); + let given = |pairs: &[(&str, &str)]| -> Vec<(String, String)> { + pairs + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() + }; + let inputs = read_inputs(&wf, &given(&[("n", "5")])).unwrap(); + assert_eq!(inputs["n"], serde_json::json!(5)); + assert_eq!(inputs["flag"], serde_json::json!(false)); + assert!(read_inputs(&wf, &given(&[("n", "five")])).is_err()); + assert!(read_inputs(&wf, &given(&[("nope", "1")])).is_err()); + } + + #[test] + fn an_input_cannot_take_a_global_flag() { + let yaml = "version: 1\nname: t\nsummary: t\ninputs:\n profile: { type: string }\n dry_run: { type: boolean }\nsteps:\n - id: a\n command: styles list\n"; + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.as_bytes().to_vec())].into(); + let found = command_problems(&app(), &parse("t", &files).unwrap()).join("\n"); + assert!(found.contains("`--profile`"), "{found}"); + assert!(found.contains("`--dry-run`"), "{found}"); + } +} diff --git a/src/workflow/store.rs b/src/workflow/store.rs new file mode 100644 index 0000000..92ebe3a --- /dev/null +++ b/src/workflow/store.rs @@ -0,0 +1,610 @@ +//! Where installed workflows live, and how they get there. +//! +//! Nothing is bundled into the binary: a workflow runs only once it has been +//! installed, from a local directory or from a GitHub repository laid out as +//! `workflow//`, into `/workflows//`. A copy +//! is taken rather than a link kept, so what runs is what was checked at +//! install time and nothing edited since. +//! +//! The same rules as `agent-skills` hold for the write, for the same +//! reasons: everything is read and checked in memory first, an existing +//! workflow stops the install unless `--force`, only regular files are +//! taken, and a path that would leave the workflow's directory stops the +//! read altogether. The new directory is staged and renamed into place. + +use std::io::Read; +use std::path::{Component, Path, PathBuf}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{anyhow, Context, Result}; +use serde::{Deserialize, Serialize}; + +use super::definition::{self, is_contained, is_workflow_name, Files, Workflow}; +use crate::auth; +use crate::executor; +use crate::http; +use crate::output::CliError; +use crate::remedy::Remedy; + +const DIR_NAME: &str = "workflows"; + +/// Written beside an installed workflow's files, and what marks a directory +/// as one this command installed — `uninstall` removes nothing without it. +const META_FILE: &str = ".install.json"; + +/// Where the tarball comes from. The API rather than codeload directly, +/// because the API is what honors a token for a private repository; it +/// answers with a redirect to a signed codeload URL, and `reqwest` drops the +/// `Authorization` header when it follows a redirect to another host. +const GITHUB_API: &str = "https://api.github.com"; +/// This repository. Not `REPO_URL`'s `mapbox/cli`: on GitHub that name +/// redirects to a different repository. +pub const DEFAULT_REPO: &str = "mapbox/mapbox-cli"; +pub const DEFAULT_REF: &str = "main"; + +/// The directory inside a repository that holds its workflows, one +/// directory each. +pub const REPO_PREFIX: &str = "workflow"; + +/// A ceiling on what is read, from a tarball or a local directory. Far above +/// any real workflow, and there so a hostile archive cannot make this +/// allocate without bound. +const MAX_BYTES: u64 = 64 * 1024 * 1024; + +/// The files `read_local` passes over rather than refusing. Finder writes +/// `.DS_Store` into any directory it opens. +const IGNORED: &[&str] = &[".DS_Store"]; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Meta { + /// Where it came from, as a person would write it: a directory, or + /// `github:owner/repo`. + pub source: String, + #[serde(default, rename = "ref")] + pub git_ref: Option, + pub installed_at: u64, +} + +/// A workflow read and checked, not yet written anywhere. +pub struct Package { + pub workflow: Workflow, + pub files: Files, + pub meta: Meta, +} + +/// A workflow on disk. +pub struct Installed { + pub root: PathBuf, + pub workflow: Workflow, + pub meta: Meta, +} + +/// `/workflows`, resolved and not created. +pub fn root_path() -> Result { + auth::config_dir_path() + .map(|dir| dir.join(DIR_NAME)) + .ok_or_else(|| anyhow!("Could not determine home directory")) +} + +pub fn invalid_workflow(name: &str, problems: &[String]) -> anyhow::Error { + let listed = problems + .iter() + .map(|problem| format!(" - {problem}")) + .collect::>() + .join("\n"); + CliError::new( + "invalid_workflow", + format!("`{name}` is not a valid workflow:\n{listed}"), + ) + .into() +} + +/// A workflow name from the command line, checked before it is ever joined +/// to a path. +pub fn check_name(name: &str) -> Result<()> { + if is_workflow_name(name) { + return Ok(()); + } + Err(CliError::new( + "invalid_name", + format!("`{name}` is not a workflow name: lower-case letters, digits and dashes"), + ) + .with_remedy(Remedy::default().with_action(Some("mapbox workflow list".to_string()))) + .into()) +} + +fn now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|elapsed| elapsed.as_secs()) + .unwrap_or(0) +} + +/// Every regular file under `dir`, by path relative to it. +/// +/// A symlink is refused rather than followed: it is a way to install +/// something from outside the directory that was named. +fn read_tree(dir: &Path, skip: &[&str]) -> Result { + let mut files = Files::new(); + let mut total: u64 = 0; + let mut pending = vec![dir.to_path_buf()]; + while let Some(current) = pending.pop() { + let entries = std::fs::read_dir(¤t) + .with_context(|| format!("Could not read {}", current.display()))?; + for entry in entries { + let entry = entry.with_context(|| format!("Could not read {}", current.display()))?; + let path = entry.path(); + let relative = path.strip_prefix(dir).expect("under the directory walked"); + let name = entry.file_name(); + if current == dir && skip.iter().any(|s| name == *s) { + continue; + } + if IGNORED.iter().any(|ignored| name == *ignored) { + continue; + } + let kind = entry + .file_type() + .with_context(|| format!("Could not read {}", path.display()))?; + if kind.is_dir() { + pending.push(path); + } else if kind.is_file() { + let bytes = std::fs::read(&path) + .with_context(|| format!("Could not read {}", path.display()))?; + total += bytes.len() as u64; + if total > MAX_BYTES { + return Err(anyhow!( + "{} holds more than a workflow should", + dir.display() + )); + } + files.insert(relative.to_path_buf(), bytes); + } else { + return Err(CliError::new( + "invalid_workflow", + format!( + "{} is not a regular file; a workflow holds only files and directories", + path.display() + ), + ) + .into()); + } + } + } + Ok(files) +} + +fn package(name: &str, files: Files, meta: Meta) -> Result { + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Package { + workflow, + files, + meta, + }) +} + +/// A workflow from a directory on this machine. The directory's own name is +/// the workflow's, as it is in a repository. +pub fn read_local(dir: &Path) -> Result { + let dir = dir + .canonicalize() + .with_context(|| format!("There is no directory {}", dir.display()))?; + if !dir.is_dir() { + return Err(anyhow!("{} is not a directory", dir.display())); + } + let name = dir + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_default(); + let files = read_tree(&dir, &[])?; + package( + &name, + files, + Meta { + source: dir.display().to_string(), + git_ref: None, + installed_at: now(), + }, + ) +} + +/// `owner/repo`, and nothing that could move the request elsewhere. +pub fn check_repo(repo: &str) -> Result<()> { + let part = |p: &str| { + !p.is_empty() + && p != "." + && p != ".." + && p.chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.')) + }; + match repo.split_once('/') { + Some((owner, name)) if part(owner) && part(name) => Ok(()), + _ => Err(CliError::new( + "invalid_repo", + format!("`{repo}` is not a GitHub repository; write it as OWNER/REPO"), + ) + .into()), + } +} + +/// A branch, tag or commit. `/` is allowed, since a branch may hold one; +/// URL syntax and `..` are not, since the ref goes into a URL path. +pub fn check_ref(git_ref: &str) -> Result<()> { + let ok = !git_ref.is_empty() + && !git_ref + .split('/') + .any(|part| part.is_empty() || part == "." || part == "..") + && git_ref + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.' | '/')); + if ok { + return Ok(()); + } + Err(CliError::new( + "invalid_ref", + format!("`{git_ref}` is not a branch, tag or commit this can fetch"), + ) + .into()) +} + +fn github_token() -> Option { + ["GH_TOKEN", "GITHUB_TOKEN"] + .iter() + .filter_map(|name| std::env::var(name).ok()) + .find(|value| !value.trim().is_empty()) +} + +/// A workflow out of a GitHub repository's tarball. +pub fn fetch_github(name: &str, repo: &str, git_ref: &str, debug: bool) -> Result { + fetch_github_from(GITHUB_API, name, repo, git_ref, debug) +} + +fn fetch_github_from( + base: &str, + name: &str, + repo: &str, + git_ref: &str, + debug: bool, +) -> Result { + check_name(name)?; + check_repo(repo)?; + check_ref(git_ref)?; + + let url = format!("{base}/repos/{repo}/tarball/{git_ref}"); + if debug { + eprintln!("[debug] GET {url}"); + } + let token = github_token(); + let mut request = http::client()? + .get(&url) + .header("Accept", "application/vnd.github+json"); + if let Some(token) = &token { + request = request.bearer_auth(token); + } + let response = http::send(request) + .map_err(|e| executor::transport_failure("Could not reach GitHub", e))?; + + let status = response.status(); + if !status.is_success() { + // A private repository answers 404 to a request without a token, the + // same as a ref that does not exist, so both are named. + let message = match (status.as_u16(), &token) { + (404, None) => format!( + "GitHub found no `{git_ref}` in {repo}. If the repository is private, set \ + GITHUB_TOKEN to a token that can read it." + ), + (404, Some(_)) => format!( + "GitHub found no `{git_ref}` in {repo}, or the token in GH_TOKEN/GITHUB_TOKEN \ + cannot read it." + ), + _ => format!("GitHub answered {status} for {url}."), + }; + let mut remedy = Remedy::default(); + if token.is_none() { + remedy = remedy.with_fix("export GITHUB_TOKEN=\"$(gh auth token)\""); + } + return Err(CliError::http(status.as_u16(), &message) + .with_remedy(remedy) + .into()); + } + + let mut bytes = vec![]; + response + .take(MAX_BYTES) + .read_to_end(&mut bytes) + .map_err(|e| anyhow!("Could not read the archive from GitHub: {e}"))?; + let files = extract(&bytes, name, repo)?; + package( + name, + files, + Meta { + source: format!("github:{repo}"), + git_ref: Some(git_ref.to_string()), + installed_at: now(), + }, + ) +} + +fn unsafe_entry(path: &Path) -> anyhow::Error { + CliError::new( + "unsafe_archive", + format!( + "Refusing to read `{}` out of the archive: it points outside the directory it \ + would be written to.", + path.display() + ), + ) + .into() +} + +/// The files of `workflow//` in a repository tarball. +fn extract(archive: &[u8], name: &str, repo: &str) -> Result { + let decoder = flate2::read::GzDecoder::new(archive); + let mut tar = tar::Archive::new(decoder.take(MAX_BYTES)); + + let mut files = Files::new(); + let mut available: std::collections::BTreeSet = Default::default(); + for entry in tar + .entries() + .context("The archive from GitHub is not readable as a tarball")? + { + let mut entry = entry.context("The archive from GitHub ended unexpectedly")?; + // Only regular files: a link is a way out of the destination no path + // check would catch. + if entry.header().entry_type() != tar::EntryType::Regular { + continue; + } + let path = entry + .path() + .context("The archive holds an entry whose path is not valid UTF-8")? + .into_owned(); + // GitHub wraps every archive in one directory named after the + // repository and the commit, so it is stripped by position. + let mut components = path.components(); + let Some(Component::Normal(_)) = components.next() else { + return Err(unsafe_entry(&path)); + }; + let inner: PathBuf = components.collect(); + if !is_contained(&inner) { + return Err(unsafe_entry(&path)); + } + let Ok(within) = inner.strip_prefix(REPO_PREFIX) else { + continue; + }; + let mut parts = within.components(); + let Some(workflow) = parts.next() else { + continue; + }; + let relative: PathBuf = parts.collect(); + if relative.as_os_str().is_empty() { + continue; + } + if workflow.as_os_str() != name { + available.insert(workflow.as_os_str().to_string_lossy().into_owned()); + continue; + } + let mut bytes = vec![]; + entry + .read_to_end(&mut bytes) + .with_context(|| format!("Could not read {} out of the archive", path.display()))?; + files.insert(relative, bytes); + } + + if files.is_empty() { + let listed = if available.is_empty() { + format!("{repo} publishes no workflows under `{REPO_PREFIX}/`.") + } else { + format!( + "It publishes: {}.", + available.into_iter().collect::>().join(", ") + ) + }; + return Err(CliError::new( + "workflow_not_found", + format!("{repo} has no workflow called `{name}`. {listed}"), + ) + .into()); + } + Ok(files) +} + +/// Writes `package` into place, replacing an existing install only when +/// `force` says so. Returns where it went. +pub fn install(package: &Package, force: bool) -> Result { + let name = &package.workflow.name; + check_name(name)?; + let root = root_path()?; + std::fs::create_dir_all(&root) + .with_context(|| format!("Could not create {}", root.display()))?; + let target = root.join(name); + if target.exists() && !force { + return Err(already_installed(name, &target, None)); + } + + let staging = root.join(format!(".{name}.staging")); + let _ = std::fs::remove_dir_all(&staging); + let result = (|| -> Result<()> { + for (relative, bytes) in &package.files { + let path = staging.join(relative); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("Could not create {}", parent.display()))?; + } + std::fs::write(&path, bytes) + .with_context(|| format!("Could not write {}", path.display()))?; + } + std::fs::write( + staging.join(META_FILE), + serde_json::to_string_pretty(&package.meta)?, + ) + .context("Could not record where the workflow came from")?; + + if target.exists() { + std::fs::remove_dir_all(&target) + .with_context(|| format!("Could not replace {}", target.display()))?; + } + std::fs::rename(&staging, &target) + .with_context(|| format!("Could not move the workflow into {}", target.display())) + })(); + let _ = std::fs::remove_dir_all(&staging); + result.map(|()| target) +} + +/// `retry` is the `install` line that would replace it, when the caller +/// knows what was typed. +pub fn already_installed(name: &str, target: &Path, retry: Option) -> anyhow::Error { + CliError::new( + "already_installed", + format!("`{name}` is already installed at {}.", super::tilde(target)), + ) + .with_remedy( + Remedy::default() + .with_fix("Pass --force to replace it.") + .with_action(retry), + ) + .into() +} + +fn not_installed(name: &str) -> anyhow::Error { + CliError::new( + "workflow_not_installed", + format!("No workflow called `{name}` is installed."), + ) + .with_remedy( + Remedy::default() + .with_action(Some(format!("mapbox workflow install {name}"))) + .with_action(Some("mapbox workflow list".to_string())), + ) + .into() +} + +/// An installed workflow's directory, if there is one. Only a directory +/// holding [`META_FILE`] counts. +pub fn installed_dir(name: &str) -> Result> { + check_name(name)?; + let dir = root_path()?.join(name); + Ok(dir.join(META_FILE).is_file().then_some(dir)) +} + +/// Reads an installed workflow back and checks it again: a file edited by +/// hand since the install is refused here rather than failing mid-run. +pub fn load(name: &str) -> Result { + let root = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + load_dir(name, root) +} + +fn load_dir(name: &str, root: PathBuf) -> Result { + let meta: Meta = serde_json::from_slice( + &std::fs::read(root.join(META_FILE)) + .with_context(|| format!("Could not read {}", root.join(META_FILE).display()))?, + ) + .with_context(|| format!("{} is not readable", root.join(META_FILE).display()))?; + let files = read_tree(&root, &[META_FILE])?; + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Installed { + root, + workflow, + meta, + }) +} + +/// Every installed workflow, by name, with the ones that no longer load +/// kept as their error. +pub fn list() -> Result)>> { + let root = root_path()?; + let Ok(entries) = std::fs::read_dir(&root) else { + return Ok(vec![]); + }; + let mut out = vec![]; + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if !is_workflow_name(&name) || !entry.path().join(META_FILE).is_file() { + continue; + } + let loaded = load_dir(&name, entry.path()); + out.push((name, loaded)); + } + out.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(out) +} + +/// Removes an installed workflow. The name is checked and the directory +/// must carry [`META_FILE`], so nothing but a workflow this command put +/// there is ever removed. +pub fn uninstall(name: &str) -> Result { + let dir = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + std::fs::remove_dir_all(&dir).with_context(|| format!("Could not remove {}", dir.display()))?; + Ok(dir) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn archive(entries: &[(&str, &[u8])]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + vec![], + flate2::Compression::fast(), + )); + for (path, bytes) in entries { + let mut header = tar::Header::new_gnu(); + header.set_size(bytes.len() as u64); + header.set_mode(0o644); + header.set_entry_type(tar::EntryType::Regular); + builder.append_data(&mut header, path, *bytes).unwrap(); + } + builder.into_inner().unwrap().finish().unwrap() + } + + #[test] + fn only_the_named_workflow_comes_out() { + let bytes = archive(&[ + ("mapbox-cli-abc/README.md", b"x"), + ("mapbox-cli-abc/workflow/copy-style/workflow.yaml", b"a"), + ("mapbox-cli-abc/workflow/copy-style/scripts/p.py", b"b"), + ("mapbox-cli-abc/workflow/other/workflow.yaml", b"c"), + ]); + let files = extract(&bytes, "copy-style", "mapbox/cli").unwrap(); + assert_eq!( + files.keys().cloned().collect::>(), + [ + PathBuf::from("scripts/p.py"), + PathBuf::from("workflow.yaml") + ] + ); + } + + #[test] + fn a_missing_workflow_lists_what_is_there() { + let bytes = archive(&[("r-abc/workflow/other/workflow.yaml", b"c")]); + let err = extract(&bytes, "copy-style", "mapbox/cli").unwrap_err(); + assert!(err.to_string().contains("It publishes: other."), "{err}"); + } + + #[test] + fn a_file_directly_under_workflow_is_not_a_workflow() { + let bytes = archive(&[("r-abc/workflow/README.md", b"c")]); + assert!(extract(&bytes, "copy-style", "mapbox/cli").is_err()); + } + + #[test] + fn repos_and_refs_cannot_carry_url_syntax() { + assert!(check_repo("mapbox/cli").is_ok()); + for bad in [ + "mapbox", + "mapbox/cli/x", + "../x", + "a/b?c", + "a/b#c", + "/cli", + "a/..", + ] { + assert!(check_repo(bad).is_err(), "{bad}"); + } + for good in ["main", "v0.3.0", "feat/copy-style", "063415e"] { + assert!(check_ref(good).is_ok(), "{good}"); + } + for bad in ["", "a?b", "a#b", "../main", "a//b", "a b", "a%2e"] { + assert!(check_ref(bad).is_err(), "{bad}"); + } + } +} diff --git a/src/workflow/template.rs b/src/workflow/template.rs new file mode 100644 index 0000000..51ce29d --- /dev/null +++ b/src/workflow/template.rs @@ -0,0 +1,351 @@ +//! `${{ … }}` expressions: how one step's values reach the next. +//! +//! Deliberately a lookup language and nothing more — no operators, no +//! functions, no conditionals. An expression names a value: an input, or a +//! path into an earlier step's output. Anything that needs logic belongs in a +//! script step, which is a real language and is tested as one. +//! +//! A string that is exactly one expression takes the value's own JSON type, +//! so an object can be handed to `stdin` whole and a number stays a number. +//! An expression inside a longer string is interpolated as text. + +use std::collections::BTreeMap; + +use anyhow::{anyhow, Result}; +use serde_json::{Map, Value}; + +const OPEN: &str = "${{"; +const CLOSE: &str = "}}"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Segment { + Key(String), + Index(usize), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Reference { + Input(String), + Step { id: String, path: Vec }, +} + +/// What an expression can see while a workflow runs. +pub struct Context<'a> { + pub inputs: &'a Map, + /// The outputs of the steps that have finished, by id. + pub steps: &'a BTreeMap, +} + +/// Every reference in `value`, for checking before anything runs. +pub fn references(value: &Value) -> Result, String> { + let mut found = vec![]; + walk_strings(value, &mut |text| { + for piece in pieces(text)? { + if let Piece::Expression(reference) = piece { + found.push(reference); + } + } + Ok(()) + })?; + Ok(found) +} + +/// `value` with every expression replaced by what it names. +pub fn resolve(value: &Value, context: &Context) -> Result { + match value { + Value::String(text) => resolve_string(text, context), + Value::Array(items) => items + .iter() + .map(|item| resolve(item, context)) + .collect::>() + .map(Value::Array), + Value::Object(fields) => fields + .iter() + .map(|(key, item)| Ok((key.clone(), resolve(item, context)?))) + .collect::>() + .map(Value::Object), + other => Ok(other.clone()), + } +} + +/// How a resolved value is spelled where only text fits: an argv entry, an +/// interpolated string. +pub fn as_text(value: &Value) -> Option { + match value { + Value::Null => None, + Value::String(text) => Some(text.clone()), + Value::Bool(_) | Value::Number(_) => Some(value.to_string()), + Value::Array(_) | Value::Object(_) => Some(value.to_string()), + } +} + +fn resolve_string(text: &str, context: &Context) -> Result { + let pieces = pieces(text).map_err(|e| anyhow!(e))?; + if let [Piece::Expression(reference)] = pieces.as_slice() { + return lookup(reference, context); + } + let mut out = String::new(); + for piece in pieces { + match piece { + Piece::Text(literal) => out.push_str(&literal), + Piece::Expression(reference) => { + let value = lookup(&reference, context)?; + // Interpolating a null would write "null" or nothing into the + // middle of a URL or a name, and neither is what was meant. + let text = as_text(&value).ok_or_else(|| { + anyhow!( + "`{}` is empty, so it cannot be written into \"{text}\"", + display(&reference) + ) + })?; + out.push_str(&text); + } + } + } + Ok(Value::String(out)) +} + +fn lookup(reference: &Reference, context: &Context) -> Result { + match reference { + Reference::Input(name) => Ok(context.inputs.get(name).cloned().unwrap_or(Value::Null)), + Reference::Step { id, path } => { + let mut current = context + .steps + .get(id) + .ok_or_else(|| anyhow!("`{}`: step `{id}` has not run", display(reference)))?; + for segment in path { + let next = match segment { + Segment::Key(key) => current.get(key.as_str()), + Segment::Index(index) => current.get(*index), + }; + current = next.ok_or_else(|| { + anyhow!( + "`{}`: step `{id}`'s output has nothing at that path", + display(reference) + ) + })?; + } + Ok(current.clone()) + } + } +} + +/// An expression as the author wrote it, for error messages. +pub fn display(reference: &Reference) -> String { + match reference { + Reference::Input(name) => format!("inputs.{name}"), + Reference::Step { id, path } => { + let mut out = format!("steps.{id}.output"); + for segment in path { + match segment { + Segment::Key(key) => { + out.push('.'); + out.push_str(key); + } + Segment::Index(index) => out.push_str(&format!("[{index}]")), + } + } + out + } + } +} + +fn walk_strings( + value: &Value, + visit: &mut dyn FnMut(&str) -> Result<(), String>, +) -> Result<(), String> { + match value { + Value::String(text) => visit(text), + Value::Array(items) => items.iter().try_for_each(|item| walk_strings(item, visit)), + Value::Object(fields) => fields + .values() + .try_for_each(|item| walk_strings(item, visit)), + _ => Ok(()), + } +} + +#[derive(Debug, PartialEq)] +enum Piece { + Text(String), + Expression(Reference), +} + +fn pieces(text: &str) -> Result, String> { + let mut out = vec![]; + let mut rest = text; + while let Some(start) = rest.find(OPEN) { + if start > 0 { + out.push(Piece::Text(rest[..start].to_string())); + } + let after = &rest[start + OPEN.len()..]; + let end = after + .find(CLOSE) + .ok_or_else(|| format!("unclosed `{OPEN}` in \"{text}\""))?; + out.push(Piece::Expression(parse(after[..end].trim())?)); + rest = &after[end + CLOSE.len()..]; + } + if !rest.is_empty() { + out.push(Piece::Text(rest.to_string())); + } + Ok(out) +} + +/// `inputs.` or `steps..output` followed by `.key` and `[n]`. +fn parse(expression: &str) -> Result { + let invalid = || { + format!( + "`{expression}` is not an expression this understands. Use \ + `inputs.` or `steps..output`, optionally followed by \ + `.key` or `[index]`" + ) + }; + + let mut segments = vec![]; + let mut chars = expression.chars().peekable(); + let mut first = true; + while chars.peek().is_some() { + if !first { + match chars.next() { + Some('.') => {} + Some('[') => { + let digits: String = chars.by_ref().take_while(|c| *c != ']').collect(); + let index = digits.parse::().map_err(|_| invalid())?; + segments.push(Segment::Index(index)); + continue; + } + _ => return Err(invalid()), + } + } + first = false; + let mut key = String::new(); + while let Some(c) = chars.peek() { + if c.is_ascii_alphanumeric() || *c == '_' || *c == '-' { + key.push(*c); + chars.next(); + } else { + break; + } + } + if key.is_empty() { + return Err(invalid()); + } + segments.push(Segment::Key(key)); + } + + let mut segments = segments.into_iter(); + match (segments.next(), segments.next(), segments.next()) { + (Some(Segment::Key(root)), Some(Segment::Key(name)), None) if root == "inputs" => { + Ok(Reference::Input(name)) + } + (Some(Segment::Key(root)), Some(Segment::Key(id)), Some(Segment::Key(output))) + if root == "steps" && output == "output" => + { + Ok(Reference::Step { + id, + path: segments.collect(), + }) + } + _ => Err(invalid()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn context_resolve(value: Value) -> Result { + let inputs = json!({ "style_id": "abc", "zoom": 12, "empty": null }); + let steps = BTreeMap::from([( + "source".to_string(), + json!({ "owner": "alice", "layers": [{ "id": "water" }] }), + )]); + resolve( + &value, + &Context { + inputs: inputs.as_object().unwrap(), + steps: &steps, + }, + ) + } + + #[test] + fn a_whole_expression_keeps_its_type() { + assert_eq!( + context_resolve(json!("${{ inputs.zoom }}")).unwrap(), + json!(12) + ); + assert_eq!( + context_resolve(json!("${{ steps.source.output.layers[0] }}")).unwrap(), + json!({ "id": "water" }) + ); + } + + #[test] + fn an_embedded_expression_is_interpolated_as_text() { + assert_eq!( + context_resolve(json!( + "styles/${{ steps.source.output.owner }}/${{inputs.style_id}}" + )) + .unwrap(), + json!("styles/alice/abc") + ); + } + + #[test] + fn objects_and_arrays_are_resolved_inside() { + assert_eq!( + context_resolve(json!({ "a": ["${{ inputs.style_id }}", 1] })).unwrap(), + json!({ "a": ["abc", 1] }) + ); + } + + #[test] + fn an_empty_value_cannot_be_interpolated() { + assert!(context_resolve(json!("x-${{ inputs.empty }}")).is_err()); + assert_eq!( + context_resolve(json!("${{ inputs.empty }}")).unwrap(), + Value::Null + ); + } + + #[test] + fn a_missing_path_names_the_expression() { + let err = context_resolve(json!("${{ steps.source.output.nope }}")).unwrap_err(); + assert!( + err.to_string().contains("steps.source.output.nope"), + "{err}" + ); + } + + #[test] + fn references_are_found_for_checking() { + assert_eq!( + references(&json!({ "a": "${{ inputs.x }}", "b": ["${{ steps.s.output.k[2] }}"] })) + .unwrap(), + vec![ + Reference::Input("x".into()), + Reference::Step { + id: "s".into(), + path: vec![Segment::Key("k".into()), Segment::Index(2)] + } + ] + ); + } + + #[test] + fn anything_else_is_refused() { + for bad in [ + "${{ inputs }}", + "${{ inputs.a.b }}", + "${{ steps.s }}", + "${{ steps.s.result }}", + "${{ env.HOME }}", + "${{ inputs.a + 1 }}", + "${{ steps.s.output[x] }}", + "${{ inputs.a", + ] { + assert!(references(&json!(bad)).is_err(), "{bad} was accepted"); + } + } +} diff --git a/src/workflow/workdir.rs b/src/workflow/workdir.rs new file mode 100644 index 0000000..bd2b699 --- /dev/null +++ b/src/workflow/workdir.rs @@ -0,0 +1,39 @@ +//! The directory one workflow run keeps its files in. +//! +//! A command step's `save` writes there, and a script finds it through +//! [`ENV`]. It is created under the system's temporary directory with a +//! random name, and removed when the run ends, however it ends — so a run +//! leaves nothing behind, and nothing it removes was ever named by a caller. + +use std::path::{Path, PathBuf}; + +use anyhow::{Context as _, Result}; + +/// The run's directory, in a script step's environment. +pub const ENV: &str = "MAPBOX_WORKFLOW_WORKDIR"; + +pub struct Workdir { + path: PathBuf, +} + +impl Workdir { + pub fn create() -> Result { + let name = format!("mapbox-workflow-{:016x}", rand::random::()); + let path = std::env::temp_dir().join(name); + // `create_dir`, not `create_dir_all`: a name that already exists is + // somebody else's directory, and this one is about to be removed. + std::fs::create_dir(&path) + .with_context(|| format!("Could not create {}", path.display()))?; + Ok(Workdir { path }) + } + + pub fn path(&self) -> &Path { + &self.path + } +} + +impl Drop for Workdir { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.path); + } +} diff --git a/tests/source_guards.rs b/tests/source_guards.rs index eb6d4e6..a1e6d6b 100644 --- a/tests/source_guards.rs +++ b/tests/source_guards.rs @@ -48,6 +48,13 @@ fn sources() -> Vec<(String, String)> { /// - `generate_skills` — the staged skill directory it renames into place. /// - `skill_dest` — a test scratch directory. /// - `uninstall` — the binary itself, which is the whole command. +/// - `workflow/store.rs` — its staging directory, the installed workflow +/// `install --force` replaces, and `workflow uninstall`. The name is +/// checked to be one plain directory name first, and only a directory +/// holding its `.install.json` marker is removed. +/// - `workflow/workdir.rs` — the directory one workflow run keeps its files +/// in, which it created itself under the system temp directory with a +/// random name. No caller names it. const MAY_DELETE: &[&str] = &[ "agent_skills.rs", "auth.rs", @@ -56,6 +63,8 @@ const MAY_DELETE: &[&str] = &[ "generate_skills.rs", "skill_dest.rs", "uninstall.rs", + "workflow/store.rs", + "workflow/workdir.rs", ]; /// A new module that deletes files has to say so here first. @@ -238,7 +247,10 @@ const CARRIES_A_REQUEST_ID: &[&str] = &["account_usage.rs", "auth.rs", "executor /// `agent_skills.rs` talks to GitHub codeload, which identifies requests with /// `x-github-request-id`. That is not something Mapbox support can look up, /// so an id there would point at the wrong company — worse than none. -const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs"]; +/// +/// `workflow/store.rs` fetches a tarball from the GitHub API, for the same +/// reason. +const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs", "workflow/store.rs"]; /// `output/error.rs` defines `CliError::http` rather than calling it over a /// wire. diff --git a/tests/workflow.rs b/tests/workflow.rs new file mode 100644 index 0000000..eb63f4e --- /dev/null +++ b/tests/workflow.rs @@ -0,0 +1,644 @@ +//! End-to-end tests for `mapbox workflow`. +//! +//! The unit tests in `src/workflow/` cover the schema, the expressions and +//! how a step's arguments become a command line. What they cannot show is +//! what a real run does: that a command step is a real child `mapbox` whose +//! JSON reaches the next step, that a script's stdin and stdout carry values +//! between steps, that only the workflow's result lands on stdout, and what +//! `install` and `uninstall` leave on disk. +//! +//! Nothing here reaches the network. The command steps run `config list` and +//! `auth whoami`, which make no request, and every install is from a local +//! directory. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use serde_json::{json, Value}; + +/// A token-shaped fake for account `example-user`. +const TOKEN: &str = "pk.eyJ1IjoiZXhhbXBsZS11c2VyIiwiYSI6IngifQ.SIGNATURE"; + +fn scratch(name: &str) -> PathBuf { + let home = PathBuf::from(env!("CARGO_TARGET_TMPDIR")).join(format!("workflow-{name}")); + let _ = std::fs::remove_dir_all(&home); + std::fs::create_dir_all(&home).expect("create the scratch home"); + home +} + +fn run(home: &Path, args: &[&str]) -> Output { + run_with(home, args, &[]) +} + +/// [`run`], with `env` set on top of what it sets. +fn run_with(home: &Path, args: &[&str], env: &[(&str, &str)]) -> Output { + Command::new(env!("CARGO_BIN_EXE_mapbox")) + .env_remove("MAPBOX_ACCESS_TOKEN") + .env_remove("MapboxAccessToken") + .env_remove("MAPBOX_USERNAME") + .env_remove("MAPBOX_OUTPUT") + .env_remove("GH_TOKEN") + .env_remove("GITHUB_TOKEN") + .env("MAPBOX_NO_UPDATE_CHECK", "1") + .env("MAPBOX_QUIET", "1") + .env("HOME", home) + .env("XDG_CONFIG_HOME", home.join(".config")) + .env("MAPBOX_CONFIG_DIR", home.join(".mapbox")) + .envs(env.iter().copied()) + .args(args) + .output() + .expect("run mapbox") +} + +fn stdout_json(out: &Output) -> Value { + serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "stdout is not one JSON document ({e}):\n{}\nstderr:\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }) +} + +/// Writes a workflow at `/src//` and returns its path. +fn write_workflow(home: &Path, name: &str, yaml: &str, scripts: &[(&str, &str)]) -> PathBuf { + let dir = home.join("src").join(name); + std::fs::create_dir_all(dir.join("scripts")).unwrap(); + std::fs::write(dir.join("workflow.yaml"), yaml).unwrap(); + for (file, body) in scripts { + std::fs::write(dir.join("scripts").join(file), body).unwrap(); + } + dir +} + +const DEMO: &str = r#"version: 1 +name: demo +summary: Pass values from a command to a script and back +inputs: + greeting: { type: string, default: hello } +steps: + - id: settings + command: config list + - id: whoami + command: auth whoami + - id: shout + script: shout.sh + args: ["${{ inputs.greeting }}"] + stdin: ${{ steps.settings.output[0] }} +outputs: + account: ${{ steps.whoami.output.account }} + first_key: ${{ steps.settings.output[0].key }} + shouted: ${{ steps.shout.output.shouted }} + seen: ${{ steps.shout.output.seen }} +"#; + +/// Echoes its argument upper-cased beside what it read on stdin, and says +/// something on stderr, which must not reach the result. +const SHOUT: &str = r#"read -r line +echo "shouting" >&2 +printf '{"shouted":"%s","seen":%s}\n' "$(printf %s "$1" | tr a-z A-Z)" "$line" +"#; + +fn install_demo(home: &Path) { + let dir = write_workflow(home, "demo", DEMO, &[("shout.sh", SHOUT)]); + let out = run( + home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); +} + +#[test] +fn a_run_carries_values_between_steps_and_prints_only_the_result() { + let home = scratch("run"); + install_demo(&home); + + // Not quiet, so the script's stderr is shown as a detail. + let out = run_with( + &home, + &[ + "--token", + TOKEN, + "workflow", + "run", + "demo", + "--greeting", + "hi", + "-o", + "json", + ], + &[("MAPBOX_QUIET", "0")], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + + let first_key = json!("update-check"); + assert_eq!( + stdout_json(&out), + json!({ + "account": "example-user", + "first_key": first_key, + "shouted": "HI", + "seen": { "key": first_key, "value": true }, + }) + ); + + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("[1/3] settings (mapbox config list)"), + "{stderr}" + ); + assert!( + stderr.contains("[3/3] shout (sh scripts/shout.sh)"), + "{stderr}" + ); + assert!(stderr.contains("shouting"), "{stderr}"); + assert!(stderr.contains("not recommended for use"), "{stderr}"); +} + +#[test] +fn a_failing_step_stops_the_run() { + let home = scratch("fail"); + let yaml = "version: 1\nname: broken\nsummary: Fails in the middle\nsteps:\n\ + \x20 - id: first\n script: fail.sh\n\ + \x20 - id: second\n command: config list\n"; + let dir = write_workflow(&home, "broken", yaml, &[("fail.sh", "exit 3\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "broken", "-o", "json"]); + assert_eq!(out.status.code(), Some(1)); + assert!( + out.stdout.is_empty(), + "{}", + String::from_utf8_lossy(&out.stdout) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("exited with 3"), "{stderr}"); + assert!(!stderr.contains("[2/2]"), "the second step ran: {stderr}"); +} + +#[test] +fn dry_run_runs_nothing() { + let home = scratch("dry-run"); + let marker = home.join("ran"); + let yaml = "version: 1\nname: touch\nsummary: Leaves a file behind\n\ + inputs:\n path: { type: string, required: true }\nsteps:\n\ + \x20 - id: touch\n script: touch.sh\n args: ['${{ inputs.path }}']\n"; + let dir = write_workflow(&home, "touch", yaml, &[("touch.sh", "touch \"$1\"\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let path = marker.display().to_string(); + let out = run( + &home, + &[ + "workflow", + "run", + "touch", + "--path", + &path, + "--dry-run", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out)["steps"][0]["run"], "sh scripts/touch.sh"); + assert!(!marker.exists(), "--dry-run ran the step"); + + let out = run(&home, &["workflow", "run", "touch", "--path", &path]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(marker.exists(), "the real run did not run the step"); +} + +/// Steps marked `dry_run` run under `--dry-run`, told so through the +/// environment, and their output is in the plan; the rest do not run. +#[test] +fn a_dry_run_runs_only_the_steps_that_support_it() { + let home = scratch("dry-run-steps"); + let marker = home.join("wrote"); + let yaml = "version: 1\nname: rehearse\nsummary: Plans, writes, reports\n\ + inputs:\n path: { type: string, required: true }\nsteps:\n\ + \x20 - id: plan\n script: env.sh\n dry_run: true\n\ + \x20 - id: write\n script: touch.sh\n args: ['${{ inputs.path }}']\n\ + \x20 - id: report\n script: echo.sh\n dry_run: true\n\ + \x20 stdin: ${{ steps.plan.output }}\n"; + let dir = write_workflow( + &home, + "rehearse", + yaml, + &[ + ( + "env.sh", + "printf '{\"dry_run\":\"%s\"}\\n' \"$MAPBOX_WORKFLOW_DRY_RUN\"\n", + ), + ("touch.sh", "touch \"$1\"\n"), + ("echo.sh", "cat\n"), + ], + ); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let path = marker.display().to_string(); + let out = run( + &home, + &[ + "workflow", + "run", + "rehearse", + "--path", + &path, + "--dry-run", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let plan = stdout_json(&out); + assert_eq!(plan["results"]["plan"], json!({ "dry_run": "1" })); + assert_eq!(plan["results"]["report"], json!({ "dry_run": "1" })); + assert!(plan["results"].get("write").is_none(), "{plan}"); + assert_eq!(plan["steps"][1]["dry_run"], false); + assert!( + !marker.exists(), + "--dry-run ran a step that does not support it" + ); + + let out = run( + &home, + &["workflow", "run", "rehearse", "--path", &path, "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out), json!({ "dry_run": "" })); + assert!(marker.exists(), "the real run did not run every step"); +} + +const REPORTING: &str = "version: 1\nname: report\nsummary: Reports as it goes\n\ + inputs:\n who: { type: string, default: world }\nsteps:\n\ + \x20 - id: work\n script: work.sh\n\ + outputs:\n said: ${{ steps.work.output.said }}\n\ + result: 'Said ${{ steps.work.output.said }} to ${{ inputs.who }}.'\n"; + +const WORK: &str = "echo '::progress halfway' >&2\n\ + echo 'Did the thing' >&2\n\ + echo '::warn Something to know' >&2\n\ + echo '{\"said\":\"hello\"}'\n"; + +fn install_reporting(home: &Path) { + let dir = write_workflow(home, "report", REPORTING, &[("work.sh", WORK)]); + assert!(run(home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); +} + +/// Away from a terminal, a script's details and warnings are kept and its +/// progress lines dropped, with no escape codes around any of them. +#[test] +fn a_script_reports_details_and_warnings_and_drops_progress_off_a_terminal() { + let home = scratch("step-reports"); + install_reporting(&home); + + let out = run_with( + &home, + &["workflow", "run", "report", "-o", "json"], + &[("MAPBOX_QUIET", "0")], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("[1/1] work"), "{stderr}"); + assert!( + stderr.lines().any(|line| line == "Did the thing"), + "{stderr}" + ); + assert!( + stderr + .lines() + .any(|line| line == "warning: Something to know"), + "{stderr}" + ); + assert!( + stderr.contains("Warning\n ! Something to know"), + "a warning is listed again once the run ends: {stderr}" + ); + assert!(!stderr.contains("halfway"), "{stderr}"); + assert!(!stderr.contains('\x1b'), "{stderr}"); +} + +/// `--quiet` drops a step's details, but never its warnings. +#[test] +fn quiet_drops_details_but_keeps_warnings() { + let home = scratch("step-reports-quiet"); + install_reporting(&home); + + let out = run(&home, &["workflow", "run", "report", "-o", "json"]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(!stderr.contains("Did the thing"), "{stderr}"); + assert!(stderr.contains("Something to know"), "{stderr}"); +} + +/// `result` is how text mode shows the outcome; `-o json` still prints the +/// outputs, which is what a script reads. +#[test] +fn result_is_the_text_and_outputs_are_the_json() { + let home = scratch("result-text"); + install_reporting(&home); + + let text = run( + &home, + &[ + "workflow", "run", "report", "--who", "Helsinki", "-o", "text", + ], + ); + assert!( + text.status.success(), + "{}", + String::from_utf8_lossy(&text.stderr) + ); + assert_eq!( + String::from_utf8_lossy(&text.stdout).trim_end(), + "Said hello to Helsinki." + ); + + let json = run(&home, &["workflow", "run", "report", "-o", "json"]); + assert_eq!(stdout_json(&json), json!({ "said": "hello" })); +} + +/// `save` keeps a command step's stdout as a file in the run's working +/// directory, which a later script reads, and which is gone once the run ends. +#[test] +fn save_keeps_a_command_output_as_a_file_for_the_run() { + let home = scratch("save"); + // The path reaches the result through `outputs`, not through the script: + // on Windows it is full of backslashes, which a script writing its own + // JSON would leave unescaped. + let yaml = "version: 1\nname: keep\nsummary: Saves then reads\nsteps:\n\ + \x20 - id: settings\n command: config list\n save: settings.json\n\ + \x20 - id: read\n script: read.sh\n stdin: ${{ steps.settings.output.path }}\n\ + outputs:\n path: ${{ steps.settings.output.path }}\n saved: ${{ steps.read.output }}\n"; + let script = "read -r path\n\ + [ \"$(dirname \"$path\")\" = \"$MAPBOX_WORKFLOW_WORKDIR\" ] || exit 7\n\ + cat \"$path\"\n"; + let dir = write_workflow(&home, "keep", yaml, &[("read.sh", script)]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "keep", "-o", "json"]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let result = stdout_json(&out); + assert!( + result["saved"].is_array(), + "the saved file is the command's JSON: {result}" + ); + let path = PathBuf::from(result["path"].as_str().unwrap()); + assert!( + !path.parent().unwrap().exists(), + "the run's working directory outlived the run: {}", + path.display() + ); +} + +#[test] +fn install_refuses_an_invalid_workflow_and_writes_nothing() { + let home = scratch("invalid"); + let yaml = "version: 1\nname: bad\nsummary: Names a command that is not one\nsteps:\n\ + \x20 - id: a\n command: styles nope\n"; + let dir = write_workflow(&home, "bad", yaml, &[("unused.sh", "")]); + + let out = run( + &home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert_eq!(out.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("invalid_workflow"), "{stderr}"); + assert!(stderr.contains("scripts/unused.sh"), "{stderr}"); + assert!(!home.join(".mapbox/workflows/bad").exists()); +} + +#[test] +fn install_replaces_only_with_force_and_uninstall_removes() { + let home = scratch("lifecycle"); + install_demo(&home); + let installed = home.join(".mapbox/workflows/demo"); + assert!(installed.join("workflow.yaml").is_file()); + assert!(installed.join(".install.json").is_file()); + + let source = home.join("src/demo"); + let again = run(&home, &["workflow", "install", source.to_str().unwrap()]); + assert_eq!(again.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&again.stderr).contains("already installed")); + + let forced = run( + &home, + &["workflow", "install", source.to_str().unwrap(), "--force"], + ); + assert!( + forced.status.success(), + "{}", + String::from_utf8_lossy(&forced.stderr) + ); + + let listed = stdout_json(&run(&home, &["workflow", "list", "-o", "json"])); + assert_eq!(listed[0]["name"], "demo"); + + let removed = run(&home, &["workflow", "uninstall", "demo", "-o", "json"]); + assert!( + removed.status.success(), + "{}", + String::from_utf8_lossy(&removed.stderr) + ); + assert!(!installed.exists()); + + let missing = run(&home, &["workflow", "run", "demo"]); + assert_eq!(missing.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&missing.stderr).contains("mapbox workflow install demo")); +} + +#[test] +fn uninstall_removes_nothing_outside_the_installed_workflows() { + let home = scratch("escape"); + std::fs::create_dir_all(home.join("keep")).unwrap(); + for name in ["../keep", "/tmp", "..", "../.."] { + let out = run(&home, &["workflow", "uninstall", name]); + assert_eq!(out.status.code(), Some(1), "{name}"); + } + assert!(home.join("keep").exists()); +} + +#[test] +fn uninstall_takes_the_directory_install_was_given() { + let home = scratch("uninstall-path"); + install_demo(&home); + let source = home.join("src/demo"); + + // Read from the JSON error rather than matched in its text: JSON escapes a + // Windows path's backslashes, so the raw text never holds the path as typed. + let again = run( + &home, + &[ + "workflow", + "install", + source.to_str().unwrap(), + "-o", + "json", + ], + ); + let stderr = String::from_utf8_lossy(&again.stderr); + let error: Value = stderr + .lines() + .rev() + .find_map(|line| serde_json::from_str(line).ok()) + .unwrap_or_else(|| panic!("no JSON error on stderr:\n{stderr}")); + assert_eq!( + error["next_actions"][0], + json!(format!( + "mapbox workflow install {} --force", + source.display() + )), + "{stderr}" + ); + + let out = run(&home, &["workflow", "uninstall", source.to_str().unwrap()]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(!home.join(".mapbox/workflows/demo").exists()); + assert!( + source.join("workflow.yaml").is_file(), + "the source directory was touched" + ); +} + +#[test] +fn inputs_are_flags_like_any_other_command() { + let home = scratch("flags"); + install_demo(&home); + + let help = run(&home, &["workflow", "run", "demo", "--help"]); + let text = String::from_utf8_lossy(&help.stdout); + assert!( + help.status.success(), + "{}", + String::from_utf8_lossy(&help.stderr) + ); + assert!(text.contains("--greeting "), "{text}"); + assert!(text.contains("[default: hello]"), "{text}"); + + // Clap's own usage error, exit 2, as for a typo on any command. + let typo = run(&home, &["workflow", "run", "demo", "--greting", "hi"]); + assert_eq!(typo.status.code(), Some(2)); + assert!(String::from_utf8_lossy(&typo.stderr).contains("--greeting")); + + // Globals still work after the workflow's name. + let out = run( + &home, + &[ + "workflow", + "run", + "demo", + "--greeting=yo", + "--token", + TOKEN, + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out)["shouted"], "YO"); +} + +#[test] +fn a_missing_required_input_names_its_flag() { + let home = scratch("required"); + let yaml = "version: 1\nname: needs\nsummary: Needs an input\n\ + inputs:\n style_id: { type: string, required: true }\nsteps:\n\ + \x20 - id: a\n command: config list\n"; + let dir = write_workflow(&home, "needs", yaml, &[]); + std::fs::remove_dir(dir.join("scripts")).unwrap(); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "needs"]); + assert_eq!(out.status.code(), Some(2)); + assert!(String::from_utf8_lossy(&out.stderr).contains("--style-id ")); + + // The underscore spelling is accepted as well. + let out = run( + &home, + &["workflow", "run", "needs", "--style_id", "x", "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); +} + +#[test] +fn a_workflow_that_is_not_installed_says_how_to_install_it() { + let home = scratch("not-installed"); + let out = run(&home, &["workflow", "run", "copy-style", "--style-id", "x"]); + assert_eq!(out.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&out.stderr).contains("mapbox workflow install copy-style")); +} + +#[test] +fn history_records_the_command_and_not_the_workflow_name() { + let home = scratch("history"); + install_demo(&home); + assert!(run(&home, &["workflow", "run", "demo", "--token", TOKEN]) + .status + .success()); + + let listed = stdout_json(&run(&home, &["history", "list", "-o", "json"])); + assert_eq!(listed[0]["command"], json!(["workflow", "run"]), "{listed}"); + assert!(!listed.to_string().contains("demo"), "{listed}"); +} diff --git a/workflow/README.md b/workflow/README.md new file mode 100644 index 0000000..e721123 --- /dev/null +++ b/workflow/README.md @@ -0,0 +1,165 @@ +# Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts. None ships inside the binary: `mapbox workflow install` copies one in, and `mapbox workflow run` runs only what is installed. See [docs/commands.md](../docs/commands.md#workflows) for the commands. + +> **Beta and in development. Not recommended for use.** The `workflow` command, the format below (`version: 1`) and the published workflows may change or be removed without notice. Every `mapbox workflow` subcommand says so on stderr. + +## Layout + +``` +workflow/ + README.md # this file + copy-style/ # the workflow's name + workflow.yaml # required + scripts/ # the scripts its steps run + copy_style.py + README.md # optional +``` + +- The directory name is the workflow's name: lower-case letters, digits and dashes, and the same as `name` in `workflow.yaml`. +- A workflow holds `workflow.yaml`, `README.md` and `scripts/`, and nothing else. +- Every file in `scripts/` is run by some step. A workflow ships only the scripts it runs. + +The same rules apply to any workflow `install` reads, from this repository, another one (`--repo OWNER/REPO`, laid out the same way) or a local directory. `every_published_workflow_is_valid` in `src/workflow/mod.rs` holds this directory to them in `cargo test`. + +## `workflow.yaml` + +```yaml +version: 1 # required; the schema version +name: copy-style # required; the directory's name +summary: Copy a style from one account to another # required; one line +description: | # optional; shown by `workflow show` + Longer text. + +inputs: # optional; each one is a flag of `run` + style_id: + type: string # string | number | boolean + required: true + description: ID of the style to copy + name: + type: string # not required and no default: null + zoom: + type: number + default: 12 + +steps: # required; run in order, first failure stops + - id: source # required; lower-case, digits, underscores + name: Read the style # optional; the progress line + command: styles get # a mapbox command + args: # names as `mapbox --schema` lists them + style-id: ${{ inputs.style_id }} + profile: work # global options too + use-login: true # a flag takes true or false + dry_run: true # optional; also runs under --dry-run (a command that changes nothing) + + - id: bundle + command: styles download + args: { style-id: "${{ inputs.style_id }}", profile: work, use-login: true } + save: style.zip # optional; keep stdout as a file, for a binary response + + - id: body + script: prepare.py # a file in scripts/ + interpreter: python3 # optional for .sh, .py and .js + dry_run: true # optional; runs under --dry-run too, told so in its environment + args: ["--zoom", "${{ inputs.zoom }}"] + stdin: # optional; sent to the step as JSON + style: ${{ steps.source.output }} + +outputs: # optional; the default is the last step's output + id: ${{ steps.source.output.id }} + +result: | # optional; what text mode prints instead of the outputs + Created ${{ steps.source.output.id }}. +``` + +A step is a `command` or a `script`, never both. + +### Inputs as flags + +`mapbox workflow run ` takes each input as a flag, spelled with dashes: `style_id` is `--style-id`, and `--style_id` works too. A required input is a required flag, a number must parse as one, and a boolean is a flag that means true on its own (`--overwrite` or `--overwrite false`). `mapbox workflow run --help` lists them. + +An input cannot be named after an option every command already has, such as `profile`, `output` or `dry_run`. `install` refuses a workflow that does. + +### `description` + +`workflow show` renders it in a fixed layout, after the name and summary and before the inputs and steps. It reads a small subset of Markdown, so every workflow's page looks the same: + +- A blank line separates paragraphs. A paragraph is reflowed to 80 columns, so line breaks inside one do not matter. +- A line indented by two or more spaces is a command. It is highlighted and kept exactly as written. +- A line starting with `- ` is a list item. An indented line under an item continues the item. +- `code` spans are highlighted. + +Anything else is shown as a plain paragraph. There is no other formatting: no headings, links or emphasis. + +### Expressions + +`${{ inputs. }}` and `${{ steps..output }}`, followed by any number of `.key` and `[index]`. Nothing else: no operators and no functions. Logic belongs in a script. + +- A value that is exactly one expression keeps its JSON type, so a whole object can go to `stdin` and a number stays a number. +- An expression inside a longer string is written in as text. One that is null there is an error rather than an empty string. +- An expression may name only a declared input or an earlier step. That is checked before any step runs. + +### Command steps + +Each command step runs this `mapbox` binary again, with `--output json`, and its JSON result becomes the step's output. It resolves its token, timeouts and path encoding exactly as the same command typed by hand would, and appears in `mapbox history` on its own. + +- `args` keys are the argument names `mapbox --schema ` lists: a flag's long name, or a positional's name. Global options such as `profile`, `username` and `use-login` are accepted too. +- `output`, `quiet`, `schema` and `dry-run` belong to the runner. `token` is refused, because a token written into a workflow is a secret in a file: log in under a profile and name the `profile` instead. +- Global options given to `mapbox workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, `--yes`, `--debug`, `--token`) reach every command step that does not set its own. +- `stdin` is sent to the command, for `--data @-`. Without it, the command reads the terminal, which is where a confirmation prompt gets its answer. +- A step cannot run `mapbox workflow`. +- `save: ` keeps the command's stdout as that file in the run's working directory instead of reading it as the output, which is how a binary response, such as `styles download`'s ZIP, reaches a later step. The step's output is then `{"path": …, "bytes": …}`. The directory is removed when the run ends, however it ends. + +### Script steps + +A script runs from the installed copy of `scripts/`, in the directory `mapbox workflow run` was started from. + +- It gets `args` as its arguments and `stdin` on standard input, as JSON unless the value is a string. With no `stdin` it reads nothing. +- Whatever it writes to stdout is its output: JSON when it parses as JSON, the text otherwise. It reports to the person running it on stderr, as below. +- A non-zero exit stops the workflow. +- `MAPBOX_CLI` is the path to this `mapbox` binary, for a script that runs commands of its own. `MAPBOX_WORKFLOW_ROOT` is the installed workflow's directory. `MAPBOX_WORKFLOW_WORKDIR` is the run's working directory, where a `save`d file is; anything a script writes there is removed with it when the run ends. +- Prefer a command step where one will do, so `workflow show` and `install` can see the command. A script is for what a step cannot do: a loop, or logic. +- The interpreter must be installed on the machine. `.sh` runs under `sh`, `.py` under `python3` and `.js` under `node`, and any other extension needs `interpreter`. + +### Reporting progress + +The runner shows every step the same way, so a script only says what it is doing and what it did: + +| A stderr line | Is | Shown | +| --- | --- | --- | +| `::progress uploading icons 150/561` | What the step is doing now | At a terminal, beside the spinner, until the next one replaces it. Dropped anywhere else. | +| `::warn Source uses a tileset that is not copied` | Something the person should act on | Under the step (`! …` at a terminal, `warning: …` elsewhere), and listed again after the last step. Also under `--quiet`. | +| anything else, such as `Uploaded font Yellow Banana Regular` | A detail: what the step did | Under the step. Not under `--quiet`. | + +At a terminal a step looks like this, and the last line becomes `✓