diff --git a/.agents/skills/cosa-basics/SKILL.md b/.agents/skills/cosa-basics/SKILL.md new file mode 100644 index 0000000000..a18c79b207 --- /dev/null +++ b/.agents/skills/cosa-basics/SKILL.md @@ -0,0 +1,185 @@ +--- +name: cosa-basics +description: Initialize a COSA workdir, build or fetch Fedora CoreOS images, and launch local QEMU VMs using coreos-assembler +--- + +# COSA Basics + +This skill guides you through the core coreos-assembler (COSA) workflow: +initializing a build directory, obtaining a CoreOS build (either from source +or by fetching a published build), and launching a local QEMU VM. + +## Out of scope + +Cloud platform image building and uploading (AWS, GCP, Azure, etc.) are not +covered here. Load the **cosa-platforms** skill for those workflows. + +## Key documentation references + +Do NOT duplicate these docs. Read them at runtime for command details: + +| File | What it covers | +|------|----------------| +| `docs/building-fcos.md` | Full build walkthrough, prerequisites, the `cosa()` bash alias | +| `docs/cosa.md` | Command reference overview (all cosa subcommands) | +| `docs/cosa/run.md` | Detailed `cosa run` and QEMU VM options | +| `docs/working.md` | Overrides, customization, advanced usage | +| `docs/devel.md` | Running cosa locally outside a container | +| `src/cmd-init` | `cosa init` source -- all supported flags | +| `src/cmd-buildfetch` | `cosa buildfetch` source -- all supported flags | + +## Running COSA + +### Container mode (default) + +COSA is designed to run inside a container. The container image is: + +``` +quay.io/coreos-assembler/coreos-assembler +``` + +Before running any cosa commands, read `docs/building-fcos.md` and extract the +`cosa()` bash alias function defined there. This alias wraps `podman run` with +the necessary privileges, device mounts (`/dev/kvm`, `/dev/fuse`), user +namespace mapping, and volume mounts. The current working directory is mounted +as `/srv/` inside the container. + +Set up the alias in the user's shell before running any cosa commands. + +### Local mode (for cosa developers) + +Developers who have cosa installed locally can run commands directly without +the container wrapper. Read `docs/devel.md` for details on this workflow. The +commands themselves are the same; only the execution environment differs. + +## Workflow + +### Step 1: Set up a COSA workdir + +1. Ask the user for a working directory path. Default to `./fcos` if they + don't specify one. +2. If the directory does not exist, create it. +3. `cd` into the directory. +4. Set up the `cosa()` bash alias (read it from `docs/building-fcos.md`). +5. Initialize the workdir: + +``` +cosa init https://github.com/coreos/fedora-coreos-config +``` + +The user can override the config repo URL. Additional useful flags (read +`src/cmd-init` for the full list): + +- `--branch BRANCH` -- use a specific branch of the config repo +- `--variant VARIANT` -- select a build variant +- `--force` -- allow init in a non-empty directory + +After init, the directory structure will contain `src/config/` (the cloned +config repo), plus `builds/`, `cache/`, `tmp/`, and `overrides/` directories +that are populated during builds. + +### Step 2: Obtain a build + +Choose one of the following paths based on what the user wants to do. + +#### Option A: Build from source + +This builds a fresh CoreOS image from the config repo and RPM sources: + +``` +cosa build +cosa osbuild qemu +``` + +`cosa build` creates the bootable OCI container (ostree commit). `cosa osbuild +qemu` derives a QEMU disk image from that container. Both are required before +`cosa run` will work. + +Note: `cosa fetch` is a no-op in the current build flow. Package fetching +happens automatically inside `cosa build` via buildah. + +#### Option B: Fetch an existing published build + +This downloads a previously published build from the Fedora CoreOS build +servers. Use this when you want to test or inspect an existing release without +building from source. + +**Default flags:** Include `--artifact qemu --decompress` when the user +needs a local QEMU disk image (e.g. for `cosa run` or QEMU-based kola +tests). The published artifacts are stored compressed on the server; +`--decompress` expands them so the resulting qcow2 can be used directly. + +If the user only needs to run tests on a cloud platform where images already +exist (e.g. AMIs in AWS), `--artifact` and `--decompress` are not needed. +A plain `cosa buildfetch -b ` fetches only a few small metadata +files (including `meta.json`), which contain cloud image references that +kola picks up automatically. See the **cosa-platforms** skill for details. + +**Fetch the latest build for a stream:** + +``` +cosa buildfetch --stream --artifact qemu --decompress +``` + +Available streams: `stable`, `testing`, `testing-devel`, `next`, `rawhide` + +If the user does not specify a stream, default to `stable`. + +**Fetch a specific build by ID:** + +``` +cosa buildfetch -b --artifact qemu --decompress +``` + +When `-b`/`--build` is given, `--stream` is not required -- the stream is +auto-detected from the build ID's version number (the third dotted component +maps to a stream, e.g. `.2.` = `testing`, `.3.` = `stable`). Read +`src/cmd-buildfetch` for the full version-to-stream mapping. If `--stream` +is also provided it must match or the command will error. + +**If the user wants a non-qemu artifact:** replace `--artifact qemu` with the +appropriate artifact name (e.g., `metal`, `aws`, `gcp`). But note that cloud +platform workflows are better served by the **cosa-platforms** skill. + +### Step 3: Launch the VM + +``` +cosa run +``` + +This launches a QEMU VM using the latest build image with `-snapshot` (changes +are discarded on exit). By default it establishes an SSH connection with +auto-login as the `core` user. + +**Common options** (read `docs/cosa/run.md` for the full list): + +| Flag | Purpose | +|------|---------| +| `-c` / `--devshell-console` | Use serial console instead of SSH (see full boot, GRUB menu) | +| `--qemu-image PATH` | Use a specific qcow2 image instead of the latest build | +| `-B` / `--butane PATH` | Pass a Butane config for the VM | +| `-i` / `--ignition PATH` | Pass an Ignition config for the VM | +| `--add-disk SIZE[:OPTS]` | Attach additional disks | +| `--kargs 'ARGS'` | Append kernel arguments | +| `-m SIZE` | Set VM memory in MB | + +**Exiting the VM:** + +- SSH mode: `exit` or `Ctrl-D` +- Console mode: `Ctrl-a x` + +## Important notes + +- Each `cosa` command (via the alias) creates a transient container. Use + `cosa shell` for a persistent interactive session inside the container. +- `/dev/kvm` is required for both building disk images and running VMs. The + host must be bare metal or have nested virtualization enabled. +- The build directory at `$PWD` is mounted at `/srv/` inside the container. + Build artifacts persist across container invocations. +- The workdir's `builds/latest` symlink always points to the most recent build. +- To run kola tests against a build, load the **cosa-kola** skill. It covers + test discovery, `cosa kola run`, upgrade tests, and result inspection. +- Fedora infrastructure services (like Bodhi or Koji) are behind Anubis to + prevent abuse. When querying APIs or web pages for information set a + User-Agent and Accept headers to increase the chance of making it + through unchallenged. diff --git a/.agents/skills/cosa-dev/SKILL.md b/.agents/skills/cosa-dev/SKILL.md new file mode 100644 index 0000000000..6eb0f61e41 --- /dev/null +++ b/.agents/skills/cosa-dev/SKILL.md @@ -0,0 +1,233 @@ +--- +name: cosa-dev +description: Develop and test changes to coreos-assembler itself -- build mantle Go binaries (kola, ore, plume), iterate on Python/shell scripts, or do full container rebuilds +--- + +# Developing coreos-assembler + +This skill covers workflows for making and testing changes to +coreos-assembler (COSA) itself, including the Go binaries in `mantle/`, +the Python/shell scripts in `src/`, and the container image. + +## Related skills + +- Load the **cosa-basics** skill for the `cosa()` bash alias, workdir + setup, and general build workflow. +- Load the **cosa-kola** skill for running kola tests. +- Load the **cosa-platforms** skill for cloud platform testing. + +## Key documentation references + +Do NOT duplicate these docs. Read them at runtime for command details: + +| File | What it covers | +|------|----------------| +| `docs/devel.md` | Developer workflows: hacking scripts, building Go binaries, container rebuilds | +| `docs/building-fcos.md` | The `cosa()` bash alias definition | +| `Makefile` | Build targets: `mantle`, `kola`, `ore`, `plume`, `kolet`, `install` | +| `mantle/build` | Mantle build script (called by `make kola` etc.) | +| `build.sh` | Container image build steps | +| `Dockerfile` | Container image definition | + +## Choosing an approach + +| What changed | Approach | Rebuild time | +|---|---|---| +| Go code in `mantle/` only | Build single binary, mount over container copy | ~2 min cold, seconds warm | +| Python/shell in `src/` only | Set `COREOS_ASSEMBLER_GIT` | Instant (no rebuild) | +| Both Go and Python/shell | Combine both approaches | ~2 min cold, seconds warm | +| Dockerfile, RPM deps, or everything | Full `podman build` | Several minutes | + +## Approach 1: Modifying mantle Go binaries (kola, ore, plume) + +Use this when changing Go code under `mantle/`. The idea is to build the +binary inside the cosa container (which has the Go toolchain), write it +to a host directory, then mount it over the container's installed copy on +subsequent runs. + +### Buildable binaries + +| Source path | Binary name | Installed at | +|---|---|---| +| `mantle/cmd/kola` | `kola` | `/usr/bin/kola` | +| `mantle/cmd/ore` | `ore` | `/usr/bin/ore` | +| `mantle/cmd/plume` | `plume` | `/usr/bin/plume` | +| `mantle/cmd/kolet` | `kolet` | `/usr/lib/kola//kolet` | + +### Step 1: Create output and cache directories + +``` +mkdir -p /tmp/cosa-go-cache /tmp/cosa-bin +chmod 777 /tmp/cosa-go-cache /tmp/cosa-bin +``` + +The `chmod 777` is needed because the cosa container runs as uid 1000 +(`builder`) via `--userns=keep-id`, and the directories must be writable +by that mapped user. + +### Step 2: Build the binary + +Mount the cosa source tree, the output directory, and optionally a +persistent Go build cache into the container: + +``` +export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS="\ + -v=/path/to/coreos-assembler:/srv/cosa-src:ro \ + -v=/tmp/cosa-bin:/srv/cosa-bin:rw \ + -v=/tmp/cosa-go-cache:/home/builder/.cache/go-build:rw" + +cosa shell -- bash -c \ + 'cd /srv/cosa-src && go build -buildvcs=false -o /srv/cosa-bin/kola ./mantle/cmd/kola' +``` + +Replace `kola` with `ore` or `plume` as needed. To build all mantle +binaries at once: + +``` +cosa shell -- bash -c \ + 'cd /srv/cosa-src && \ + for cmd in kola ore plume; do \ + go build -buildvcs=false -o /srv/cosa-bin/$cmd ./mantle/cmd/$cmd; \ + done' +``` + +**Key flags:** + +- `-buildvcs=false` -- required to avoid git ownership errors inside the + container (the source tree is mounted from the host with a different uid). +- The persistent Go build cache (`/home/builder/.cache/go-build`) makes + subsequent rebuilds incremental. The first build downloads and compiles + all dependencies (~2 minutes); rebuilds after a small code change take + only a few seconds. + +### Step 3: Quick compilation check (no binary output) + +To verify that the code compiles without producing a binary: + +``` +cosa shell -- bash -c \ + 'cd /srv/cosa-src && go build -buildvcs=false ./mantle/...' +``` + +### Step 4: Test with the modified binary + +Mount the built binary over the container's installed copy using +`COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS`: + +``` +export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS="\ + -v=/tmp/cosa-bin/kola:/usr/bin/kola:ro" + +cosa kola run -p qemu basic +``` + +This can be combined with other mounts. For example, to also mount +AWS credentials: + +``` +export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS="\ + -v=/tmp/cosa-bin/kola:/usr/bin/kola:ro \ + -v=/path/to/aws-creds:/srv/aws-creds:ro" + +cosa kola run -p aws --aws-credentials-file /srv/aws-creds basic +``` + +### Step 5: Iterate + +After making further code changes, re-run the build command from Step 2. +With the persistent Go cache, only the changed packages are recompiled. +Then re-run the test from Step 4 -- the mount picks up the new binary +automatically. + +## Approach 2: Modifying Python/shell scripts (src/) + +Use this when changing files under `src/` (e.g. `src/cmd-build`, +`src/cmd-init`, `src/cosalib/*.py`). No rebuild is needed. + +Set the `COREOS_ASSEMBLER_GIT` environment variable to point at your +local coreos-assembler checkout. The `cosa()` bash alias will +automatically mount `$COREOS_ASSEMBLER_GIT/src/` over +`/usr/lib/coreos-assembler/` inside the container: + +``` +export COREOS_ASSEMBLER_GIT=/path/to/coreos-assembler +cosa init https://github.com/coreos/fedora-coreos-config +cosa build +``` + +Changes to files under `src/` take effect immediately on the next `cosa` +invocation. No rebuild or remount is required. + +**Limitation:** This only covers files installed to +`/usr/lib/coreos-assembler/` (i.e. `src/` contents). It does NOT affect +Go binaries (`kola`, `ore`, `plume`) or system packages. For Go changes, +combine this with Approach 1. + +### Combining with Go binary changes + +``` +export COREOS_ASSEMBLER_GIT=/path/to/coreos-assembler +export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS="\ + -v=/tmp/cosa-bin/kola:/usr/bin/kola:ro" + +cosa kola run basic +``` + +This uses the modified Python/shell scripts from `COREOS_ASSEMBLER_GIT` +AND the modified kola binary from the mount. + +## Approach 3: Full container rebuild + +Use this when changing the `Dockerfile`, adding or updating RPM +dependencies, or when you want to verify everything works together as a +complete image. + +### Build the container image + +From the coreos-assembler repo root: + +``` +podman build -t localhost/coreos-assembler . +``` + +To speed things up by reusing the official image as a base (useful when +only cosa code changed, not dependencies): + +``` +podman build -t localhost/coreos-assembler . \ + --from quay.io/coreos-assembler/coreos-assembler:latest +``` + +### Use the locally-built container + +Set `COREOS_ASSEMBLER_CONTAINER` so the `cosa()` alias uses your local +image instead of the upstream one: + +``` +export COREOS_ASSEMBLER_CONTAINER=localhost/coreos-assembler +cosa init https://github.com/coreos/fedora-coreos-config +cosa build +``` + +This is the slowest approach but the most complete. It is needed when: + +- The `Dockerfile` or `build.sh` changed. +- System RPM dependencies were added or updated. +- You want a final integration check before submitting a PR. + +## Important notes + +- The `cosa()` bash alias creates a **transient container** for each + invocation. All state persists in the cosa workdir (mounted at + `/srv/`), not inside the container. This is why mounting files in via + `COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS` works -- each run gets the + latest version of whatever is mounted. +- Use `cosa shell` to get a persistent interactive session inside the + container. This is useful for running multiple build/test commands + without container startup overhead. However, be aware that mounted + binaries are fixed at container start time. +- The Go build cache directory inside the container is + `/home/builder/.cache/go-build`. Persisting it to the host avoids + cold rebuilds. +- When mounting binaries with `:ro`, the container cannot modify them. + This is intentional -- it prevents accidental overwrites. diff --git a/.agents/skills/cosa-kola/SKILL.md b/.agents/skills/cosa-kola/SKILL.md new file mode 100644 index 0000000000..23e96c1dd3 --- /dev/null +++ b/.agents/skills/cosa-kola/SKILL.md @@ -0,0 +1,122 @@ +--- +name: cosa-kola +description: Run kola tests against CoreOS builds using coreos-assembler, including native and external tests +--- + +# Running Tests with cosa kola + +This skill guides you through running kola tests against CoreOS builds using +coreos-assembler. + +## Out of scope + +Writing new native (Go) kola tests and cloud platform testing are not covered +here. For writing external tests, refer to the docs listed below. + +## Key documentation references + +Do NOT duplicate these docs. Read them at runtime for command details: + +| File | What it covers | +|------|----------------| +| `docs/kola.md` | Kola overview, `cosa kola` wrapper behavior, automatic test discovery, common flags, denylists, test output logs | +| `docs/kola/external-tests.md` | How to write and structure external tests, `kola.json` metadata, reboot support | +| `docs/kola/adding-tests.md` | Adding native Go tests to kola (for reference) | +| `src/cmd-kola` | `cosa kola` wrapper source -- shows how args are passed through to kola | + +## Prerequisites + +Before running tests you need a QEMU image in the cosa workdir and the +`cosa()` bash alias must be set up. Load the **cosa-basics** skill for both +of these -- it covers container invocation, building from source, and fetching +published builds. + +## Workflow + +All commands below assume you are in a cosa workdir (or inside the cosa +container). Read `docs/kola.md` for full flag documentation. + +### Step 1: Find the right test(s) + +List available tests to find the one the user wants: + +``` +cosa kola list +``` + +For JSON output with descriptions: + +``` +cosa kola list --json | jq -r '.[] | [.Name, .Description] | @tsv' +``` + +External tests from `src/config/tests/kola/` are discovered automatically +and named `ext.config.` (e.g., `ext.config.files.aleph-version`). + +When you need to inspect or understand a test whose name starts with +`ext.config.`, find its source under `src/config/tests/kola/` in the cosa +workdir. The path maps directly from the test name: replace dots with `/` +after `ext.config.` and look for `test.sh` (and optionally `kola.json` or +`config.bu`) in that directory. For example: + +| Test name | Source path | +|-----------|-------------| +| `ext.config.files.aleph-version` | `src/config/tests/kola/files/aleph-version/test.sh` | +| `ext.config.upgrade.extended` | `src/config/tests/kola/upgrade/extended/test.sh` | + +The `src/config/` directory is the cloned config repo (e.g., +`fedora-coreos-config`). If it does not exist in the current directory, +check for it relative to the cosa workdir root. + +**Important:** The cosa workdir (created by `cosa init`) is NOT the same as +the coreos-assembler source repository. The `src/config/` directory only +exists inside a cosa workdir. Do not search the coreos-assembler source tree +for `ext.config.*` test files -- they won't be there. If you need to inspect +these test sources and are not currently in a cosa workdir, either navigate to +one or clone the config repo (e.g., `fedora-coreos-config`) directly. + +### Step 2: Run the test(s) + +Test names are glob patterns: + +``` +cosa kola run basic +cosa kola run ext.config.files.aleph-version +cosa kola run 'ext.config.*' +``` + +Running `cosa kola run` with no pattern runs all tests (can take a long time). + +Read `docs/kola.md` for flags like `--parallel`, `--tag`, `--rerun`, +`--denylist-test`, `--ssh-on-test-failure`, `--append-butane`, and others. + +For running tests on platforms other than QEMU (e.g. AWS, GCP, Azure, ISO, +metal), load the **cosa-platforms** skill. + +### Step 3: Check results + +Test logs are written to `tmp/kola//`. Key files: `journal.txt`, +`console.txt`, `ignition.json`, `journal-raw.txt.gz`. Read `docs/kola.md` +("More information on tests") for details. + +## Special test types + +### ISO install tests + +ISO install tests are regular kola tests (the `testiso` subcommand was removed +in https://github.com/coreos/coreos-assembler/pull/4377). Run them with +`cosa kola run` like any other test: + +``` +cosa kola run iso.iso-offline-install.mpath.bios +cosa kola run 'iso.*' +``` + +### Upgrade tests + +``` +cosa kola --upgrades +``` + +This runs the `run-upgrade` subcommand, which tests OS upgrades. It +automatically finds parent images and caches them in `tmp/kola-qemu-cache/`. diff --git a/.agents/skills/cosa-platforms/SKILL.md b/.agents/skills/cosa-platforms/SKILL.md new file mode 100644 index 0000000000..121352e252 --- /dev/null +++ b/.agents/skills/cosa-platforms/SKILL.md @@ -0,0 +1,163 @@ +--- +name: cosa-platforms +description: Build disk images, upload to cloud providers, and run tests or spawn instances on non-QEMU platforms (AWS, GCP, Azure, ISO, metal, etc.) +--- + +# Platform Images and Cloud Testing with cosa + +This skill covers working with non-QEMU platforms in coreos-assembler: +building platform-specific disk images, uploading them to cloud providers, +running kola tests on cloud platforms, and spawning interactive cloud +instances. + +## Related skills + +- Load the **cosa-basics** skill for workdir setup, the `cosa()` bash alias, + and general build workflow. +- Load the **cosa-kola** skill for general kola test mechanics (test + discovery, result inspection, common flags, external tests). +- This skill adds the platform-specific details on top of those. + +## Key documentation references + +Do NOT duplicate these docs. Read them at runtime for command details: + +| File | What it covers | +|------|----------------| +| `docs/mantle/credentials.md` | Credential setup for all platforms (AWS, Azure, GCP, OpenStack, DO, ESX, etc.) | +| `docs/cosa.md` | Command reference including `osbuild` and `imageupload-*` | +| `docs/cosa/imageupload-aws.md` | AWS AMI upload (IAM permissions, credentials) | +| `docs/cosa/osbuild-secex.md` | IBM Secure Execution image building | +| `docs/cosa/run.md` | `cosa run` (QEMU-only) and `kola spawn` for cloud instances | +| `docs/kola.md` ("Run tests on cloud platforms") | Cloud testing examples for AWS, GCP, Azure | +| `mantle/cmd/kola/options.go` | Source of truth for all platform-specific flags | + +## Supported platforms + +Kola supports the following platforms via the `-p`/`--platform` flag: + +| Platform | `-p` value | Image flag | Default credentials | +|----------|-----------|------------|-------------------| +| AWS | `aws` | `--aws-ami` | `~/.aws/credentials` | +| Azure | `azure` | `--azure-disk-uri` | `~/.azure/azureCreds.json` | +| DigitalOcean | `do` | `--do-image` | `~/.config/digitalocean.json` | +| ESX/VMware | `esx` | `--esx-base-vm` | `~/.config/esx.json` | +| GCP | `gcp` | `--gcp-image` | GCP service account JSON key | +| OpenStack | `openstack` | `--openstack-image` | `~/.config/openstack.json` | +| QEMU | `qemu` | `--qemu-image` | (none) | +| QEMU ISO | `qemu-iso` | `--qemu-iso` | (none) | + +Read `docs/mantle/credentials.md` for full credential setup details per +platform, and `mantle/cmd/kola/options.go` or `cosa kola run -h` for the +complete flag list. + +## Workflow + +### Step 1: Build the platform image + +``` +cosa osbuild +``` + +Use `cosa osbuild --supported-platforms` to list all available platforms. +Multiple platforms can be built in one invocation: + +``` +cosa osbuild aws azure gcp +``` + +This creates disk images in the appropriate format for each platform (e.g. +VMDK for AWS, VHD for Azure, tar.gz for GCP, raw for metal). + +### Step 2: Upload to the cloud provider + +For cloud platforms that require image upload: + +``` +cosa imageupload- +``` + +Available upload targets: `aliyun`, `aws`, `azure`, `gcp`, `powervs`. + +Read `docs/cosa/imageupload-aws.md` for AWS-specific IAM and credential +setup. Not all platforms have an `imageupload-*` command; for those that +don't, use `ore` directly or upload manually. + +### Step 3: Set up credentials + +Read `docs/mantle/credentials.md` for per-platform credential configuration. +Each platform has its own credential file format and default location (see +the table above). + +### Step 4: Run tests on a cloud platform + +``` +cosa kola run -p [platform-specific flags] +``` + +Examples: + +``` +# AWS +cosa kola run -p aws --aws-ami --aws-region us-east-1 basic + +# GCP +cosa kola run -p gcp --gcp-image --gcp-json-key --gcp-project basic + +# Azure +cosa kola run -p azure --azure-credentials --azure-disk-uri basic +``` + +Read `docs/kola.md` ("Run tests on cloud platforms") for more examples. +For the full list of flags per platform, read `mantle/cmd/kola/options.go` +or run `cosa kola run -h`. + +### Step 5: Spawn instances for interactive debugging + +``` +cosa kola spawn -p [platform-specific flags] +``` + +This launches an instance and opens an SSH shell without running tests. +Useful for interactive debugging. Key flags: + +| Flag | Purpose | +|------|---------| +| `--shell` / `-s` | Open an SSH shell (default: true) | +| `--detach` / `-t` | Spawn in the background (implies `--shell=false --remove=false --verbose --keys`) | +| `--remove=false` | Keep the instance running after the shell exits | +| `--keys` / `-k` | Inject SSH keys from the agent and `~/.ssh/` | +| `--nodecount` / `-c` | Number of instances to spawn | +| `--userdata` / `-u` | Path to Ignition/Butane config for the instance | +| `--ssh-command` / `-x` | Run a command over SSH instead of opening a shell | + +Note: `--reconnect` is only available on QEMU platforms. + +Read `docs/cosa/run.md` ("Running on cloud platforms") for more details. + +## Important notes + +- `cosa run` is QEMU-only. For all other platforms, use + `cosa kola run -p ` (for tests) or + `cosa kola spawn -p ` (for interactive instances). +- When a cosa build exists in the workdir, kola auto-fills platform image + references from the build metadata (e.g. AMI for AWS, image name for GCP). + This means you may not need to specify `--aws-ami` etc. if the build + already has that info in `meta.json`. Check `builds///meta.json` + to see if the cloud image info is present. +- If images already exist in the cloud for a build, you can skip both + `cosa osbuild` and `cosa imageupload-*` entirely. Just fetch the build + metadata with a lightweight `cosa buildfetch` (no `--artifact` or + `--decompress` needed): + ``` + cosa buildfetch -b + ``` + Without `--artifact`, this only downloads a few small metadata files + (`meta.json`, `commitmeta.json`, etc.) -- not any disk images. The + `meta.json` contains the cloud image references (AMI IDs, GCP image + names, etc.) and kola will pick them up automatically when you run + `cosa kola run -p `. +- Instance types default to reasonable values per platform and architecture + (e.g. `m5.large` for AWS x86_64, `c6g.xlarge` for aarch64). Override with + platform-specific type flags (e.g. `--aws-type`, `--gcp-machinetype`, + `--azure-size`). diff --git a/docs/cosa.md b/docs/cosa.md index 27b8e9ead4..008df0e4e6 100644 --- a/docs/cosa.md +++ b/docs/cosa.md @@ -38,6 +38,19 @@ various platforms, such as cloud providers. | `osbuild --supported-platforms` | List of all supported platforms +## Image upload commands + +Upload disk images to cloud providers. Build the platform image first with +`cosa osbuild `. + +| Name | Description | +| ---- | ----------- | +| `imageupload-aliyun` | Upload an image to Alibaba Cloud (Aliyun) | +| `imageupload-aws` | Upload an image to AWS as an AMI | +| `imageupload-azure` | Upload an image to Azure | +| `imageupload-gcp` | Upload an image to Google Cloud Platform | +| `imageupload-powervs` | Upload an image to IBM Power Virtual Server | + ## Misc commands Those less commonly used commands are listed here: diff --git a/docs/cosa/buildextend-aws.md b/docs/cosa/imageupload-aws.md similarity index 90% rename from docs/cosa/buildextend-aws.md rename to docs/cosa/imageupload-aws.md index f9ac3d94b4..5364aad59a 100644 --- a/docs/cosa/buildextend-aws.md +++ b/docs/cosa/imageupload-aws.md @@ -3,9 +3,10 @@ parent: CoreOS Assembler Command Line Reference nav_order: 1 --- -# cosa buildextend-aws +# cosa imageupload-aws -Using `cosa buildextend-aws` looks for AWS credentials in the standard places; +`cosa imageupload-aws` uploads a disk image to AWS as an AMI. It looks for +AWS credentials in the standard places; Common methods are `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables, and using `AWS_PROFILE`. For more information, consult the documentation for the `aws` CLI tool, in particular @@ -86,5 +87,7 @@ The full list of permisions required for your IAM policy should look similar to } ``` -This is currently a wrapper around the `ore` subcommand of mantle; there is an +Before uploading, build the AWS disk image with `cosa osbuild aws`. + +This command is a wrapper around the `ore` subcommand of mantle; there is an `ore aws initialize` subcommand which may be useful. diff --git a/docs/cosa/buildextend-secex.md b/docs/cosa/osbuild-secex.md similarity index 90% rename from docs/cosa/buildextend-secex.md rename to docs/cosa/osbuild-secex.md index 5eaa6a63a0..70025f4eb8 100644 --- a/docs/cosa/buildextend-secex.md +++ b/docs/cosa/osbuild-secex.md @@ -3,9 +3,9 @@ parent: CoreOS Assembler Command Line Reference nav_order: 1 --- -# cosa buildextend-secex +# cosa osbuild qemu-secex -This buildextend command is used to build QEMU images that are enabled for IBM Secure Execution on IBM Z. +This command builds QEMU images that are enabled for IBM Secure Execution on IBM Z. In order to build a QEMU image protected by IBM Secure Execution, you need to provide a host key to encrypt it. For more information on IBM Secure Execution on IBM Z, refer to the [IBM Documentation](https://www.ibm.com/docs/en/linux-on-systems?topic=ibmz-secure-execution). @@ -18,7 +18,7 @@ This results in a few specifics to note: - The firstboot service will fail when no host key is provided, as the sdboot-image can not be recreated. - Write the host key(s) to: `/etc/se-hostkeys/ibm-z-hostkey-.crt` -To facilitate this, `buildextend-secex` can take 2 mutually exclusive additional arguments: `--genprotimgvm ` and `--hostkey `. +To facilitate this, `cosa osbuild qemu-secex` can take 2 mutually exclusive additional arguments: `--genprotimgvm ` and `--hostkey `. If none is provided, `--genprotimgvm` is used with default values. ## `--genprotimgvm ` (default) diff --git a/docs/cosa/run.md b/docs/cosa/run.md index f17c253248..2bce573c9a 100644 --- a/docs/cosa/run.md +++ b/docs/cosa/run.md @@ -17,11 +17,41 @@ $ cosa run --qemu-image rhcos-qemu.qcow2 Many additional options are supported. Use `--help` to see them. The following sections discuss a few in more details. +## Running on cloud platforms + +`cosa run` only supports local QEMU VMs. To spawn instances on cloud +platforms, use `kola spawn` with the `-p` flag: + +``` +$ cosa kola spawn -p aws --aws-ami --aws-region us-east-1 +$ cosa kola spawn -p gcp --gcp-image --gcp-json-key +``` + +This launches an instance and opens an SSH shell. Use `--remove=false` to +keep the instance running after you exit, or `--detach` to spawn in the +background. See `cosa kola spawn --help` for all options and +[credentials](../mantle/credentials.md) for platform authentication setup. + ## Testing Butane or Ignition configs Using `cosa run` is a very effective way to iterate on your Butane or Ignition config. Use the `--butane/-B` or `--ignition/-i` switches respectively to pass -the config path. +the config path: + +``` +$ cosa run -B my-config.bu +$ cosa run -i my-config.ign +``` + +A few things to keep in mind: + +- `--butane` and `--ignition` are **mutually exclusive** — you can pass one or + the other, but not both. +- Butane configs are **automatically transpiled** to Ignition, so there is no + need to run `butane` manually before passing the file. +- Your config becomes the **base** Ignition config for the VM. COSA + automatically merges its own built-in fragments on top (e.g., autologin when + using devshell mode), so your config is never replaced. ## Using the serial console @@ -146,7 +176,7 @@ installed system automatically just as the live environment itself was set up.) Of course, one can also use an Ignition config or [a customized ISO](https://coreos.github.io/coreos-installer/cmd/iso/#coreos-installer-iso-customize) or the `coreos.inst.*` kargs using `--kargs` to also manually test automated -flows. (Many of these flows are covered by our `kola testiso` tests.) +flows. (Many of these flows are covered by kola's ISO install tests.) ## Multipath @@ -214,7 +244,7 @@ TARGET SOURCE FSTYPE OPTIONS (The `--ignition /run/ignition.json` is a trick for getting auto-login on the installed system automatically just as the live environment itself was set up.) -This is equivalent to our `kola testiso` multipath tests. +This is equivalent to kola's ISO multipath install tests. ## Netbooting diff --git a/docs/design.md b/docs/design.md index 6b5864b771..0afee62d5a 100644 --- a/docs/design.md +++ b/docs/design.md @@ -37,9 +37,10 @@ commit. There is also a `builds/builds.json` which maintains the list of builds. The reason for this is that HTTP doesn't offer a way to enumerate a directory. -After a build is generated there are a variety of `buildextend-$x` commands, -for example `buildextend-ec2` which can upload to AWS, and `buildextend-metal` -which generates a bare metal disk image. +After a build is generated, `cosa osbuild ` creates disk images for +specific platforms (e.g. `cosa osbuild metal` for bare metal, `cosa osbuild aws` +for AWS). To upload images to cloud providers, use `cosa imageupload-` +(e.g. `cosa imageupload-aws`). By default, builds are pruned (as is the OSTree repository), although one can use `build --no-prune` to prevent this. diff --git a/docs/kola.md b/docs/kola.md index d0f683b2c0..594abb66a5 100644 --- a/docs/kola.md +++ b/docs/kola.md @@ -31,6 +31,46 @@ By default, kola uses the `qemu` platform with the most recently built image 1. TOC {:toc} +## Running kola via cosa + +The recommended way to run kola is through the `cosa kola` wrapper. It handles +several defaults automatically: + +- **Platform:** defaults to `-p qemu` (when not running as root and no + `-p`/`--platform` is given). +- **Output directory:** defaults to `tmp/kola` (or `$ARTIFACT_DIR/kola` if + `$ARTIFACT_DIR` is set). +- **Subcommand:** defaults to `run` if no subcommand is given. For upgrade + tests, use `cosa kola --upgrades`. +- **Build:** uses the latest build in the cosa workdir by default. Use + `--build` to select a specific build. + +For example, `cosa kola run basic` is equivalent to running: + +``` +kola -p qemu run basic --output-dir tmp/kola +``` + +## Automatic external test discovery + +When running inside a cosa workdir, kola automatically discovers external tests +from these locations (no `-E` flag required): + +1. **`src/config/tests/kola/`** -- tests from the config repo, registered with + the prefix `ext.config` (e.g., a test at + `src/config/tests/kola/files/aleph-version/test.sh` becomes + `ext.config.files.aleph-version`). +2. **`/usr/lib/coreos-assembler/tests/kola/`** -- tests installed by packages, + registered with the prefix `ext`. +3. **`./tests/kola/`** (current directory) -- if no `-E` flag is given and no + cosa build is detected, kola checks the current directory for a + `tests/kola/` subdirectory. + +The `-E`/`--exttest` flag is only needed for *additional* external test +directories beyond these defaults. See +[kola/external-tests.md](kola/external-tests.md) for details on how to write +external tests. + ## kola run The run command invokes the main kola test harness. It @@ -87,6 +127,22 @@ Example format of the file: The special pattern `skip-console-warnings` suppresses the default check for kernel errors on the console which would otherwise fail a test. +### Common flags + +| Flag | Description | +|------|-------------| +| `-j` / `--parallel N` | Run N tests in parallel (default: 1). Use `auto` to match CPU count. | +| `--tag TAG` | Run only tests with the given tag. May be repeated. Prefix with `!` to exclude. | +| `--denylist-test PATTERN` | Skip tests matching the pattern. May be repeated. | +| `-E` / `--exttest DIR` | Add an external test directory (see above for automatic discovery). | +| `--rerun` | Re-run failed tests once. | +| `--append-butane PATH` | Merge a Butane config snippet with every test's Ignition config. | +| `--append-ignition PATH` | Merge an Ignition config snippet with every test's Ignition config. | +| `--ssh-on-test-failure` | SSH into the machine when a test fails (useful for debugging). | +| `--multiply N` | Run matched tests N times (useful for finding race conditions). | + +Use `cosa kola run -h` for the full list of options. + ## kola list The list command lists all of the available tests. @@ -198,29 +254,28 @@ After you run the kola test, you can find more information in `tmp/kola/` Is the most generic way of testing extended artifacts, this is mostly useful for the cloud platforms +3. `cosa osbuild ` (Build a disk image for a specific platform, e.g. `cosa osbuild aws`) +4. `kola run -p ` Is the most generic way of testing platform artifacts, this is mostly useful for the cloud platforms 5. For running the likes of metal/metal4k artifacts there's not much difference than running `kola run` from the coreos-assembler 6. `cd builds/latest/` (This will show your latest build information) 7. `cosa list` (This will show you the most recent CoreOS builds that have been made and the artifacts that were created) -8. In the case of the `testiso` command, you can determine what tests are running by looking for the pattern in the test name. It will follow: `test-to-run.disk-type.networking.multipath.firmware`. For example, the `iso-live-login.4k.uefi`, attempts to install FCOS/RHCOS to a disk that uses 4k sector size. If you don't see the 4k pattern, the `testiso` command will attempt to install FCOS/RHCOS to a non 4k disk (512b sector size). -9. `cosa kola testiso iso-offline-install.mpath.uefi` (This is an example testing the live ISO build with no internet access using multipath and the uefi firmware.) - -Example output: +8. ISO install tests are regular kola tests. The test name pattern tries to indicate what the test is running. For example, `iso.pxe-online-install.4k.uefi` attempts to install FCOS/RHCOS to a disk that uses 4k sector size. If you don't see the `4k` pattern, the test uses a standard 512-byte sector size disk. +9. `cosa kola run iso.iso-offline-install.mpath.bios` is an example testing the live ISO build with no internet access using multipath and the bios firmware.) ``` -kola -p qemu testiso --inst-insecure --output-dir tmp/kola -Ignoring verification of signature on metal image -Running test: iso-as-disk.bios -PASS: iso-as-disk.bios (12.408s) -Running test: iso-as-disk.uefi -PASS: iso-as-disk.uefi (16.039s) -Running test: iso-as-disk.uefi-secure -PASS: iso-as-disk.uefi-secure (16.994s) +cosa kola run iso.pxe-online-install.4k.uefi iso.iso-offline-install.mpath.bios + kola -p qemu run iso.pxe-online-install.4k.uefi iso.iso-offline-install.mpath.bios --output-dir tmp/kola + === RUN iso.pxe-online-install.4k.uefi + === RUN iso.iso-offline-install.mpath.bios + Detected development build; disabling signature verification + --- PASS: iso.pxe-online-install.4k.uefi (88.66s) + Detected development build; disabling signature verification + --- PASS: iso.iso-offline-install.mpath.bios (193.72s) + PASS, output in tmp/kola ``` ## Useful commands diff --git a/docs/using-ai-skills.md b/docs/using-ai-skills.md new file mode 100644 index 0000000000..d302e82ddb --- /dev/null +++ b/docs/using-ai-skills.md @@ -0,0 +1,43 @@ +--- +nav_order: 10 +--- + +# Working with COSA's AI skills +{: .no_toc } + +Example tasks to hand to an AI tool (i.e. opencode, goose) leveraging +the skills under .agents/skills. To leverage a skill run `/skills` and +choose a skill and then give a prompt like the examples below: + +1. TOC +{:toc} + +## Simple Task + + Run the ext.config.upgrade.extended test starting from 44.20260510.2.1. When done look at the + journal in the logs from the test and tell me how long the rpm-ostree operations took for every boot. + +## Intermediate Task + + Do a local build against rawhide, but override the kernel in the build with the kernel build from + this bodhi update https://bodhi.fedoraproject.org/updates/FEDORA-2026-62bf55e4d8 and run the + ext.config.security.lockdown test against the built image. + +## Advanced Task + + With the local changes to the git repo here compile a new kola. Then write a new external test that + is a minimal version of ext.config.upgrade.extended but really only does: + + 1. leaves zincati disabled + 2. does a skopeo copy of quay.io/fedora/fedora-coreos:stable to an ociarchive + 3. does a manual `sudo rpm-ostree rebase ostree-unverified-image:oci-archive://path/to/ociarchive` + + Then we should run kola (newly compiled) to start an instance at the earliest `stable` stream build + based on Fedora 43 and run the test. + + I want to run the test with and without one change to benchmark them and compare the differences: + + - updating fsync to false in the ostree repo config (see https://ostreedev.github.io/ostree/man/ostree.repo-config.html) + inside the instance + + Run this test on AWS m8a.xlarge instances using the credentials in the `aws-creds` file. diff --git a/docs/working.md b/docs/working.md index a5fd78aadb..ada5e15850 100644 --- a/docs/working.md +++ b/docs/working.md @@ -357,7 +357,7 @@ requires access to podman and /dev/kvm. # to run the freshly built image cosa run # to build the live ISO - cosa buildextend-metal && cosa buildextend-live + cosa osbuild metal live # to run the live ISO cosa run -p qemu-iso ``` diff --git a/mantle/README.md b/mantle/README.md index ff5ad4a3aa..36724179c0 100644 --- a/mantle/README.md +++ b/mantle/README.md @@ -159,7 +159,7 @@ api for each cloud provider closely, so the interface for each cloud provider is different. See each providers `help` command for the available actions. Note, when uploading to cloud platforms, be sure to use the image built for -that particular platform (with `cosa buildextend-...`). +that particular platform (with `cosa osbuild `). ### plume Plume is the Container Linux release utility. Releases are done in two stages,