diff --git a/.github/workflows/README.md b/.github/workflows/README.md index bca786ec..38832869 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -18,6 +18,7 @@ deleted.) | `integration-tests.yml` | Bench: Integration Tests | push to `main`, `workflow_call`, dispatch | self-hosted `lager-bench` | Drives every bench instrument through `lager python`, plus the J-Link CLI suite | | `update-regression.yml` | Bench: Box Lifecycle | `workflow_call`, dispatch | self-hosted `lager-bench` | Downgrade -> update -> no-op -> forced rebuild -> uninstall -> install, with a hardware smoke per phase | | `nightly-bench.yml` | Bench: Nightly | cron 10:17 UTC, dispatch | (calls the two above) | Nightly ordering wrapper: lifecycle first, instruments only if it succeeded; files/closes the `bench-alert` issue | +| `bench-extended.yml` | Bench: Extended | cron Sat 14:17 UTC, dispatch | self-hosted `lager-bench` | Weekly run of five of the seven infrastructure integration suites (no instruments; bench stays dark); alerts on failure, never closes the alert | | `bench-watchdog.yml` | Bench: Watchdog | cron every 6h at :41, dispatch | `ubuntu-latest` | Alerts when nightly runs stop FLOWING (queued too long, stuck, cron dead) — the failure the notify jobs cannot see | | `unit-tests.yml` | PR Gate: Unit Tests | `pull_request`, push to `main`, dispatch | `ubuntu-latest` | Unit suites (one pytest process per suite) + Python-version compat matrix | | `static-checks.yml` | PR Gate: Static Checks | `pull_request`, push to `main`, dispatch | `ubuntu-latest` | Syntax/lint floors over the tree the unit gate cannot reach, plus a coverage report | diff --git a/.github/workflows/bench-extended.yml b/.github/workflows/bench-extended.yml new file mode 100644 index 00000000..404de33a --- /dev/null +++ b/.github/workflows/bench-extended.yml @@ -0,0 +1,403 @@ +# Copyright 2024-2026 Lager Data +# SPDX-License-Identifier: Apache-2.0 +# +# Weekly execution for bench tests that exist but run nowhere. Of the 38 bash +# integration suites, nightly CI executes exactly one (jlink_script.sh); the +# other 37 were syntax-checked and shellchecked but never run, so a real +# regression in the CLI surface they cover -- argument parsing, exit codes, +# box-level commands -- surfaced only when a human happened to run one against +# a bench. This workflow starts working that debt down. +# +# WAVE 1 (this file): five of the seven test/integration/infrastructure/ +# suites -- deployment, devenv, nets, box_config and generic. They need a box +# but NO instruments, so this workflow has none of integration-tests.yml's +# relay power, relay-net self-heal, or firmware staging, and the bench stays +# dark for the whole run. +# +# The other two are deliberately excluded, each for its own reason recorded +# beside the suite steps below: boxes_config.sh would print a network +# inventory into a public log, and python.sh tests a CLI surface that no +# longer exists. Wave 2 (test/api/ scripts the existing instruments support) +# is follow-up work and will need the instrument steps when it lands. +# +# Ordering inside the run is load-bearing: +# - generic.sh runs LAST: it exercises `lager boxes delete-all` against the +# runner's own lager config, which can remove this workflow's box +# registration; everything after it would fail to resolve the box. +# - box_config.sh runs late: it exercises box-config apply including the +# container bounce + rollback path, so the box may be briefly degraded. +# - A best-effort re-registration step follows the suites either way: these +# scripts mutate the RUNNER's persistent lager config by design, and +# every bench workflow re-registers its box at start, so damage is +# bounded to one run -- but healing it here keeps this run's own tail +# steps working too. +# +# Weekly, Saturday 14:17 UTC: clear of the nightly's worst case (10:17 UTC +# start + ~2.5h), off the hour for GitHub's scheduled-queue congestion, and +# serialized against the other bench workflows by the shared concurrency +# group below. This is a LEAF workflow (no workflow_call children), so the +# caller-holds-the-slot deadlock documented in nightly-bench.yml does not +# apply here. +# +# Security: this repo is PUBLIC and the runner drives real hardware, so this +# workflow deliberately does NOT trigger on `pull_request` -- a fork PR must +# not be able to execute code on the bench. To bench-test a branch, push it +# to this repo and workflow_dispatch on that ref. +# +# Output surface: several CLI errors print an "Available boxes:" list, so this +# log carries whatever the RUNNER's own box registry holds. That is safe only +# because the runner registers exactly one box and its address is a repository +# secret, which the runner masks. Registering anything else on the runner +# turns those error paths into a fleet listing in a public log -- keep the +# runner's registry to the CI box. +# +# Notification: a red run files/updates the single `bench-alert` issue via +# tools/bench_alert.sh. There is deliberately NO recovery job here: closing +# the alert issue is exclusively the nightly's signal, because a green weekly +# must never mask a nightly that is still failing. + +name: "Bench: Extended" + +on: + schedule: + - cron: "17 14 * * 6" + workflow_dispatch: + inputs: + box_name: + description: "Lager box name (overrides vars.LAGER_BOX)" + required: false + type: string + +permissions: + contents: read + +concurrency: + # One physical bench: queue behind nightly/integration, never collide. + group: hardware-ci-${{ vars.LAGER_BOX || 'MASTER' }} + cancel-in-progress: false + +jobs: + extended: + name: Run infrastructure suites on the bench + runs-on: [self-hosted, lager-bench] + timeout-minutes: 75 + env: + LAGER_BOX: ${{ inputs.box_name || vars.LAGER_BOX || 'MASTER' }} + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + # No job here pushes anything; do not persist the token into + # the checkout's config (zizmor: artipacked). + persist-credentials: false + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 + with: + python-version: '3.11' + + - name: Install lager CLI (editable, from this repo) + run: pip install -e cli/ + shell: bash + + - name: Sign in to the box access gateway + env: + LAGER_AUTH_URL: ${{ vars.LAGER_AUTH_URL }} + LAGER_CI_EMAIL: ${{ secrets.LAGER_CI_EMAIL }} + LAGER_CI_PASSWORD: ${{ secrets.LAGER_CI_PASSWORD }} + run: lager login "$LAGER_AUTH_URL" --email "$LAGER_CI_EMAIL" --password "$LAGER_CI_PASSWORD" + shell: bash + + - name: Register lager box + run: | + lager boxes add \ + --name "$LAGER_BOX" \ + --ip "${{ secrets.LAGER_BOX_IP }}" \ + --user lagerdata \ + --yes + shell: bash + + - name: Set up lager SSH key + # Same identity mapping integration-tests.yml uses: box_config.sh and + # deployment.sh ssh into the box directly with the default identity. + run: | + if [ ! -f ~/.ssh/lager_box ]; then + ssh-keygen -t ed25519 -f ~/.ssh/lager_box -N "" -C "lager-box-access" + fi + grep -qF "$(cat ~/.ssh/lager_box.pub)" ~/.ssh/authorized_keys 2>/dev/null || \ + cat ~/.ssh/lager_box.pub >> ~/.ssh/authorized_keys + touch ~/.ssh/config && chmod 600 ~/.ssh/config + if ! grep -q "^# lager integration CI box identity" ~/.ssh/config; then + { + echo "# lager integration CI box identity" + echo "Host ${{ secrets.LAGER_BOX_IP }}" + echo " IdentityFile ~/.ssh/lager_box" + echo "Host $LAGER_BOX" + echo " HostName ${{ secrets.LAGER_BOX_IP }}" + echo " IdentityFile ~/.ssh/lager_box" + } >> ~/.ssh/config + fi + shell: bash + + - name: Release a still-held box lock from a killed run + # The CLI releases its auto-lock three ways (try/finally, atexit, and + # a SIGTERM/SIGINT/SIGHUP handler), but a step timeout ends in + # SIGKILL, which cannot be caught. The client-side release never + # runs, and the box-side TTL becomes the only backstop -- 1800s by + # default (cli/box_storage.py _DEFAULT_LOCK_TTL_SECONDS). An expired + # lock is reaped automatically and needs no --force; an unexpired one + # blocks the next run outright. + # + # That is a real collision for back-to-back runs -- dispatches during + # a debugging session, or a re-run after a timeout, land well inside + # 30 minutes. It is NOT a threat to the schedule: the nightly and + # this weekly are hours apart, so any leaked lock has long expired. + # + # Forcing is safe here because the hardware-ci concurrency group + # serialises every bench workflow against this box, so a lock still + # held when this job starts belongs to a run that is no longer + # running. update-regression.yml carries the same call for its own + # leak path (a lost release POST rather than a signal). + run: lager boxes unlock --box "$LAGER_BOX" --force 2>/dev/null || true + shell: bash + + - name: Verify box connectivity + # First authenticated contact after a fresh login can fail exactly + # once while the gateway records the box<->auth-server link. + run: lager hello --box "$LAGER_BOX" || lager hello --box "$LAGER_BOX" + shell: bash + + # Each suite is continue-on-error with an id so every suite runs and + # the aggregation gate at the end turns any failure into a job failure + # -- same pattern as integration-tests.yml, same reasoning. + # + # Each suite also declares a BASELINE: the number of its checks known to + # fail today. tools/bench_suite_gate.sh ratchets two-sided -- more + # failures than the baseline is red, fewer prints a notice telling you to + # lower it. This is what lets the workflow land green on day one without + # softening a single assertion. Every non-zero baseline names the issue + # that owns it; a bare number with no issue behind it is softening. + # + # The suite's own exit status is deliberately dropped (`|| true`): it + # carries the same information as the failure count, and reading it + # through `tee` is how a gate ends up reporting tee's status. The gate + # fails loudly if a suite produces no summary at all, which is what a + # crash or a step timeout looks like. + + - name: Suite deployment.sh (box verification, no instruments) + id: suite-deployment + continue-on-error: true + timeout-minutes: 10 + env: + # The suite defaults to ssh user `lager`; this bench logs in as + # `lagerdata`. + SSH_USER: lagerdata + # Expected clean. Its two failures were a stale expectation: the + # suite inspected containers named `controller` and `python`, which + # no longer exist -- there is one container, named `lager`. Fixed + # rather than baselined. + DEPLOYMENT_BASELINE: "0" + run: | + LOG="$RUNNER_TEMP/deployment.log" + bash test/integration/infrastructure/deployment.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh deployment deployment.sh "$DEPLOYMENT_BASELINE" "$LOG" + shell: bash + + # python.sh is deliberately NOT wired, for the same reason as + # boxes_config.sh: it does not test the CLI we ship. 53 of its checks + # invoke `lager python -c ""` for inline execution, and there is + # no `-c` option -- the runner reports "Error: No such option '-c'" + # and the suite fails 43 of 84. Three of its checks pass `--image`, + # which is likewise gone. It needs the rewrite devenv.sh already had, + # against the current interface (a script path as the positional + # RUNNABLE), and that is its own reviewable change. Baselining 43 + # failures here would record "calls an option that does not exist" as + # the expected state and leave the ratchet watching for a 44th. + # + # Its other findings stand on their own and are filed as issue #274: + # `--timeout` never fires, and a detached launch does not return. + + - name: Stop anything a suite left running on the box + # Cheap insurance that survives python.sh being unwired: any suite + # that starts work on the box and is then killed mid-step leaves it + # behind, and the next suite inherits a busy box -- failures that + # belong to no suite and would otherwise be baselined as if they did. + if: always() + continue-on-error: true + timeout-minutes: 3 + run: | + # --kill-all, not --kill: --kill takes a process ID. + timeout 45 lager python --box "$LAGER_BOX" --kill-all --signal sigkill >/dev/null 2>&1 || true + for attempt in $(seq 1 24); do + if lager hello --box "$LAGER_BOX" >/dev/null 2>&1; then + echo "box responsive on attempt $attempt" + exit 0 + fi + sleep 5 + done + echo "::warning title=Bench Extended::box unresponsive after the python suite" + shell: bash + + - name: Suite devenv.sh (local dev environment, runner-side docker) + id: suite-devenv + continue-on-error: true + timeout-minutes: 10 + env: + # Clean, and expected to stay that way. + DEVENV_BASELINE: "0" + run: | + LOG="$RUNNER_TEMP/devenv.log" + bash test/integration/infrastructure/devenv.sh 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness devenv.sh "$DEVENV_BASELINE" "$LOG" + shell: bash + + # boxes_config.sh is deliberately NOT wired: its discovery section + # inventories every box reachable from the runner's network and prints + # the results, and a full network inventory does not belong in a CI + # log. It stays runnable by hand; wiring it here needs a CI-safe output + # mode for the discovery commands first (counts, not names). + + - name: Suite nets.sh (net add/delete/rename lifecycle) + # Its delete-all section is self-skipped in the script; individual + # test-net operations only. Even so, the relay-net self-heal in the + # instrumented workflows covers any net damage on their next run. + id: suite-nets + continue-on-error: true + timeout-minutes: 10 + env: + # Clean. It measured 4 failures while python.sh ran ahead of it; + # with that suite unwired every check passes. Keep at 0. + NETS_BASELINE: "0" + run: | + LOG="$RUNNER_TEMP/nets.log" + bash test/integration/infrastructure/nets.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness nets.sh "$NETS_BASELINE" "$LOG" + shell: bash + + - name: Suite box_config.sh (box-config apply, bounce + rollback) + # Late on purpose: exercises the container bounce path, so the box + # may be briefly degraded; nothing after it needs full box health + # except generic.sh's CLI-level checks. + id: suite-box-config + continue-on-error: true + timeout-minutes: 15 + env: + # 6 of 56: the sysctl/cargo apply+clear cluster, the disabled + # box_http_server still running, and the box-side validate + # wording. Owned by issue #275; lower this as they are fixed. + BOX_CONFIG_BASELINE: "6" + run: | + LOG="$RUNNER_TEMP/box-config.log" + bash test/integration/infrastructure/box_config.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness box_config.sh "$BOX_CONFIG_BASELINE" "$LOG" + shell: bash + + - name: Wait for the box to come back after the config bounce + # box_config.sh ends by restarting the container, and generic.sh opens + # with `lager hello`. Without this the first two hello checks failed + # and the third passed eleven seconds later -- a race this workflow's + # own suite ordering creates, which would otherwise be charged to + # generic.sh's baseline and hide a real regression behind it. + # continue-on-error so a box that never comes back is reported by the + # suites themselves rather than as an opaque step failure here. + continue-on-error: true + timeout-minutes: 3 + run: | + for attempt in $(seq 1 30); do + if lager hello --box "$LAGER_BOX" >/dev/null 2>&1; then + echo "box responded on attempt $attempt" + exit 0 + fi + sleep 5 + done + echo "::warning title=Bench Extended::box did not respond within 150s after the box-config bounce" + shell: bash + + - name: Suite generic.sh (CLI-generic commands; LAST -- see header) + id: suite-generic + continue-on-error: true + timeout-minutes: 15 + env: + # 5 of 95: the two 50-iteration stress loops, the mixed-command + # sequence, and both works-after-errors checks. Single calls and + # the 5- and 10-call loops all pass, so this is specifically + # sustained repetition. Owned by issue #273; lower this as it + # is fixed. + GENERIC_BASELINE: "5" + run: | + LOG="$RUNNER_TEMP/generic.log" + bash test/integration/infrastructure/generic.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness generic.sh "$GENERIC_BASELINE" "$LOG" + shell: bash + + - name: Restore box registration (best effort) + # generic.sh exercises `boxes delete-all` against the runner's lager + # config; re-register so this run's remaining steps -- and any human + # using the runner before the next workflow -- see a sane state. + if: always() + continue-on-error: true + run: | + lager boxes add \ + --name "$LAGER_BOX" \ + --ip "${{ secrets.LAGER_BOX_IP }}" \ + --user lagerdata \ + --yes || true + shell: bash + + - name: Fail job if any suite failed + # Reads the steps context rather than a hand-kept id list -- adding a + # suite needs no edit here. `outcome` is the pre-continue-on-error + # result. + env: + STEPS_JSON: ${{ toJSON(steps) }} + run: | + python3 - <<'PY' + import json, os, sys + steps = json.loads(os.environ["STEPS_JSON"]) + bad = sorted(k for k, v in steps.items() + if isinstance(v, dict) and v.get("outcome") == "failure") + if bad: + print("One or more infrastructure suites failed:") + for name in bad: + print(f" {name}") + sys.exit(1) + print("All infrastructure suites passed.") + PY + shell: bash + + notify-failure: + # Same alert path as the nightly; see the header for why there is no + # recovery counterpart here. + name: notify (failure) + needs: [extended] + if: ${{ !cancelled() && needs.extended.result != 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + issues: write + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + # No job here pushes anything; do not persist the token into + # the checkout's config (zizmor: artipacked). + persist-credentials: false + + - name: File or update the bench-alert issue + env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + RESULT: ${{ needs.extended.result }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + cat > "$RUNNER_TEMP/alert-body.md" < # @@ -79,7 +85,10 @@ lager hello --box "$BOX" 2>&1 | grep -qi "online" \ && track_test "pass" || track_test "fail" echo "Test 1.4: lager box config show reports no config" -lager box config show --box "$BOX" 2>&1 | grep -qi "no box_config" \ +# Bounded: the first CI run of this suite hung here for 15 minutes right +# after the container restart in 1.2 -- `show` never returned. A hang is a +# FAIL with a visible reason, not a stall that eats the job timeout. +timeout 60 lager box config show --box "$BOX" 2>&1 | grep -qi "no box_config" \ && track_test "pass" || track_test "fail" # Test mount paths. Use fresh paths under /tmp so auto-prep can mkdir/chown @@ -462,8 +471,20 @@ EOF track_test "pass" echo "Test 5.2: validate exits non-zero" -lager box config validate --box "$BOX" 2>&1 | grep -qi "cannot be '/'" \ - && track_test "pass" || track_test "fail" +# The assertion stays as-is on purpose. validate and apply share one +# validate() call (box/lager/box_config/config.py _validate_mounts), and this +# tree's unit tests pin the "cannot be '/'" wording, so a bench box that does +# not produce it is running box-side code older than main -- which is a real +# finding, not a reason to loosen the grep. Print what it actually said so the +# next reader of this log does not have to reproduce it to find out. +VALIDATE_OUT=$(lager box config validate --box "$BOX" 2>&1) +if echo "$VALIDATE_OUT" | grep -qi "cannot be '/'"; then + track_test "pass" +else + echo " validate did not report the root-mount error; it said:" + echo "$VALIDATE_OUT" | sed 's/^/ /' + track_test "fail" +fi echo "Test 5.3: apply refuses" OUT=$(lager box config apply --box "$BOX" --yes 2>&1) diff --git a/test/integration/infrastructure/boxes_config.sh b/test/integration/infrastructure/boxes_config.sh index 4ae29e1c..c4476681 100755 --- a/test/integration/infrastructure/boxes_config.sh +++ b/test/integration/infrastructure/boxes_config.sh @@ -119,12 +119,12 @@ lager boxes import "$EXPORT_FILE" --merge --yes && track_test "pass" || track_te echo "" echo "Test 3.3: Import non-existent file (error case)" -lager boxes import "/tmp/nonexistent_file_$$.json" --yes 2>&1 | grep -qi "error\|not found\|no such" && track_test "pass" || track_test "pass" +lager boxes import "/tmp/nonexistent_file_$$.json" --yes 2>&1 | grep -qi "error\|not found\|no such" && track_test "pass" || track_test "fail" echo "" echo "Test 3.4: Import invalid JSON file" echo "not valid json" > "/tmp/invalid_json_$$.json" -lager boxes import "/tmp/invalid_json_$$.json" --yes 2>&1 | grep -qi "error\|invalid\|json" && track_test "pass" || track_test "pass" +lager boxes import "/tmp/invalid_json_$$.json" --yes 2>&1 | grep -qi "error\|invalid\|json" && track_test "pass" || track_test "fail" rm -f "/tmp/invalid_json_$$.json" echo "" @@ -135,8 +135,8 @@ start_section "Round-Trip" echo "Test 4.1: Add test boxes, export, delete-all, import, verify" # Add test boxes -lager boxes add --name "roundtrip_test_1" --ip "192.168.200.1" --yes >/dev/null 2>&1 -lager boxes add --name "roundtrip_test_2" --ip "192.168.200.2" --yes >/dev/null 2>&1 +lager boxes add --name "roundtrip_test_1" --ip "192.168.200.1" --user testuser --yes >/dev/null 2>&1 +lager boxes add --name "roundtrip_test_2" --ip "192.168.200.2" --user testuser --yes >/dev/null 2>&1 # Export ROUNDTRIP_FILE="/tmp/lager_roundtrip_$$.json" @@ -172,7 +172,7 @@ start_section "Add-All" echo "Test 5.1: Add-all (auto-discover boxes)" # This requires Tailscale -- may fail if not on Tailscale network -lager boxes add-all --yes 2>&1 +lager boxes add-all --user testuser --yes 2>&1 # Pass regardless since Tailscale may not be available track_test "pass" echo "" diff --git a/test/integration/infrastructure/deployment.sh b/test/integration/infrastructure/deployment.sh index fd95fa01..a832193d 100755 --- a/test/integration/infrastructure/deployment.sh +++ b/test/integration/infrastructure/deployment.sh @@ -13,7 +13,11 @@ # Example: ./test_deployment.sh # Example: ./test_deployment.sh my-box -set -e +# DON'T exit on error - we want to track failures, matching every other suite +# in this directory. Under `set -e` the first failing run_test aborted the +# script, so the summary block at the end never ran and a single early failure +# hid every later result. +set +e SSH_USER="${SSH_USER:-lager}" @@ -42,35 +46,31 @@ if [ -z "$1" ]; then echo "Example: $0 " echo "" echo "Note: For best results, add the DUT first using:" - echo " lager duts add --name --ip [--user ]" + echo " lager boxes add --name --ip --user --yes" exit 1 fi DUT="$1" -# Check if DUT exists in saved list (by name or IP) -DUT_CHECK=$(lager duts 2>/dev/null | grep "^${DUT}" | head -1) +# Check if the box exists in the saved registry (by name or IP). This used +# to read `lager duts`, a registry that no longer exists in the CLI -- every +# run took the not-found path -- and then blocked on an interactive prompt, +# which in a non-TTY context reads EOF and exits. The registry is +# `lager boxes` now, and the not-found case warns and continues, which was +# the stated intent all along ("don't fail - allow testing with raw IPs"). +DUT_CHECK=$(lager boxes 2>/dev/null | grep "^${DUT}" | head -1) if [ -z "$DUT_CHECK" ]; then # Try to find by IP - DUT_CHECK=$(lager duts 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') + DUT_CHECK=$(lager boxes 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') fi -# Warn if DUT not found (but don't fail - allow testing with raw IPs) if [ -z "$DUT_CHECK" ]; then - echo -e "${YELLOW}Warning: DUT '${DUT}' not found in saved DUTs${NC}" + echo -e "${YELLOW}Warning: box '${DUT}' not found in saved boxes${NC}" echo -e "${YELLOW}Tests may fail if SSH authentication is not configured${NC}" echo "" - echo "Saved DUTs:" - lager duts 2>/dev/null || echo " (none)" + echo "To register it:" + echo " lager boxes add --name ${DUT} --ip --user --yes" echo "" - echo "To add this DUT:" - echo " lager duts add --name ${DUT} --ip [--user ]" - echo "" - read -p "Continue anyway? (y/N): " -n 1 -r - echo - if [[ ! $REPLY =~ ^[Yy]$ ]]; then - exit 1 - fi fi # Print header @@ -103,11 +103,13 @@ if ! command -v lager &> /dev/null; then exit 1 fi -# Detect DUT username early for test skipping logic +# Detect DUT username early for test skipping logic. Same registry rename as +# the check above: `lager boxes` columns are name / ip / user / version / +# status, so the $3 read below still lands on the user. DETECTED_USER="" -DUT_INFO_EARLY=$(lager duts 2>/dev/null | grep "^${DUT}" | head -1) +DUT_INFO_EARLY=$(lager boxes 2>/dev/null | grep "^${DUT}" | head -1) if [ -z "$DUT_INFO_EARLY" ]; then - DUT_INFO_EARLY=$(lager duts 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') + DUT_INFO_EARLY=$(lager boxes 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') fi if [ -n "$DUT_INFO_EARLY" ]; then DETECTED_USER=$(echo "$DUT_INFO_EARLY" | awk '{print $3}') @@ -238,7 +240,7 @@ run_test_with_output "lager hello --box ${DUT}" "timeout 30 lager hello --box ${ print_test_header "2. Configuration Management Tests" run_test "lager defaults --help" "lager defaults --help" -run_test "lager duts --help" "lager duts --help" +run_test "lager dut --help" "lager dut --help" # Note: Some lager commands (like instruments, nets) may not work with custom usernames # They try to connect as the default SSH user regardless of box configuration @@ -342,17 +344,18 @@ run_test "lager ssh --help" "lager ssh --help" # Test SSH command execution (non-interactive) - use direct SSH since lager ssh doesn't support --command # First resolve DUT to IP and username for SSH # Handle both old format (just IP) and new format (with user field) -# lager duts output format: +# lager boxes output format: # name ip user # - # test-deploy test-deploy -# Try to find DUT by name first -DUT_INFO=$(lager duts 2>/dev/null | grep "^${DUT}" | head -1) +# Try to find the box by name first (`lager boxes`; see the registry note +# at the top of the script) +DUT_INFO=$(lager boxes 2>/dev/null | grep "^${DUT}" | head -1) # If not found by name, try to find by IP address (2nd column match) if [ -z "$DUT_INFO" ]; then - DUT_INFO=$(lager duts 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') + DUT_INFO=$(lager boxes 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') fi if [ -n "$DUT_INFO" ]; then @@ -444,9 +447,11 @@ print_test_header "10. Container Health Tests" # Check if all expected containers are running (use direct SSH) run_test_with_output "Docker containers running" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker ps --filter name=controller --filter name=python --format \"{{.Names}}\"'" -# Check container health (use direct SSH) -run_test_with_output "Controller container health" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker inspect controller --format \"{{.State.Status}}\"'" -run_test_with_output "Python container health" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker inspect python --format \"{{.State.Status}}\"'" +# Check container health (use direct SSH). One container, named `lager` +# (box/start_box.sh). This used to inspect `controller` and `python`, which +# no longer exist -- so both checks failed on every box, reporting a stale +# expectation as if it were a box fault. +run_test_with_output "Lager container health" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker inspect lager --format \"{{.State.Status}}\"'" # ============================================================================= # Test Section 11: Box Services diff --git a/test/integration/infrastructure/devenv.sh b/test/integration/infrastructure/devenv.sh index c0afc646..144d222b 100755 --- a/test/integration/infrastructure/devenv.sh +++ b/test/integration/infrastructure/devenv.sh @@ -2,15 +2,24 @@ # Copyright 2024-2026 Lager Data # SPDX-License-Identifier: Apache-2.0 -# Integration test suite for lager devenv commands -# Tests: create, list, remove +# Integration test suite for lager devenv commands. # -# Usage: ./test/integration/infrastructure/devenv.sh -# Example: ./test/integration/infrastructure/devenv.sh +# Tests the CURRENT devenv model: one devenv per project, configured in the +# project's .lager file (create/show/set/unset, commands add/delete, mount +# and env config). The previous version of this suite tested the removed +# named-environments model (create --name / list / remove) and failed on +# every one of those commands the first time it ran in CI. +# +# Everything here is config-level: `create` writes .lager and no container +# is ever started, so Docker is NOT required and no image is pulled. The +# container-side behavior (terminal, exec) stays out of scope -- it needs a +# TTY and an image pull, neither of which belongs in this suite. +# +# All work happens in a throwaway directory: devenv config is project-local, +# and .lager discovery walks parent directories, so running from a real +# checkout would read (or write!) that checkout's config. # -# Note: These tests require Docker to be installed locally. -# Tests are non-destructive -- they use a dedicated test -# environment name and clean up after themselves. +# Usage: ./test/integration/infrastructure/devenv.sh SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -21,24 +30,20 @@ set +e init_harness -TEST_ENV_NAME="lager_test_devenv_$$" +WORKDIR=$(mktemp -d "${TMPDIR:-/tmp}/lager_devenv_test.XXXXXX") +cleanup() { rm -rf "$WORKDIR"; } +trap cleanup EXIT +cd "$WORKDIR" || exit 1 + +TEST_IMAGE="debian:bookworm-slim" echo "========================================================================" echo "LAGER DEVENV COMPREHENSIVE TEST SUITE" echo "========================================================================" echo "" -echo "Test environment: $TEST_ENV_NAME" +echo "Working directory: $WORKDIR" echo "" -# Check Docker prerequisite -if ! command -v docker &>/dev/null; then - echo "Docker is not installed. Skipping devenv tests." - start_section "Prerequisites" - skip_test "Docker check" "Docker not installed" - print_summary - exit_with_status -fi - # ============================================================ # SECTION 1: HELP COMMANDS # ============================================================ @@ -52,81 +57,127 @@ echo "Test 1.2: Devenv create help" lager devenv create --help && track_test "pass" || track_test "fail" echo "" -echo "Test 1.3: Devenv list help" -lager devenv list --help && track_test "pass" || track_test "fail" +echo "Test 1.3: Devenv show help" +lager devenv show --help && track_test "pass" || track_test "fail" echo "" -echo "Test 1.4: Devenv remove help" -lager devenv remove --help && track_test "pass" || track_test "fail" +echo "Test 1.4: Devenv mount help" +lager devenv mount --help && track_test "pass" || track_test "fail" echo "" # ============================================================ -# SECTION 2: LIST +# SECTION 2: NO-CONFIG ERROR PATHS # ============================================================ -start_section "List Environments" +start_section "No-Config Errors" -echo "Test 2.1: List dev environments" -lager devenv list && track_test "pass" || track_test "fail" +echo "Test 2.1: show without a .lager file fails informatively" +OUTPUT=$(lager devenv show 2>&1) +if [ ! -e .lager ] && echo "$OUTPUT" | grep -qi "No .lager config"; then + track_test "pass" +else + echo "$OUTPUT" | head -3 + track_test "fail" +fi echo "" -echo "Test 2.2: List stability (5 iterations)" -FAILED=0 -for _ in {1..5}; do - lager devenv list >/dev/null 2>&1 || FAILED=1 -done -[ $FAILED -eq 0 ] && track_test "pass" || track_test "fail" +echo "Test 2.2: set without a .lager file fails informatively" +OUTPUT=$(lager devenv set shell /bin/sh 2>&1) +if echo "$OUTPUT" | grep -qi "No .lager config"; then + track_test "pass" +else + echo "$OUTPUT" | head -3 + track_test "fail" +fi echo "" # ============================================================ -# SECTION 3: CREATE AND REMOVE LIFECYCLE +# SECTION 3: CREATE AND CONFIG LIFECYCLE # ============================================================ -start_section "Create and Remove" - -echo "Test 3.1: Create dev environment" -lager devenv create --name "$TEST_ENV_NAME" && track_test "pass" || track_test "fail" +start_section "Create and Configure" + +echo "Test 3.1: create with all options is non-interactive and writes .lager" +# All three options given: create prompts for anything it was not told, and +# a prompt in a non-TTY context aborts -- see Test 5.1, which pins exactly +# that. +lager devenv create --image "$TEST_IMAGE" --mount-dir /app --shell /bin/sh \ + && [ -f .lager ] && track_test "pass" || track_test "fail" echo "" -echo "Test 3.2: Verify environment appears in list" -if lager devenv list 2>/dev/null | grep -q "$TEST_ENV_NAME"; then +echo "Test 3.2: show reflects what create wrote" +OUTPUT=$(lager devenv show 2>&1) +if echo "$OUTPUT" | grep -q "$TEST_IMAGE" \ + && echo "$OUTPUT" | grep -q "/app" \ + && echo "$OUTPUT" | grep -q "/bin/sh"; then track_test "pass" else + echo "$OUTPUT" | head -6 track_test "fail" fi echo "" -echo "Test 3.3: Remove dev environment" -lager devenv remove --name "$TEST_ENV_NAME" --yes && track_test "pass" || track_test "fail" +echo "Test 3.3: set/show roundtrip" +lager devenv set shell /bin/bash >/dev/null 2>&1 +if lager devenv show 2>&1 | grep -q "/bin/bash"; then + track_test "pass" +else + track_test "fail" +fi echo "" -echo "Test 3.4: Verify environment removed from list" -if lager devenv list 2>/dev/null | grep -q "$TEST_ENV_NAME"; then +echo "Test 3.4: unset removes the key" +lager devenv unset shell >/dev/null 2>&1 +if lager devenv show 2>&1 | grep -q "/bin/bash"; then track_test "fail" else track_test "pass" fi echo "" -# ============================================================ -# SECTION 4: ERROR CASES -# ============================================================ -start_section "Error Cases" +echo "Test 3.5: commands starts empty and add/delete roundtrips" +lager devenv add build "make all" >/dev/null 2>&1 +ADDED=$(lager devenv commands 2>&1) +lager devenv delete build >/dev/null 2>&1 +GONE=$(lager devenv commands 2>&1) +if echo "$ADDED" | grep -q "build" && ! echo "$GONE" | grep -q "build"; then + track_test "pass" +else + echo "after add: $(echo "$ADDED" | head -2)" + echo "after delete: $(echo "$GONE" | head -2)" + track_test "fail" +fi +echo "" -echo "Test 4.1: Remove non-existent environment" -lager devenv remove --name "nonexistent_env_12345" --yes 2>&1 | grep -qi "error\|not found" && track_test "pass" || track_test "pass" +echo "Test 3.6: env config help is reachable" +lager devenv env --help && track_test "pass" || track_test "fail" echo "" -echo "Test 4.2: Create with invalid image name" -lager devenv create --name "test_invalid_$$" --image "!!!invalid!!!" 2>&1 | grep -qi "error\|invalid" && track_test "pass" || track_test "pass" -# Clean up in case it somehow succeeded -lager devenv remove --name "test_invalid_$$" --yes >/dev/null 2>&1 || true +# ============================================================ +# SECTION 4: HEADLESS BEHAVIOR +# ============================================================ +start_section "Headless Behavior" + +echo "Test 5.1: create with missing options ABORTS headless, never hangs" +# Without --mount-dir, create prompts. In a non-TTY context the prompt must +# abort promptly; hanging forever is the failure mode this pins (same +# headless-prompt class as the box_config `show` hang and the historical +# "Invalid sudoers syntax" install failure). +rm -f .lager +timeout 15 lager devenv create --image "$TEST_IMAGE" < /dev/null >/dev/null 2>&1 +RC=$? +if [ $RC -eq 124 ]; then + echo "create HUNG on a prompt in a non-TTY context (timeout hit)" + track_test "fail" +elif [ $RC -ne 0 ] && [ ! -f .lager ]; then + track_test "pass" +else + echo "expected a prompt abort; rc=$RC, .lager present: $([ -f .lager ] && echo yes || echo no)" + track_test "fail" +fi echo "" # ============================================================ # SUMMARY # ============================================================ -# Final cleanup -lager devenv remove --name "$TEST_ENV_NAME" --yes >/dev/null 2>&1 || true - print_summary exit_with_status diff --git a/test/integration/infrastructure/generic.sh b/test/integration/infrastructure/generic.sh index f1c471ae..2df0acc1 100755 --- a/test/integration/infrastructure/generic.sh +++ b/test/integration/infrastructure/generic.sh @@ -41,7 +41,7 @@ if echo "$BOX_INPUT" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'; then BOX_IP="$BOX_INPUT" echo "Detected IP address: $BOX_IP" echo "Registering as temporary box: $BOX_NAME" - lager boxes add --name "$BOX_NAME" --ip "$BOX_IP" --yes >/dev/null 2>&1 || true + lager boxes add --name "$BOX_NAME" --ip "$BOX_IP" --user lagerdata --yes >/dev/null 2>&1 || true BOX="$BOX_NAME" else # Input is a box name - use it directly @@ -90,7 +90,7 @@ lager hello --box $BOX && track_test "pass" || track_test "fail" echo "" echo "Test 1.3: Hello with invalid box (error case)" -lager hello --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 1.4: Multiple hello commands (stability test)" @@ -133,7 +133,7 @@ lager instruments --box $BOX && track_test "pass" || track_test "fail" echo "" echo "Test 2.3: Instruments with invalid box (error case)" -lager instruments --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager instruments --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 2.4: Multiple instrument listings (stability test)" @@ -199,7 +199,7 @@ lager boxes add --help && track_test "pass" || track_test "fail" echo "" echo "Test 4.2: Add a test box" -lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.100" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.100" --user testuser --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 4.3: List boxes to verify addition" @@ -211,7 +211,7 @@ fi echo "" echo "Test 4.4: Attempt to add duplicate box name (should show warning)" -if lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.200" --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then +if lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.200" --user testuser --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then track_test "pass" else track_test "fail" @@ -228,9 +228,9 @@ echo "" echo "Test 4.6: Attempt to add duplicate IP (should show warning)" # First add a new box with unique name -lager boxes add --name "${TEST_BOX_NAME}_unique" --ip "192.168.1.150" --yes >/dev/null 2>&1 || true +lager boxes add --name "${TEST_BOX_NAME}_unique" --ip "192.168.1.150" --user testuser --yes >/dev/null 2>&1 || true # Try to add another box with same IP but different name -if lager boxes add --name "${TEST_BOX_NAME}_duplicate_ip" --ip "192.168.1.150" --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then +if lager boxes add --name "${TEST_BOX_NAME}_duplicate_ip" --ip "192.168.1.150" --user testuser --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then track_test "pass" else track_test "fail" @@ -238,7 +238,7 @@ fi echo "" echo "Test 4.7: Add box with special characters in name" -lager boxes add --name "test-box_123.special" --ip "192.168.1.101" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "test-box_123.special" --ip "192.168.1.101" --user testuser --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 4.8: Boxes delete subcommand help" @@ -256,13 +256,13 @@ done echo "" echo "Test 4.10: Delete non-existent box (error case)" -lager boxes delete --name "nonexistent_box_12345" --yes 2>&1 | grep -qi "not found" && track_test "pass" || track_test "pass" +lager boxes delete --name "nonexistent_box_12345" --yes 2>&1 | grep -qi "not found" && track_test "pass" || track_test "fail" echo "" echo "Test 4.11: Add and delete multiple boxes" FAILED=0 for i in {1..3}; do - lager boxes add --name "${TEST_BOX_NAME}_${i}" --ip "192.168.1.$((100+i))" --yes >/dev/null 2>&1 || FAILED=1 + lager boxes add --name "${TEST_BOX_NAME}_${i}" --ip "192.168.1.$((100+i))" --user testuser --yes >/dev/null 2>&1 || FAILED=1 done echo "Added test boxes" lager boxes list @@ -278,7 +278,7 @@ echo "" echo "Test 4.13: Edit box IP address" # Add a test box first -lager boxes add --name "${TEST_BOX_NAME}_edit" --ip "192.168.1.50" --yes >/dev/null 2>&1 || true +lager boxes add --name "${TEST_BOX_NAME}_edit" --ip "192.168.1.50" --user testuser --yes >/dev/null 2>&1 || true # Edit its IP if lager boxes edit --name "${TEST_BOX_NAME}_edit" --ip "192.168.1.51" --yes 2>&1 | grep -q "Updated box"; then track_test "pass" @@ -295,7 +295,9 @@ echo "" echo "Test 4.14: Edit box name" # Rename the box -if lager boxes edit --name "${TEST_BOX_NAME}_edit" --new-name "${TEST_BOX_NAME}_renamed" --yes 2>&1 | grep -q "Renamed box"; then +# `boxes edit` reports every change through one "Updated box: ..." line; there +# is no separate rename message (cli/commands/box/boxes.py success_msg). +if lager boxes edit --name "${TEST_BOX_NAME}_edit" --new-name "${TEST_BOX_NAME}_renamed" --yes 2>&1 | grep -q "Updated box"; then track_test "pass" else track_test "fail" @@ -331,7 +333,9 @@ fi echo "" echo "Test 4.17: Edit with invalid IP (error case)" -if lager boxes edit --name "${TEST_BOX_NAME}_final" --ip "invalid_ip" --yes 2>&1 | grep -qi "not a valid IP"; then +# "invalid_ip" is refused for the underscore, not for failing to be an IP: +# --ip accepts a hostname too. See the note above Test 9.8. +if lager boxes edit --name "${TEST_BOX_NAME}_final" --ip "invalid_ip" --yes 2>&1 | grep -qi "invalid hostname label"; then track_test "pass" else track_test "fail" @@ -376,7 +380,7 @@ BOX_COUNT_BEFORE=$(lager boxes list 2>/dev/null | wc -l || echo "0") # Try delete-all when there might be existing boxes (from user's .lager file) if [ "$BOX_COUNT_BEFORE" -gt 1 ]; then echo "Note: .lager file contains user boxes, skipping empty delete-all test" - track_test "pass" + track_test "skip" else lager boxes delete-all --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" fi @@ -394,7 +398,7 @@ fi lager boxes delete-all --yes >/dev/null 2>&1 || true # Add test boxes for i in {1..5}; do - lager boxes add --name "deleteall_test_${i}" --ip "192.168.2.${i}" --yes >/dev/null 2>&1 || true + lager boxes add --name "deleteall_test_${i}" --ip "192.168.2.${i}" --user testuser --yes >/dev/null 2>&1 || true done DELETEALL_COUNT=$(lager boxes list 2>/dev/null | grep -c "deleteall_test" || echo "0") echo "Added $DELETEALL_COUNT test boxes for delete-all test" @@ -481,7 +485,7 @@ echo "$DEFAULTS_BEFORE" echo "" echo "Test 6.3: Add box to saved boxes first" -lager boxes add --name "test_default_box" --ip "$BOX" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "test_default_box" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 6.4: Set default box" @@ -499,7 +503,7 @@ fi echo "" echo "Test 6.6: Set default with non-existent box (error case)" -if lager defaults add --box "INVALID_BOX_12345" 2>&1 | grep -qi "does not exist in saved boxes"; then +if lager defaults add --box "INVALID_BOX_12345" 2>&1 | grep -qi "not found in saved boxes"; then track_test "pass" else track_test "fail" @@ -553,7 +557,7 @@ done echo "" echo "Test 7.5: Set both box and serial port at once" -lager boxes add --name "test_combo_default" --ip "192.168.1.123" --yes >/dev/null 2>&1 +lager boxes add --name "test_combo_default" --ip "192.168.1.123" --user testuser --yes >/dev/null 2>&1 if lager defaults add --box "test_combo_default" --serial-port "/dev/ttyUSB2" 2>&1 | grep -q "Set defaults"; then track_test "pass" else @@ -572,7 +576,7 @@ echo "========================================================================" echo "" echo "Test 8.1: Add test box for persistence tests" -lager boxes add --name "test_persist_box" --ip "$BOX" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "test_persist_box" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 8.2: Set default box and verify persistence" @@ -614,7 +618,7 @@ echo "========================================================================" echo "" echo "Test 9.1: Empty box name in lager boxes add" -if lager boxes add --name "" --ip "192.168.1.100" --yes 2>&1 | grep -qi "name cannot be empty"; then +if lager boxes add --name "" --ip "192.168.1.100" --user testuser --yes 2>&1 | grep -qi "name cannot be empty"; then track_test "pass" else track_test "fail" @@ -622,7 +626,7 @@ fi echo "" echo "Test 9.2: Whitespace-only box name" -if lager boxes add --name " " --ip "192.168.1.100" --yes 2>&1 | grep -qi "name cannot be empty"; then +if lager boxes add --name " " --ip "192.168.1.100" --user testuser --yes 2>&1 | grep -qi "name cannot be empty"; then track_test "pass" else track_test "fail" @@ -630,39 +634,47 @@ fi echo "" echo "Test 9.3: Empty box name in lager hello" -lager hello --box "" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 9.4: Very long box name" LONG_BOX_NAME=$(printf 'a%.0s' {1..500}) -lager hello --box "$LONG_BOX_NAME" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "$LONG_BOX_NAME" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 9.5: Box name with special characters" -lager hello --box "test@#$%^&*()" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "test@#$%^&*()" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 9.6: Box name with spaces" -lager hello --box "test box with spaces" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "test box with spaces" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" # The name mixes BMP (CJK) and non-BMP (U+1D400, 4-byte UTF-8) codepoints on # purpose: they exercise different encoding paths, and a name that is only CJK # would not catch a surrogate-pair bug. echo "Test 9.7: Box name with Unicode characters" -lager hello --box "test_设备_𝐀" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "test_设备_𝐀" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" -echo "Test 9.8: Invalid IP address format in box add" -if lager boxes add --name "test_invalid_ip" --ip "999.999.999.999" --yes 2>&1 | grep -qi "not a valid IP"; then +# --ip takes an IP address OR a DNS hostname (cli/address_utils.py +# validate_ip_or_hostname), so anything shaped like a hostname is accepted by +# design: "999.999.999.999", "192.168.1.256" and "string" are all syntactically +# valid hostname labels and are NOT rejected. These six checks therefore cover +# the inputs the validator genuinely refuses. Do not "restore" assertions that +# expect an IP-only contract -- that contract does not exist, and asserting it +# only produces failures that say nothing about the product. + +echo "Test 9.8: URL rejected in box add" +if lager boxes add --name "test_url_ip" --ip "http://box.example.com" --user testuser --yes 2>&1 | grep -qi "looks like a URL"; then track_test "pass" else track_test "fail" fi echo "" -echo "Test 9.9: Malformed IP address" -if lager boxes add --name "test_malformed" --ip "not.an.ip.address" --yes 2>&1 | grep -qi "not a valid IP"; then +echo "Test 9.9: Address with a path rejected" +if lager boxes add --name "test_path_ip" --ip "box.example.com/path" --user testuser --yes 2>&1 | grep -qi "contains a path"; then track_test "pass" else track_test "fail" @@ -670,7 +682,7 @@ fi echo "" echo "Test 9.10: Empty IP address" -if lager boxes add --name "test_empty_ip" --ip "" --yes 2>&1 | grep -qi "IP.*cannot be empty"; then +if lager boxes add --name "test_empty_ip" --ip "" --user testuser --yes 2>&1 | grep -qi "cannot be empty"; then track_test "pass" else track_test "fail" @@ -678,23 +690,23 @@ fi echo "" echo "Test 9.11: IP address with whitespace" -if lager boxes add --name "test_whitespace_ip" --ip " " --yes 2>&1 | grep -qi "IP.*cannot be empty"; then +if lager boxes add --name "test_whitespace_ip" --ip " " --user testuser --yes 2>&1 | grep -qi "cannot be empty"; then track_test "pass" else track_test "fail" fi echo "" -echo "Test 9.12: String instead of IP address" -if lager boxes add --name "test_string_ip" --ip "string" --yes 2>&1 | grep -qi "not a valid IP"; then +echo "Test 9.12: Address carrying an explicit port rejected" +if lager boxes add --name "test_port_ip" --ip "192.168.1.100:8080" --user testuser --yes 2>&1 | grep -qi "contains a port"; then track_test "pass" else track_test "fail" fi echo "" -echo "Test 9.13: IP address with invalid octets" -if lager boxes add --name "test_octet_ip" --ip "192.168.1.256" --yes 2>&1 | grep -qi "not a valid IP"; then +echo "Test 9.13: Invalid hostname label rejected" +if lager boxes add --name "test_label_ip" --ip "bad_label.example.com" --user testuser --yes 2>&1 | grep -qi "invalid hostname label"; then track_test "pass" else track_test "fail" @@ -702,7 +714,7 @@ fi echo "" echo "Test 9.14: Box operations without required parameters" -lager boxes add 2>&1 | grep -qi "error\|missing" && track_test "pass" || track_test "pass" +lager boxes add 2>&1 | grep -qi "error\|missing" && track_test "pass" || track_test "fail" echo "" echo "Test 9.15: Defaults add without any parameters" @@ -714,7 +726,7 @@ fi echo "" echo "Test 9.16: Valid IPv4 address acceptance" -if lager boxes add --name "test_valid_ipv4" --ip "192.168.1.100" --yes 2>&1 | grep -q "Added box"; then +if lager boxes add --name "test_valid_ipv4" --ip "192.168.1.100" --user testuser --yes 2>&1 | grep -q "Added box"; then track_test "pass" lager boxes delete --name "test_valid_ipv4" --yes >/dev/null 2>&1 || true else @@ -723,7 +735,7 @@ fi echo "" echo "Test 9.17: Valid IPv6 address acceptance" -if lager boxes add --name "test_valid_ipv6" --ip "2001:0db8:85a3:0000:0000:8a2e:0370:7334" --yes 2>&1 | grep -q "Added box"; then +if lager boxes add --name "test_valid_ipv6" --ip "2001:0db8:85a3:0000:0000:8a2e:0370:7334" --user testuser --yes 2>&1 | grep -q "Added box"; then track_test "pass" lager boxes delete --name "test_valid_ipv6" --yes >/dev/null 2>&1 || true else @@ -732,7 +744,7 @@ fi echo "" echo "Test 9.18: Localhost IP acceptance" -if lager boxes add --name "test_localhost" --ip "127.0.0.1" --yes 2>&1 | grep -q "Added box"; then +if lager boxes add --name "test_localhost" --ip "127.0.0.1" --user testuser --yes 2>&1 | grep -q "Added box"; then track_test "pass" lager boxes delete --name "test_localhost" --yes >/dev/null 2>&1 || true else @@ -759,7 +771,7 @@ else TEST_COMBO_IP="192.168.1.100" fi FAILED=0 -lager boxes add --name "${TEST_BOX_NAME}_combo" --ip "$TEST_COMBO_IP" --yes >/dev/null 2>&1 || FAILED=1 +lager boxes add --name "${TEST_BOX_NAME}_combo" --ip "$TEST_COMBO_IP" --user testuser --yes >/dev/null 2>&1 || FAILED=1 # Only test set/hello if we used the real BOX IP if [ "$TEST_COMBO_IP" = "$BOX" ]; then lager defaults add --box "${TEST_BOX_NAME}_combo" >/dev/null 2>&1 || FAILED=1 @@ -770,7 +782,7 @@ lager boxes delete --name "${TEST_BOX_NAME}_combo" --yes >/dev/null 2>&1 || FAIL echo "" echo "Test 10.2: Interleaved list/add operations" -lager boxes add --name "test_interleave" --ip "$BOX" --yes >/dev/null 2>&1 +lager boxes add --name "test_interleave" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 FAILED=0 lager defaults >/dev/null 2>&1 || FAILED=1 lager defaults add --box "test_interleave" >/dev/null 2>&1 || FAILED=1 @@ -838,7 +850,7 @@ echo "" echo "Test 11.4: Rapid box add/delete cycles (20 iterations)" FAILED=0 for i in {1..20}; do - lager boxes add --name "stress_box_${i}" --ip "192.168.100.${i}" --yes >/dev/null 2>&1 || FAILED=1 + lager boxes add --name "stress_box_${i}" --ip "192.168.100.${i}" --user testuser --yes >/dev/null 2>&1 || FAILED=1 lager boxes delete --name "stress_box_${i}" --yes >/dev/null 2>&1 || FAILED=1 done [ $FAILED -eq 0 ] && track_test "pass" || track_test "fail" @@ -878,7 +890,7 @@ lager hello --box "$BOX" >/dev/null && track_test "pass" || track_test "fail" echo "" echo "Test 12.3: Verify box list after failed add" -lager boxes add --name "" --ip "" --yes >/dev/null 2>&1 || true +lager boxes add --name "" --ip "" --user testuser --yes >/dev/null 2>&1 || true lager boxes list >/dev/null && track_test "pass" || track_test "fail" echo "" @@ -888,7 +900,7 @@ lager defaults >/dev/null && track_test "pass" || track_test "fail" echo "" echo "Test 12.5: Verify configuration consistency after multiple operations" -lager boxes add --name "test_regression" --ip "$BOX" --yes >/dev/null 2>&1 +lager boxes add --name "test_regression" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 DEFAULTS_START=$(lager defaults 2>&1) lager defaults add --box "test_regression" >/dev/null 2>&1 || true lager hello --box "$BOX" >/dev/null 2>&1 diff --git a/test/integration/infrastructure/nets.sh b/test/integration/infrastructure/nets.sh index b092d225..b1985ab4 100755 --- a/test/integration/infrastructure/nets.sh +++ b/test/integration/infrastructure/nets.sh @@ -101,7 +101,11 @@ fi echo "" echo "Test 1.2: Nets command help" -if lager nets --help 2>&1 | grep -q 'List all saved nets'; then +# Match the durable part of the group help ("... saved nets"), not the full +# sentence: the wording has already drifted once (it now reads "List and +# manage saved nets") and a whole-sentence match makes this a copy-editing +# tripwire rather than a test that the command exists and describes itself. +if lager nets --help 2>&1 | grep -q 'saved nets'; then echo -e "${GREEN}[OK]${NC}" track_test "pass" else diff --git a/test/integration/infrastructure/python.sh b/test/integration/infrastructure/python.sh index e70016a0..4a47dd72 100755 --- a/test/integration/infrastructure/python.sh +++ b/test/integration/infrastructure/python.sh @@ -12,7 +12,11 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" source "${SCRIPT_DIR}/../../framework/colors.sh" source "${SCRIPT_DIR}/../../framework/harness.sh" -DOCKER_IMAGE="${DOCKER_IMAGE:?Set DOCKER_IMAGE to your lagerbox image tag}" +# Optional: only the two `lager python --image` checks need it, and a hard +# `:?` guard here aborted the whole suite before a single test ran -- which is +# why none of the ~100 checks below had ever executed anywhere. Unset means +# those two checks report as skipped; everything else runs. +DOCKER_IMAGE="${DOCKER_IMAGE:-}" set +e # DON'T exit on error - we want to track failures @@ -22,6 +26,41 @@ init_harness # Safety delay between tests (seconds) TEST_DELAY=0.5 +# Kill checks below use `--kill-all`, not `--kill`. `--kill` takes a process +# ID (`--kill TEXT`); passing it none made click consume the following +# `--signal` as its argument and then reject the signal name as a script path. +# Every kill check in this suite was malformed that way, which only became +# visible once the suite actually ran. +# +# Both the launch and the kill are bounded. The first CI run of this section +# spent over nine minutes here and was killed by the step timeout with the +# infinite loop it had started still running on the box; the suites after it +# then ran against a box that was still churning. Bounding turns that into a +# visible result for the check that caused it instead of damage charged to +# whatever ran next. +KILL_TIMEOUT=45 +LAUNCH_TIMEOUT=30 + +# Bound EVERY lager call in this suite, not just the ones already known to +# hang. Two separate paths have now stalled indefinitely on the bench -- +# a detached launch that never returned, and `--timeout 3` against a 30s +# script that ran for seventeen minutes -- and each one cost a full bench +# round to find because an unbounded hang looks identical to a slow suite +# until the step timeout kills it. With this wrapper a hang becomes a failed +# check that names itself, and the suites after this one stop inheriting a +# box that is still busy. +# +# Resolve the real binary first: calling `lager` inside the function would +# recurse. The ceiling is far above any legitimate operation here (the +# longest deliberate wait in this file is --timeout 10). +LAGER_CALL_TIMEOUT="${LAGER_CALL_TIMEOUT:-90}" +LAGER_BIN="$(command -v lager)" +if [ -z "$LAGER_BIN" ]; then + echo "lager CLI not found on PATH; install it first (pip install -e cli/)" >&2 + exit 1 +fi +lager() { timeout "$LAGER_CALL_TIMEOUT" "$LAGER_BIN" "$@"; } + # Check if required arguments are provided if [ $# -lt 1 ]; then echo "Usage: $0 " @@ -168,7 +207,7 @@ if lager python --box $BOX --image "nonexistent:tag999" -c "print('test')" 2>&1 track_test "pass" else echo -e "${YELLOW}[WARNING] Invalid image may not be validated upfront${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -185,7 +224,7 @@ echo "" sleep $TEST_DELAY echo "Test 2.6: Invalid --signal value" -if lager python --box $BOX --kill --signal invalid_signal 2>&1 | grep -qi "error\|invalid"; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal invalid_signal 2>&1 | grep -qi "error\|invalid"; then echo -e "${GREEN}[OK] Invalid signal caught${NC}" track_test "pass" else @@ -201,18 +240,18 @@ if lager python --box $BOX -p "invalid:port:format" -c "print('test')" 2>&1 | gr track_test "pass" else echo -e "${YELLOW}[WARNING] Port format may not be validated upfront${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY -echo "Test 2.8: --kill without running script" -if lager python --box $BOX --kill 2>&1 | grep -qi "error\|not running\|no script"; then +echo "Test 2.8: --kill-all without running script" +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all 2>&1 | grep -qi "error\|not running\|no script"; then echo -e "${GREEN}[OK] Kill without script handled${NC}" track_test "pass" else echo -e "${YELLOW}[WARNING] Kill without script may be silently ignored${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -542,7 +581,7 @@ if lager python "$TEST_DIR/no_overwrite.py" --box $BOX --download no_overwrite.t track_test "pass" else echo -e "${YELLOW}[WARNING] Overwrite may not be prevented${NC}" - track_test "pass" + track_test "skip" fi rm -f no_overwrite.txt echo "" @@ -585,7 +624,10 @@ echo "" sleep $TEST_DELAY echo "Test 6.2: Specify custom Docker image (using default)" -if lager python --box $BOX --image "$DOCKER_IMAGE" -c "print('Custom image')" 2>&1 | grep -q "Custom image"; then +if [ -z "$DOCKER_IMAGE" ]; then + echo -e "${YELLOW}[SKIP] DOCKER_IMAGE not set${NC}" + track_test "skip" +elif lager python --box $BOX --image "$DOCKER_IMAGE" -c "print('Custom image')" 2>&1 | grep -q "Custom image"; then echo -e "${GREEN}[OK] Custom image specified${NC}" track_test "pass" else @@ -676,7 +718,7 @@ if lager python "$TEST_DIR/long_running.py" --box $BOX -d 2>&1 | grep -qi "detac track_test "pass" else echo -e "${YELLOW}[SKIP] Detached mode may not show confirmation${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -697,6 +739,11 @@ echo "========================================================================" echo "" sleep $TEST_DELAY +# This section starts an INFINITE loop on the box and then asks the CLI to +# kill it, so it is where an unbounded call does the most damage. Timeouts +# come from the top of the file; the cleanup at the end of the section does +# not depend on `--kill` having worked. + echo "Test 9.1: Kill running script (--kill with default signal)" cat > "$TEST_DIR/infinite.py" <<'EOF' import time @@ -705,57 +752,63 @@ while True: time.sleep(1) EOF # Start in background -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill 2>&1 | grep -qi "killed\|terminated\|stopped"; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all 2>&1 | grep -qi "killed\|terminated\|stopped"; then echo -e "${GREEN}[OK] Script killed${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] Kill may have succeeded without confirmation${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] Kill did not confirm within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY echo "Test 9.2: Kill with SIGINT signal" -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill --signal sigint 2>&1; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigint 2>&1; then echo -e "${GREEN}[OK] SIGINT sent${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] SIGINT may have been sent${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] SIGINT not confirmed within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY echo "Test 9.3: Kill with SIGTERM signal" -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill --signal sigterm 2>&1; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigterm 2>&1; then echo -e "${GREEN}[OK] SIGTERM sent${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] SIGTERM may have been sent${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] SIGTERM not confirmed within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY echo "Test 9.4: Kill with SIGKILL signal" -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill --signal sigkill 2>&1; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigkill 2>&1; then echo -e "${GREEN}[OK] SIGKILL sent${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] SIGKILL may have been sent${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] SIGKILL not confirmed within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY +# Leave no infinite loop behind, whatever the four checks above concluded. +# Not a tracked check: this is cleanup, and its failure is reported by the +# suites that follow rather than counted here. +echo "Section 9 cleanup: stop any script still running on the box" +timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigkill >/dev/null 2>&1 || true + # ============================================================ # SECTION 10: TIMEOUT OPERATIONS # ============================================================ @@ -789,7 +842,7 @@ if lager python "$TEST_DIR/slow.py" --box $BOX --timeout 3 2>&1 | grep -qi "time track_test "pass" else echo -e "${YELLOW}[SKIP] Timeout may have triggered without message${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -800,7 +853,7 @@ if lager python --box $BOX --timeout 1 -c "import time; time.sleep(10); print('s track_test "pass" else echo -e "${YELLOW}[SKIP] Short timeout may have triggered${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -811,7 +864,7 @@ if lager python --box $BOX --timeout 0 -c "print('test')" 2>&1 | grep -qi "error track_test "pass" else echo -e "${YELLOW}[SKIP] Zero timeout may use default${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -822,7 +875,7 @@ if lager python --box $BOX --timeout -1 -c "print('test')" 2>&1 | grep -qi "erro track_test "pass" else echo -e "${YELLOW}[SKIP] Negative timeout may be rejected${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -907,7 +960,10 @@ echo "" sleep $TEST_DELAY echo "Test 11.5: Custom image with environment variables" -if lager python --box $BOX --image "$DOCKER_IMAGE" --env CUSTOM=test -c "import os; print(os.environ.get('CUSTOM'))" 2>&1 | grep -q "test"; then +if [ -z "$DOCKER_IMAGE" ]; then + echo -e "${YELLOW}[SKIP] DOCKER_IMAGE not set${NC}" + track_test "skip" +elif lager python --box $BOX --image "$DOCKER_IMAGE" --env CUSTOM=test -c "import os; print(os.environ.get('CUSTOM'))" 2>&1 | grep -q "test"; then echo -e "${GREEN}[OK] Custom image + env works${NC}" track_test "pass" else @@ -998,7 +1054,10 @@ echo "" sleep $TEST_DELAY echo "Test 12.5: Unicode and UTF-8" -if lager python --box $BOX -c "print('Hello 世界 🌍')" 2>&1 | grep -q "世界"; then +# Mixes BMP (CJK, 3-byte) and non-BMP (U+1D400, 4-byte) codepoints, the same +# pair generic.sh and nets.sh use: they exercise different encoding paths, and +# a string that is only CJK would not catch a surrogate-pair bug. +if lager python --box $BOX -c "print('Hello 世界 𝐀')" 2>&1 | grep -q "世界"; then echo -e "${GREEN}[OK] Unicode support works${NC}" track_test "pass" else @@ -1051,7 +1110,7 @@ if lager python --box $BOX --add-file "$TEST_DIR/large_file.bin" -c "import os; track_test "pass" else echo -e "${YELLOW}[WARNING] Large file may not have uploaded correctly${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1250,7 +1309,7 @@ if lager python --box $BOX -c "" 2>&1; then track_test "pass" else echo -e "${YELLOW}[SKIP] Empty script may cause error${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1261,7 +1320,7 @@ if lager python --box $BOX -c " " 2>&1; then track_test "pass" else echo -e "${YELLOW}[SKIP] Whitespace-only script may cause error${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1288,7 +1347,7 @@ if lager python --box $BOX -c "$LONG_SCRIPT" >/dev/null 2>&1; then track_test "pass" else echo -e "${YELLOW}[SKIP] Very long command may be truncated${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1358,7 +1417,7 @@ echo "" sleep $TEST_DELAY # Make sure no scripts are still running -lager python --box $BOX --kill >/dev/null 2>&1 || true +timeout $KILL_TIMEOUT lager python --box $BOX --kill-all >/dev/null 2>&1 || true # ============================================================ # TEST SUMMARY diff --git a/tools/bench_suite_gate.sh b/tools/bench_suite_gate.sh new file mode 100644 index 00000000..3ddd4df9 --- /dev/null +++ b/tools/bench_suite_gate.sh @@ -0,0 +1,121 @@ +#!/bin/bash +# Copyright 2024-2026 Lager Data +# SPDX-License-Identifier: Apache-2.0 +# +# Two-sided baseline ratchet for one bench suite. +# +# The bench integration suites are wired into CI carrying known failures. +# A suite that simply "must pass" would either sit permanently red (a check +# people learn to ignore) or force the failures to be softened away. Instead +# each suite declares how many checks are expected to fail today, and this +# gate gets stricter automatically: +# +# failures > baseline -> ERROR, exit 1. New breakage beyond the known gap. +# failures == baseline -> warning. Known gap, run stays green. +# failures < baseline -> notice telling the reader to lower the baseline +# and lock the improvement in. +# +# Every non-zero baseline must name an owning issue where it is set. A number +# with no issue behind it is softening, not baselining. +# +# Usage: bench_suite_gate.sh