From 6904d4b7f19a8fa76801871246b63b8fd62dd672 Mon Sep 17 00:00:00 2001 From: DanielRMErskine Date: Thu, 13 Aug 2026 17:26:33 -0700 Subject: [PATCH] Run the infrastructure bench suites weekly, and fix what they found Of the 38 bash integration suites, CI executed exactly one; the rest were syntax-checked but never run. New "Bench: Extended" (bench-extended.yml), Saturday 14:17 UTC, runs the infrastructure suites -- box-level tests needing NO instruments, so the workflow carries none of integration-tests.yml's relay/firmware machinery and the bench stays dark throughout. Leaf workflow on the shared hardware-ci concurrency group; no pull_request trigger ever (public repo, self-hosted runner); failure notifies the shared bench-alert issue, and there is deliberately NO recovery job -- closing that alert is exclusively the nightly's signal. Five of the seven suites are wired. boxes_config.sh is not: its discovery section inventories every box reachable from the runner's network and prints the results, and a full network inventory does not belong in a CI log. It stays runnable by hand; wiring it needs a CI-safe output mode (counts, not names) first. python.sh is not wired either: 53 of its checks call `lager python -c` for inline execution and there is no `-c` option, so it fails 43 of 84. Three more pass `--image`, also gone. It needs the rewrite devenv.sh already had, against the current interface, and that is its own change. Baselining 43 would record "calls an option that does not exist" as the expected state. Each suite declares a baseline -- the number of its checks known to fail today -- and tools/bench_suite_gate.sh ratchets two-sided: more failures than the baseline is red, fewer prints a notice telling you to lower it. That is what lets this land green without softening a single assertion. Every non-zero baseline names the issue that owns it. The gate also fails when a suite produces no summary at all, which is what a crash or a step timeout looks like; that case is not a pass. Reading the gate matters, because the step ticks lie: a continue-on-error step shows green whether its suite passed, failed, or timed out. WHAT THE RUNS FOUND Test defects, fixed here: - `lager boxes add` now requires --user; 33 call sites predated that. - deployment.sh read the removed `lager duts` registry in three spots and called `lager duts --help`, which is gone -- and its `set -e` meant the first failure killed the run before its own summary printed, hiding every later result. Now `lager boxes`, `lager dut`, `set +e` to match every other suite in the directory. - python.sh had a hard DOCKER_IMAGE guard on line 15 that aborted the whole suite before a single check ran, for the sake of two optional checks. That is why none of its ~100 checks had ever run anywhere. - devenv.sh tested the removed named-environment CLI; rewritten against the current per-project model. 13/13. - generic.sh asserted that `boxes add --ip` rejects things like "999.999.999.999" and "not.an.ip.address". It does not, by design: --ip takes an IP *or* a DNS hostname, and those are valid hostname syntax. Those checks now cover what the validator genuinely refuses: URL, path, port, empty, invalid label. - Assertions pinned to strings the CLI no longer prints: `nets --help`, `boxes edit`, `defaults add`. - deployment.sh inspected containers named `controller` and `python`. There is one container, named `lager`, so both checks failed on every box -- a stale expectation reported as a box fault. - python.sh called `lager python --kill` with no process ID in eight places. --kill takes a PID; killing everything is --kill-all. Passing neither made click consume the following --signal as the PID and then reject the signal name as a script path, so every kill check in the suite was malformed. Product behaviour the suites exposed, filed separately: `lager hello` failing under sustained repetition (#273), and `box config` sysctl and cargo entries not applying or clearing (#275). python.sh additionally surfaced `--timeout` never firing and a detached launch that does not return (#274). Three of the five suites end up expecting zero failures. The two that do not each name the issue that owns them. HANGS AND THE BOX LOCK Two CLI paths stalled indefinitely on the bench. Bounding them one at a time cost a bench round each, so python.sh now wraps `lager` in a shell function that bounds every call. A hang is a failed check that names itself instead of a suite that stalls until its step timeout. That mattered more than it first appeared. The CLI releases its auto-lock three ways -- try/finally, atexit, and a SIGTERM/SIGINT/SIGHUP handler -- but a step timeout ends in SIGKILL, which cannot be caught, so the box-side TTL (1800s) is the only backstop. An expired lock is reaped without --force; an unexpired one blocks the next run outright, a real collision for back-to-back runs that cost two rounds here. This workflow force-releases before its connectivity check, matching the call update-regression.yml already makes for its own leak path. Forcing is safe because the hardware-ci concurrency group serialises bench workflows against this box. ALSO static-checks now shellchecks and syntax-checks tools/ as well as test/. Those scripts are CI's own machinery -- one files public issues, one gates these suites -- and they were never linted. They pass with no exclusions. --- .github/workflows/README.md | 1 + .github/workflows/bench-extended.yml | 403 ++++++++++++++++++ .github/workflows/static-checks.yml | 10 +- test/integration/infrastructure/box_config.sh | 27 +- .../infrastructure/boxes_config.sh | 10 +- test/integration/infrastructure/deployment.sh | 61 +-- test/integration/infrastructure/devenv.sh | 163 ++++--- test/integration/infrastructure/generic.sh | 102 +++-- test/integration/infrastructure/nets.sh | 6 +- test/integration/infrastructure/python.sh | 133 ++++-- tools/bench_suite_gate.sh | 121 ++++++ 11 files changed, 860 insertions(+), 177 deletions(-) create mode 100644 .github/workflows/bench-extended.yml create mode 100644 tools/bench_suite_gate.sh diff --git a/.github/workflows/README.md b/.github/workflows/README.md index bca786ec..38832869 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -18,6 +18,7 @@ deleted.) | `integration-tests.yml` | Bench: Integration Tests | push to `main`, `workflow_call`, dispatch | self-hosted `lager-bench` | Drives every bench instrument through `lager python`, plus the J-Link CLI suite | | `update-regression.yml` | Bench: Box Lifecycle | `workflow_call`, dispatch | self-hosted `lager-bench` | Downgrade -> update -> no-op -> forced rebuild -> uninstall -> install, with a hardware smoke per phase | | `nightly-bench.yml` | Bench: Nightly | cron 10:17 UTC, dispatch | (calls the two above) | Nightly ordering wrapper: lifecycle first, instruments only if it succeeded; files/closes the `bench-alert` issue | +| `bench-extended.yml` | Bench: Extended | cron Sat 14:17 UTC, dispatch | self-hosted `lager-bench` | Weekly run of five of the seven infrastructure integration suites (no instruments; bench stays dark); alerts on failure, never closes the alert | | `bench-watchdog.yml` | Bench: Watchdog | cron every 6h at :41, dispatch | `ubuntu-latest` | Alerts when nightly runs stop FLOWING (queued too long, stuck, cron dead) — the failure the notify jobs cannot see | | `unit-tests.yml` | PR Gate: Unit Tests | `pull_request`, push to `main`, dispatch | `ubuntu-latest` | Unit suites (one pytest process per suite) + Python-version compat matrix | | `static-checks.yml` | PR Gate: Static Checks | `pull_request`, push to `main`, dispatch | `ubuntu-latest` | Syntax/lint floors over the tree the unit gate cannot reach, plus a coverage report | diff --git a/.github/workflows/bench-extended.yml b/.github/workflows/bench-extended.yml new file mode 100644 index 00000000..404de33a --- /dev/null +++ b/.github/workflows/bench-extended.yml @@ -0,0 +1,403 @@ +# Copyright 2024-2026 Lager Data +# SPDX-License-Identifier: Apache-2.0 +# +# Weekly execution for bench tests that exist but run nowhere. Of the 38 bash +# integration suites, nightly CI executes exactly one (jlink_script.sh); the +# other 37 were syntax-checked and shellchecked but never run, so a real +# regression in the CLI surface they cover -- argument parsing, exit codes, +# box-level commands -- surfaced only when a human happened to run one against +# a bench. This workflow starts working that debt down. +# +# WAVE 1 (this file): five of the seven test/integration/infrastructure/ +# suites -- deployment, devenv, nets, box_config and generic. They need a box +# but NO instruments, so this workflow has none of integration-tests.yml's +# relay power, relay-net self-heal, or firmware staging, and the bench stays +# dark for the whole run. +# +# The other two are deliberately excluded, each for its own reason recorded +# beside the suite steps below: boxes_config.sh would print a network +# inventory into a public log, and python.sh tests a CLI surface that no +# longer exists. Wave 2 (test/api/ scripts the existing instruments support) +# is follow-up work and will need the instrument steps when it lands. +# +# Ordering inside the run is load-bearing: +# - generic.sh runs LAST: it exercises `lager boxes delete-all` against the +# runner's own lager config, which can remove this workflow's box +# registration; everything after it would fail to resolve the box. +# - box_config.sh runs late: it exercises box-config apply including the +# container bounce + rollback path, so the box may be briefly degraded. +# - A best-effort re-registration step follows the suites either way: these +# scripts mutate the RUNNER's persistent lager config by design, and +# every bench workflow re-registers its box at start, so damage is +# bounded to one run -- but healing it here keeps this run's own tail +# steps working too. +# +# Weekly, Saturday 14:17 UTC: clear of the nightly's worst case (10:17 UTC +# start + ~2.5h), off the hour for GitHub's scheduled-queue congestion, and +# serialized against the other bench workflows by the shared concurrency +# group below. This is a LEAF workflow (no workflow_call children), so the +# caller-holds-the-slot deadlock documented in nightly-bench.yml does not +# apply here. +# +# Security: this repo is PUBLIC and the runner drives real hardware, so this +# workflow deliberately does NOT trigger on `pull_request` -- a fork PR must +# not be able to execute code on the bench. To bench-test a branch, push it +# to this repo and workflow_dispatch on that ref. +# +# Output surface: several CLI errors print an "Available boxes:" list, so this +# log carries whatever the RUNNER's own box registry holds. That is safe only +# because the runner registers exactly one box and its address is a repository +# secret, which the runner masks. Registering anything else on the runner +# turns those error paths into a fleet listing in a public log -- keep the +# runner's registry to the CI box. +# +# Notification: a red run files/updates the single `bench-alert` issue via +# tools/bench_alert.sh. There is deliberately NO recovery job here: closing +# the alert issue is exclusively the nightly's signal, because a green weekly +# must never mask a nightly that is still failing. + +name: "Bench: Extended" + +on: + schedule: + - cron: "17 14 * * 6" + workflow_dispatch: + inputs: + box_name: + description: "Lager box name (overrides vars.LAGER_BOX)" + required: false + type: string + +permissions: + contents: read + +concurrency: + # One physical bench: queue behind nightly/integration, never collide. + group: hardware-ci-${{ vars.LAGER_BOX || 'MASTER' }} + cancel-in-progress: false + +jobs: + extended: + name: Run infrastructure suites on the bench + runs-on: [self-hosted, lager-bench] + timeout-minutes: 75 + env: + LAGER_BOX: ${{ inputs.box_name || vars.LAGER_BOX || 'MASTER' }} + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + # No job here pushes anything; do not persist the token into + # the checkout's config (zizmor: artipacked). + persist-credentials: false + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 + with: + python-version: '3.11' + + - name: Install lager CLI (editable, from this repo) + run: pip install -e cli/ + shell: bash + + - name: Sign in to the box access gateway + env: + LAGER_AUTH_URL: ${{ vars.LAGER_AUTH_URL }} + LAGER_CI_EMAIL: ${{ secrets.LAGER_CI_EMAIL }} + LAGER_CI_PASSWORD: ${{ secrets.LAGER_CI_PASSWORD }} + run: lager login "$LAGER_AUTH_URL" --email "$LAGER_CI_EMAIL" --password "$LAGER_CI_PASSWORD" + shell: bash + + - name: Register lager box + run: | + lager boxes add \ + --name "$LAGER_BOX" \ + --ip "${{ secrets.LAGER_BOX_IP }}" \ + --user lagerdata \ + --yes + shell: bash + + - name: Set up lager SSH key + # Same identity mapping integration-tests.yml uses: box_config.sh and + # deployment.sh ssh into the box directly with the default identity. + run: | + if [ ! -f ~/.ssh/lager_box ]; then + ssh-keygen -t ed25519 -f ~/.ssh/lager_box -N "" -C "lager-box-access" + fi + grep -qF "$(cat ~/.ssh/lager_box.pub)" ~/.ssh/authorized_keys 2>/dev/null || \ + cat ~/.ssh/lager_box.pub >> ~/.ssh/authorized_keys + touch ~/.ssh/config && chmod 600 ~/.ssh/config + if ! grep -q "^# lager integration CI box identity" ~/.ssh/config; then + { + echo "# lager integration CI box identity" + echo "Host ${{ secrets.LAGER_BOX_IP }}" + echo " IdentityFile ~/.ssh/lager_box" + echo "Host $LAGER_BOX" + echo " HostName ${{ secrets.LAGER_BOX_IP }}" + echo " IdentityFile ~/.ssh/lager_box" + } >> ~/.ssh/config + fi + shell: bash + + - name: Release a still-held box lock from a killed run + # The CLI releases its auto-lock three ways (try/finally, atexit, and + # a SIGTERM/SIGINT/SIGHUP handler), but a step timeout ends in + # SIGKILL, which cannot be caught. The client-side release never + # runs, and the box-side TTL becomes the only backstop -- 1800s by + # default (cli/box_storage.py _DEFAULT_LOCK_TTL_SECONDS). An expired + # lock is reaped automatically and needs no --force; an unexpired one + # blocks the next run outright. + # + # That is a real collision for back-to-back runs -- dispatches during + # a debugging session, or a re-run after a timeout, land well inside + # 30 minutes. It is NOT a threat to the schedule: the nightly and + # this weekly are hours apart, so any leaked lock has long expired. + # + # Forcing is safe here because the hardware-ci concurrency group + # serialises every bench workflow against this box, so a lock still + # held when this job starts belongs to a run that is no longer + # running. update-regression.yml carries the same call for its own + # leak path (a lost release POST rather than a signal). + run: lager boxes unlock --box "$LAGER_BOX" --force 2>/dev/null || true + shell: bash + + - name: Verify box connectivity + # First authenticated contact after a fresh login can fail exactly + # once while the gateway records the box<->auth-server link. + run: lager hello --box "$LAGER_BOX" || lager hello --box "$LAGER_BOX" + shell: bash + + # Each suite is continue-on-error with an id so every suite runs and + # the aggregation gate at the end turns any failure into a job failure + # -- same pattern as integration-tests.yml, same reasoning. + # + # Each suite also declares a BASELINE: the number of its checks known to + # fail today. tools/bench_suite_gate.sh ratchets two-sided -- more + # failures than the baseline is red, fewer prints a notice telling you to + # lower it. This is what lets the workflow land green on day one without + # softening a single assertion. Every non-zero baseline names the issue + # that owns it; a bare number with no issue behind it is softening. + # + # The suite's own exit status is deliberately dropped (`|| true`): it + # carries the same information as the failure count, and reading it + # through `tee` is how a gate ends up reporting tee's status. The gate + # fails loudly if a suite produces no summary at all, which is what a + # crash or a step timeout looks like. + + - name: Suite deployment.sh (box verification, no instruments) + id: suite-deployment + continue-on-error: true + timeout-minutes: 10 + env: + # The suite defaults to ssh user `lager`; this bench logs in as + # `lagerdata`. + SSH_USER: lagerdata + # Expected clean. Its two failures were a stale expectation: the + # suite inspected containers named `controller` and `python`, which + # no longer exist -- there is one container, named `lager`. Fixed + # rather than baselined. + DEPLOYMENT_BASELINE: "0" + run: | + LOG="$RUNNER_TEMP/deployment.log" + bash test/integration/infrastructure/deployment.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh deployment deployment.sh "$DEPLOYMENT_BASELINE" "$LOG" + shell: bash + + # python.sh is deliberately NOT wired, for the same reason as + # boxes_config.sh: it does not test the CLI we ship. 53 of its checks + # invoke `lager python -c ""` for inline execution, and there is + # no `-c` option -- the runner reports "Error: No such option '-c'" + # and the suite fails 43 of 84. Three of its checks pass `--image`, + # which is likewise gone. It needs the rewrite devenv.sh already had, + # against the current interface (a script path as the positional + # RUNNABLE), and that is its own reviewable change. Baselining 43 + # failures here would record "calls an option that does not exist" as + # the expected state and leave the ratchet watching for a 44th. + # + # Its other findings stand on their own and are filed as issue #274: + # `--timeout` never fires, and a detached launch does not return. + + - name: Stop anything a suite left running on the box + # Cheap insurance that survives python.sh being unwired: any suite + # that starts work on the box and is then killed mid-step leaves it + # behind, and the next suite inherits a busy box -- failures that + # belong to no suite and would otherwise be baselined as if they did. + if: always() + continue-on-error: true + timeout-minutes: 3 + run: | + # --kill-all, not --kill: --kill takes a process ID. + timeout 45 lager python --box "$LAGER_BOX" --kill-all --signal sigkill >/dev/null 2>&1 || true + for attempt in $(seq 1 24); do + if lager hello --box "$LAGER_BOX" >/dev/null 2>&1; then + echo "box responsive on attempt $attempt" + exit 0 + fi + sleep 5 + done + echo "::warning title=Bench Extended::box unresponsive after the python suite" + shell: bash + + - name: Suite devenv.sh (local dev environment, runner-side docker) + id: suite-devenv + continue-on-error: true + timeout-minutes: 10 + env: + # Clean, and expected to stay that way. + DEVENV_BASELINE: "0" + run: | + LOG="$RUNNER_TEMP/devenv.log" + bash test/integration/infrastructure/devenv.sh 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness devenv.sh "$DEVENV_BASELINE" "$LOG" + shell: bash + + # boxes_config.sh is deliberately NOT wired: its discovery section + # inventories every box reachable from the runner's network and prints + # the results, and a full network inventory does not belong in a CI + # log. It stays runnable by hand; wiring it here needs a CI-safe output + # mode for the discovery commands first (counts, not names). + + - name: Suite nets.sh (net add/delete/rename lifecycle) + # Its delete-all section is self-skipped in the script; individual + # test-net operations only. Even so, the relay-net self-heal in the + # instrumented workflows covers any net damage on their next run. + id: suite-nets + continue-on-error: true + timeout-minutes: 10 + env: + # Clean. It measured 4 failures while python.sh ran ahead of it; + # with that suite unwired every check passes. Keep at 0. + NETS_BASELINE: "0" + run: | + LOG="$RUNNER_TEMP/nets.log" + bash test/integration/infrastructure/nets.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness nets.sh "$NETS_BASELINE" "$LOG" + shell: bash + + - name: Suite box_config.sh (box-config apply, bounce + rollback) + # Late on purpose: exercises the container bounce path, so the box + # may be briefly degraded; nothing after it needs full box health + # except generic.sh's CLI-level checks. + id: suite-box-config + continue-on-error: true + timeout-minutes: 15 + env: + # 6 of 56: the sysctl/cargo apply+clear cluster, the disabled + # box_http_server still running, and the box-side validate + # wording. Owned by issue #275; lower this as they are fixed. + BOX_CONFIG_BASELINE: "6" + run: | + LOG="$RUNNER_TEMP/box-config.log" + bash test/integration/infrastructure/box_config.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness box_config.sh "$BOX_CONFIG_BASELINE" "$LOG" + shell: bash + + - name: Wait for the box to come back after the config bounce + # box_config.sh ends by restarting the container, and generic.sh opens + # with `lager hello`. Without this the first two hello checks failed + # and the third passed eleven seconds later -- a race this workflow's + # own suite ordering creates, which would otherwise be charged to + # generic.sh's baseline and hide a real regression behind it. + # continue-on-error so a box that never comes back is reported by the + # suites themselves rather than as an opaque step failure here. + continue-on-error: true + timeout-minutes: 3 + run: | + for attempt in $(seq 1 30); do + if lager hello --box "$LAGER_BOX" >/dev/null 2>&1; then + echo "box responded on attempt $attempt" + exit 0 + fi + sleep 5 + done + echo "::warning title=Bench Extended::box did not respond within 150s after the box-config bounce" + shell: bash + + - name: Suite generic.sh (CLI-generic commands; LAST -- see header) + id: suite-generic + continue-on-error: true + timeout-minutes: 15 + env: + # 5 of 95: the two 50-iteration stress loops, the mixed-command + # sequence, and both works-after-errors checks. Single calls and + # the 5- and 10-call loops all pass, so this is specifically + # sustained repetition. Owned by issue #273; lower this as it + # is fixed. + GENERIC_BASELINE: "5" + run: | + LOG="$RUNNER_TEMP/generic.log" + bash test/integration/infrastructure/generic.sh "$LAGER_BOX" 2>&1 | tee "$LOG" || true + bash tools/bench_suite_gate.sh harness generic.sh "$GENERIC_BASELINE" "$LOG" + shell: bash + + - name: Restore box registration (best effort) + # generic.sh exercises `boxes delete-all` against the runner's lager + # config; re-register so this run's remaining steps -- and any human + # using the runner before the next workflow -- see a sane state. + if: always() + continue-on-error: true + run: | + lager boxes add \ + --name "$LAGER_BOX" \ + --ip "${{ secrets.LAGER_BOX_IP }}" \ + --user lagerdata \ + --yes || true + shell: bash + + - name: Fail job if any suite failed + # Reads the steps context rather than a hand-kept id list -- adding a + # suite needs no edit here. `outcome` is the pre-continue-on-error + # result. + env: + STEPS_JSON: ${{ toJSON(steps) }} + run: | + python3 - <<'PY' + import json, os, sys + steps = json.loads(os.environ["STEPS_JSON"]) + bad = sorted(k for k, v in steps.items() + if isinstance(v, dict) and v.get("outcome") == "failure") + if bad: + print("One or more infrastructure suites failed:") + for name in bad: + print(f" {name}") + sys.exit(1) + print("All infrastructure suites passed.") + PY + shell: bash + + notify-failure: + # Same alert path as the nightly; see the header for why there is no + # recovery counterpart here. + name: notify (failure) + needs: [extended] + if: ${{ !cancelled() && needs.extended.result != 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + issues: write + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + # No job here pushes anything; do not persist the token into + # the checkout's config (zizmor: artipacked). + persist-credentials: false + + - name: File or update the bench-alert issue + env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + RESULT: ${{ needs.extended.result }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + cat > "$RUNNER_TEMP/alert-body.md" < # @@ -79,7 +85,10 @@ lager hello --box "$BOX" 2>&1 | grep -qi "online" \ && track_test "pass" || track_test "fail" echo "Test 1.4: lager box config show reports no config" -lager box config show --box "$BOX" 2>&1 | grep -qi "no box_config" \ +# Bounded: the first CI run of this suite hung here for 15 minutes right +# after the container restart in 1.2 -- `show` never returned. A hang is a +# FAIL with a visible reason, not a stall that eats the job timeout. +timeout 60 lager box config show --box "$BOX" 2>&1 | grep -qi "no box_config" \ && track_test "pass" || track_test "fail" # Test mount paths. Use fresh paths under /tmp so auto-prep can mkdir/chown @@ -462,8 +471,20 @@ EOF track_test "pass" echo "Test 5.2: validate exits non-zero" -lager box config validate --box "$BOX" 2>&1 | grep -qi "cannot be '/'" \ - && track_test "pass" || track_test "fail" +# The assertion stays as-is on purpose. validate and apply share one +# validate() call (box/lager/box_config/config.py _validate_mounts), and this +# tree's unit tests pin the "cannot be '/'" wording, so a bench box that does +# not produce it is running box-side code older than main -- which is a real +# finding, not a reason to loosen the grep. Print what it actually said so the +# next reader of this log does not have to reproduce it to find out. +VALIDATE_OUT=$(lager box config validate --box "$BOX" 2>&1) +if echo "$VALIDATE_OUT" | grep -qi "cannot be '/'"; then + track_test "pass" +else + echo " validate did not report the root-mount error; it said:" + echo "$VALIDATE_OUT" | sed 's/^/ /' + track_test "fail" +fi echo "Test 5.3: apply refuses" OUT=$(lager box config apply --box "$BOX" --yes 2>&1) diff --git a/test/integration/infrastructure/boxes_config.sh b/test/integration/infrastructure/boxes_config.sh index 4ae29e1c..c4476681 100755 --- a/test/integration/infrastructure/boxes_config.sh +++ b/test/integration/infrastructure/boxes_config.sh @@ -119,12 +119,12 @@ lager boxes import "$EXPORT_FILE" --merge --yes && track_test "pass" || track_te echo "" echo "Test 3.3: Import non-existent file (error case)" -lager boxes import "/tmp/nonexistent_file_$$.json" --yes 2>&1 | grep -qi "error\|not found\|no such" && track_test "pass" || track_test "pass" +lager boxes import "/tmp/nonexistent_file_$$.json" --yes 2>&1 | grep -qi "error\|not found\|no such" && track_test "pass" || track_test "fail" echo "" echo "Test 3.4: Import invalid JSON file" echo "not valid json" > "/tmp/invalid_json_$$.json" -lager boxes import "/tmp/invalid_json_$$.json" --yes 2>&1 | grep -qi "error\|invalid\|json" && track_test "pass" || track_test "pass" +lager boxes import "/tmp/invalid_json_$$.json" --yes 2>&1 | grep -qi "error\|invalid\|json" && track_test "pass" || track_test "fail" rm -f "/tmp/invalid_json_$$.json" echo "" @@ -135,8 +135,8 @@ start_section "Round-Trip" echo "Test 4.1: Add test boxes, export, delete-all, import, verify" # Add test boxes -lager boxes add --name "roundtrip_test_1" --ip "192.168.200.1" --yes >/dev/null 2>&1 -lager boxes add --name "roundtrip_test_2" --ip "192.168.200.2" --yes >/dev/null 2>&1 +lager boxes add --name "roundtrip_test_1" --ip "192.168.200.1" --user testuser --yes >/dev/null 2>&1 +lager boxes add --name "roundtrip_test_2" --ip "192.168.200.2" --user testuser --yes >/dev/null 2>&1 # Export ROUNDTRIP_FILE="/tmp/lager_roundtrip_$$.json" @@ -172,7 +172,7 @@ start_section "Add-All" echo "Test 5.1: Add-all (auto-discover boxes)" # This requires Tailscale -- may fail if not on Tailscale network -lager boxes add-all --yes 2>&1 +lager boxes add-all --user testuser --yes 2>&1 # Pass regardless since Tailscale may not be available track_test "pass" echo "" diff --git a/test/integration/infrastructure/deployment.sh b/test/integration/infrastructure/deployment.sh index fd95fa01..a832193d 100755 --- a/test/integration/infrastructure/deployment.sh +++ b/test/integration/infrastructure/deployment.sh @@ -13,7 +13,11 @@ # Example: ./test_deployment.sh # Example: ./test_deployment.sh my-box -set -e +# DON'T exit on error - we want to track failures, matching every other suite +# in this directory. Under `set -e` the first failing run_test aborted the +# script, so the summary block at the end never ran and a single early failure +# hid every later result. +set +e SSH_USER="${SSH_USER:-lager}" @@ -42,35 +46,31 @@ if [ -z "$1" ]; then echo "Example: $0 " echo "" echo "Note: For best results, add the DUT first using:" - echo " lager duts add --name --ip [--user ]" + echo " lager boxes add --name --ip --user --yes" exit 1 fi DUT="$1" -# Check if DUT exists in saved list (by name or IP) -DUT_CHECK=$(lager duts 2>/dev/null | grep "^${DUT}" | head -1) +# Check if the box exists in the saved registry (by name or IP). This used +# to read `lager duts`, a registry that no longer exists in the CLI -- every +# run took the not-found path -- and then blocked on an interactive prompt, +# which in a non-TTY context reads EOF and exits. The registry is +# `lager boxes` now, and the not-found case warns and continues, which was +# the stated intent all along ("don't fail - allow testing with raw IPs"). +DUT_CHECK=$(lager boxes 2>/dev/null | grep "^${DUT}" | head -1) if [ -z "$DUT_CHECK" ]; then # Try to find by IP - DUT_CHECK=$(lager duts 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') + DUT_CHECK=$(lager boxes 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') fi -# Warn if DUT not found (but don't fail - allow testing with raw IPs) if [ -z "$DUT_CHECK" ]; then - echo -e "${YELLOW}Warning: DUT '${DUT}' not found in saved DUTs${NC}" + echo -e "${YELLOW}Warning: box '${DUT}' not found in saved boxes${NC}" echo -e "${YELLOW}Tests may fail if SSH authentication is not configured${NC}" echo "" - echo "Saved DUTs:" - lager duts 2>/dev/null || echo " (none)" + echo "To register it:" + echo " lager boxes add --name ${DUT} --ip --user --yes" echo "" - echo "To add this DUT:" - echo " lager duts add --name ${DUT} --ip [--user ]" - echo "" - read -p "Continue anyway? (y/N): " -n 1 -r - echo - if [[ ! $REPLY =~ ^[Yy]$ ]]; then - exit 1 - fi fi # Print header @@ -103,11 +103,13 @@ if ! command -v lager &> /dev/null; then exit 1 fi -# Detect DUT username early for test skipping logic +# Detect DUT username early for test skipping logic. Same registry rename as +# the check above: `lager boxes` columns are name / ip / user / version / +# status, so the $3 read below still lands on the user. DETECTED_USER="" -DUT_INFO_EARLY=$(lager duts 2>/dev/null | grep "^${DUT}" | head -1) +DUT_INFO_EARLY=$(lager boxes 2>/dev/null | grep "^${DUT}" | head -1) if [ -z "$DUT_INFO_EARLY" ]; then - DUT_INFO_EARLY=$(lager duts 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') + DUT_INFO_EARLY=$(lager boxes 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') fi if [ -n "$DUT_INFO_EARLY" ]; then DETECTED_USER=$(echo "$DUT_INFO_EARLY" | awk '{print $3}') @@ -238,7 +240,7 @@ run_test_with_output "lager hello --box ${DUT}" "timeout 30 lager hello --box ${ print_test_header "2. Configuration Management Tests" run_test "lager defaults --help" "lager defaults --help" -run_test "lager duts --help" "lager duts --help" +run_test "lager dut --help" "lager dut --help" # Note: Some lager commands (like instruments, nets) may not work with custom usernames # They try to connect as the default SSH user regardless of box configuration @@ -342,17 +344,18 @@ run_test "lager ssh --help" "lager ssh --help" # Test SSH command execution (non-interactive) - use direct SSH since lager ssh doesn't support --command # First resolve DUT to IP and username for SSH # Handle both old format (just IP) and new format (with user field) -# lager duts output format: +# lager boxes output format: # name ip user # - # test-deploy test-deploy -# Try to find DUT by name first -DUT_INFO=$(lager duts 2>/dev/null | grep "^${DUT}" | head -1) +# Try to find the box by name first (`lager boxes`; see the registry note +# at the top of the script) +DUT_INFO=$(lager boxes 2>/dev/null | grep "^${DUT}" | head -1) # If not found by name, try to find by IP address (2nd column match) if [ -z "$DUT_INFO" ]; then - DUT_INFO=$(lager duts 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') + DUT_INFO=$(lager boxes 2>/dev/null | awk -v ip="${DUT}" '$2 == ip {print; exit}') fi if [ -n "$DUT_INFO" ]; then @@ -444,9 +447,11 @@ print_test_header "10. Container Health Tests" # Check if all expected containers are running (use direct SSH) run_test_with_output "Docker containers running" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker ps --filter name=controller --filter name=python --format \"{{.Names}}\"'" -# Check container health (use direct SSH) -run_test_with_output "Controller container health" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker inspect controller --format \"{{.State.Status}}\"'" -run_test_with_output "Python container health" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker inspect python --format \"{{.State.Status}}\"'" +# Check container health (use direct SSH). One container, named `lager` +# (box/start_box.sh). This used to inspect `controller` and `python`, which +# no longer exist -- so both checks failed on every box, reporting a stale +# expectation as if it were a box fault. +run_test_with_output "Lager container health" "timeout 30 ssh ${DUT_USER}@${DUT_IP} 'docker inspect lager --format \"{{.State.Status}}\"'" # ============================================================================= # Test Section 11: Box Services diff --git a/test/integration/infrastructure/devenv.sh b/test/integration/infrastructure/devenv.sh index c0afc646..144d222b 100755 --- a/test/integration/infrastructure/devenv.sh +++ b/test/integration/infrastructure/devenv.sh @@ -2,15 +2,24 @@ # Copyright 2024-2026 Lager Data # SPDX-License-Identifier: Apache-2.0 -# Integration test suite for lager devenv commands -# Tests: create, list, remove +# Integration test suite for lager devenv commands. # -# Usage: ./test/integration/infrastructure/devenv.sh -# Example: ./test/integration/infrastructure/devenv.sh +# Tests the CURRENT devenv model: one devenv per project, configured in the +# project's .lager file (create/show/set/unset, commands add/delete, mount +# and env config). The previous version of this suite tested the removed +# named-environments model (create --name / list / remove) and failed on +# every one of those commands the first time it ran in CI. +# +# Everything here is config-level: `create` writes .lager and no container +# is ever started, so Docker is NOT required and no image is pulled. The +# container-side behavior (terminal, exec) stays out of scope -- it needs a +# TTY and an image pull, neither of which belongs in this suite. +# +# All work happens in a throwaway directory: devenv config is project-local, +# and .lager discovery walks parent directories, so running from a real +# checkout would read (or write!) that checkout's config. # -# Note: These tests require Docker to be installed locally. -# Tests are non-destructive -- they use a dedicated test -# environment name and clean up after themselves. +# Usage: ./test/integration/infrastructure/devenv.sh SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -21,24 +30,20 @@ set +e init_harness -TEST_ENV_NAME="lager_test_devenv_$$" +WORKDIR=$(mktemp -d "${TMPDIR:-/tmp}/lager_devenv_test.XXXXXX") +cleanup() { rm -rf "$WORKDIR"; } +trap cleanup EXIT +cd "$WORKDIR" || exit 1 + +TEST_IMAGE="debian:bookworm-slim" echo "========================================================================" echo "LAGER DEVENV COMPREHENSIVE TEST SUITE" echo "========================================================================" echo "" -echo "Test environment: $TEST_ENV_NAME" +echo "Working directory: $WORKDIR" echo "" -# Check Docker prerequisite -if ! command -v docker &>/dev/null; then - echo "Docker is not installed. Skipping devenv tests." - start_section "Prerequisites" - skip_test "Docker check" "Docker not installed" - print_summary - exit_with_status -fi - # ============================================================ # SECTION 1: HELP COMMANDS # ============================================================ @@ -52,81 +57,127 @@ echo "Test 1.2: Devenv create help" lager devenv create --help && track_test "pass" || track_test "fail" echo "" -echo "Test 1.3: Devenv list help" -lager devenv list --help && track_test "pass" || track_test "fail" +echo "Test 1.3: Devenv show help" +lager devenv show --help && track_test "pass" || track_test "fail" echo "" -echo "Test 1.4: Devenv remove help" -lager devenv remove --help && track_test "pass" || track_test "fail" +echo "Test 1.4: Devenv mount help" +lager devenv mount --help && track_test "pass" || track_test "fail" echo "" # ============================================================ -# SECTION 2: LIST +# SECTION 2: NO-CONFIG ERROR PATHS # ============================================================ -start_section "List Environments" +start_section "No-Config Errors" -echo "Test 2.1: List dev environments" -lager devenv list && track_test "pass" || track_test "fail" +echo "Test 2.1: show without a .lager file fails informatively" +OUTPUT=$(lager devenv show 2>&1) +if [ ! -e .lager ] && echo "$OUTPUT" | grep -qi "No .lager config"; then + track_test "pass" +else + echo "$OUTPUT" | head -3 + track_test "fail" +fi echo "" -echo "Test 2.2: List stability (5 iterations)" -FAILED=0 -for _ in {1..5}; do - lager devenv list >/dev/null 2>&1 || FAILED=1 -done -[ $FAILED -eq 0 ] && track_test "pass" || track_test "fail" +echo "Test 2.2: set without a .lager file fails informatively" +OUTPUT=$(lager devenv set shell /bin/sh 2>&1) +if echo "$OUTPUT" | grep -qi "No .lager config"; then + track_test "pass" +else + echo "$OUTPUT" | head -3 + track_test "fail" +fi echo "" # ============================================================ -# SECTION 3: CREATE AND REMOVE LIFECYCLE +# SECTION 3: CREATE AND CONFIG LIFECYCLE # ============================================================ -start_section "Create and Remove" - -echo "Test 3.1: Create dev environment" -lager devenv create --name "$TEST_ENV_NAME" && track_test "pass" || track_test "fail" +start_section "Create and Configure" + +echo "Test 3.1: create with all options is non-interactive and writes .lager" +# All three options given: create prompts for anything it was not told, and +# a prompt in a non-TTY context aborts -- see Test 5.1, which pins exactly +# that. +lager devenv create --image "$TEST_IMAGE" --mount-dir /app --shell /bin/sh \ + && [ -f .lager ] && track_test "pass" || track_test "fail" echo "" -echo "Test 3.2: Verify environment appears in list" -if lager devenv list 2>/dev/null | grep -q "$TEST_ENV_NAME"; then +echo "Test 3.2: show reflects what create wrote" +OUTPUT=$(lager devenv show 2>&1) +if echo "$OUTPUT" | grep -q "$TEST_IMAGE" \ + && echo "$OUTPUT" | grep -q "/app" \ + && echo "$OUTPUT" | grep -q "/bin/sh"; then track_test "pass" else + echo "$OUTPUT" | head -6 track_test "fail" fi echo "" -echo "Test 3.3: Remove dev environment" -lager devenv remove --name "$TEST_ENV_NAME" --yes && track_test "pass" || track_test "fail" +echo "Test 3.3: set/show roundtrip" +lager devenv set shell /bin/bash >/dev/null 2>&1 +if lager devenv show 2>&1 | grep -q "/bin/bash"; then + track_test "pass" +else + track_test "fail" +fi echo "" -echo "Test 3.4: Verify environment removed from list" -if lager devenv list 2>/dev/null | grep -q "$TEST_ENV_NAME"; then +echo "Test 3.4: unset removes the key" +lager devenv unset shell >/dev/null 2>&1 +if lager devenv show 2>&1 | grep -q "/bin/bash"; then track_test "fail" else track_test "pass" fi echo "" -# ============================================================ -# SECTION 4: ERROR CASES -# ============================================================ -start_section "Error Cases" +echo "Test 3.5: commands starts empty and add/delete roundtrips" +lager devenv add build "make all" >/dev/null 2>&1 +ADDED=$(lager devenv commands 2>&1) +lager devenv delete build >/dev/null 2>&1 +GONE=$(lager devenv commands 2>&1) +if echo "$ADDED" | grep -q "build" && ! echo "$GONE" | grep -q "build"; then + track_test "pass" +else + echo "after add: $(echo "$ADDED" | head -2)" + echo "after delete: $(echo "$GONE" | head -2)" + track_test "fail" +fi +echo "" -echo "Test 4.1: Remove non-existent environment" -lager devenv remove --name "nonexistent_env_12345" --yes 2>&1 | grep -qi "error\|not found" && track_test "pass" || track_test "pass" +echo "Test 3.6: env config help is reachable" +lager devenv env --help && track_test "pass" || track_test "fail" echo "" -echo "Test 4.2: Create with invalid image name" -lager devenv create --name "test_invalid_$$" --image "!!!invalid!!!" 2>&1 | grep -qi "error\|invalid" && track_test "pass" || track_test "pass" -# Clean up in case it somehow succeeded -lager devenv remove --name "test_invalid_$$" --yes >/dev/null 2>&1 || true +# ============================================================ +# SECTION 4: HEADLESS BEHAVIOR +# ============================================================ +start_section "Headless Behavior" + +echo "Test 5.1: create with missing options ABORTS headless, never hangs" +# Without --mount-dir, create prompts. In a non-TTY context the prompt must +# abort promptly; hanging forever is the failure mode this pins (same +# headless-prompt class as the box_config `show` hang and the historical +# "Invalid sudoers syntax" install failure). +rm -f .lager +timeout 15 lager devenv create --image "$TEST_IMAGE" < /dev/null >/dev/null 2>&1 +RC=$? +if [ $RC -eq 124 ]; then + echo "create HUNG on a prompt in a non-TTY context (timeout hit)" + track_test "fail" +elif [ $RC -ne 0 ] && [ ! -f .lager ]; then + track_test "pass" +else + echo "expected a prompt abort; rc=$RC, .lager present: $([ -f .lager ] && echo yes || echo no)" + track_test "fail" +fi echo "" # ============================================================ # SUMMARY # ============================================================ -# Final cleanup -lager devenv remove --name "$TEST_ENV_NAME" --yes >/dev/null 2>&1 || true - print_summary exit_with_status diff --git a/test/integration/infrastructure/generic.sh b/test/integration/infrastructure/generic.sh index f1c471ae..2df0acc1 100755 --- a/test/integration/infrastructure/generic.sh +++ b/test/integration/infrastructure/generic.sh @@ -41,7 +41,7 @@ if echo "$BOX_INPUT" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'; then BOX_IP="$BOX_INPUT" echo "Detected IP address: $BOX_IP" echo "Registering as temporary box: $BOX_NAME" - lager boxes add --name "$BOX_NAME" --ip "$BOX_IP" --yes >/dev/null 2>&1 || true + lager boxes add --name "$BOX_NAME" --ip "$BOX_IP" --user lagerdata --yes >/dev/null 2>&1 || true BOX="$BOX_NAME" else # Input is a box name - use it directly @@ -90,7 +90,7 @@ lager hello --box $BOX && track_test "pass" || track_test "fail" echo "" echo "Test 1.3: Hello with invalid box (error case)" -lager hello --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 1.4: Multiple hello commands (stability test)" @@ -133,7 +133,7 @@ lager instruments --box $BOX && track_test "pass" || track_test "fail" echo "" echo "Test 2.3: Instruments with invalid box (error case)" -lager instruments --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager instruments --box INVALID_BOX_12345 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 2.4: Multiple instrument listings (stability test)" @@ -199,7 +199,7 @@ lager boxes add --help && track_test "pass" || track_test "fail" echo "" echo "Test 4.2: Add a test box" -lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.100" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.100" --user testuser --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 4.3: List boxes to verify addition" @@ -211,7 +211,7 @@ fi echo "" echo "Test 4.4: Attempt to add duplicate box name (should show warning)" -if lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.200" --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then +if lager boxes add --name "$TEST_BOX_NAME" --ip "192.168.1.200" --user testuser --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then track_test "pass" else track_test "fail" @@ -228,9 +228,9 @@ echo "" echo "Test 4.6: Attempt to add duplicate IP (should show warning)" # First add a new box with unique name -lager boxes add --name "${TEST_BOX_NAME}_unique" --ip "192.168.1.150" --yes >/dev/null 2>&1 || true +lager boxes add --name "${TEST_BOX_NAME}_unique" --ip "192.168.1.150" --user testuser --yes >/dev/null 2>&1 || true # Try to add another box with same IP but different name -if lager boxes add --name "${TEST_BOX_NAME}_duplicate_ip" --ip "192.168.1.150" --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then +if lager boxes add --name "${TEST_BOX_NAME}_duplicate_ip" --ip "192.168.1.150" --user testuser --yes 2>&1 | grep -qi "WARNING.*Duplicate"; then track_test "pass" else track_test "fail" @@ -238,7 +238,7 @@ fi echo "" echo "Test 4.7: Add box with special characters in name" -lager boxes add --name "test-box_123.special" --ip "192.168.1.101" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "test-box_123.special" --ip "192.168.1.101" --user testuser --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 4.8: Boxes delete subcommand help" @@ -256,13 +256,13 @@ done echo "" echo "Test 4.10: Delete non-existent box (error case)" -lager boxes delete --name "nonexistent_box_12345" --yes 2>&1 | grep -qi "not found" && track_test "pass" || track_test "pass" +lager boxes delete --name "nonexistent_box_12345" --yes 2>&1 | grep -qi "not found" && track_test "pass" || track_test "fail" echo "" echo "Test 4.11: Add and delete multiple boxes" FAILED=0 for i in {1..3}; do - lager boxes add --name "${TEST_BOX_NAME}_${i}" --ip "192.168.1.$((100+i))" --yes >/dev/null 2>&1 || FAILED=1 + lager boxes add --name "${TEST_BOX_NAME}_${i}" --ip "192.168.1.$((100+i))" --user testuser --yes >/dev/null 2>&1 || FAILED=1 done echo "Added test boxes" lager boxes list @@ -278,7 +278,7 @@ echo "" echo "Test 4.13: Edit box IP address" # Add a test box first -lager boxes add --name "${TEST_BOX_NAME}_edit" --ip "192.168.1.50" --yes >/dev/null 2>&1 || true +lager boxes add --name "${TEST_BOX_NAME}_edit" --ip "192.168.1.50" --user testuser --yes >/dev/null 2>&1 || true # Edit its IP if lager boxes edit --name "${TEST_BOX_NAME}_edit" --ip "192.168.1.51" --yes 2>&1 | grep -q "Updated box"; then track_test "pass" @@ -295,7 +295,9 @@ echo "" echo "Test 4.14: Edit box name" # Rename the box -if lager boxes edit --name "${TEST_BOX_NAME}_edit" --new-name "${TEST_BOX_NAME}_renamed" --yes 2>&1 | grep -q "Renamed box"; then +# `boxes edit` reports every change through one "Updated box: ..." line; there +# is no separate rename message (cli/commands/box/boxes.py success_msg). +if lager boxes edit --name "${TEST_BOX_NAME}_edit" --new-name "${TEST_BOX_NAME}_renamed" --yes 2>&1 | grep -q "Updated box"; then track_test "pass" else track_test "fail" @@ -331,7 +333,9 @@ fi echo "" echo "Test 4.17: Edit with invalid IP (error case)" -if lager boxes edit --name "${TEST_BOX_NAME}_final" --ip "invalid_ip" --yes 2>&1 | grep -qi "not a valid IP"; then +# "invalid_ip" is refused for the underscore, not for failing to be an IP: +# --ip accepts a hostname too. See the note above Test 9.8. +if lager boxes edit --name "${TEST_BOX_NAME}_final" --ip "invalid_ip" --yes 2>&1 | grep -qi "invalid hostname label"; then track_test "pass" else track_test "fail" @@ -376,7 +380,7 @@ BOX_COUNT_BEFORE=$(lager boxes list 2>/dev/null | wc -l || echo "0") # Try delete-all when there might be existing boxes (from user's .lager file) if [ "$BOX_COUNT_BEFORE" -gt 1 ]; then echo "Note: .lager file contains user boxes, skipping empty delete-all test" - track_test "pass" + track_test "skip" else lager boxes delete-all --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" fi @@ -394,7 +398,7 @@ fi lager boxes delete-all --yes >/dev/null 2>&1 || true # Add test boxes for i in {1..5}; do - lager boxes add --name "deleteall_test_${i}" --ip "192.168.2.${i}" --yes >/dev/null 2>&1 || true + lager boxes add --name "deleteall_test_${i}" --ip "192.168.2.${i}" --user testuser --yes >/dev/null 2>&1 || true done DELETEALL_COUNT=$(lager boxes list 2>/dev/null | grep -c "deleteall_test" || echo "0") echo "Added $DELETEALL_COUNT test boxes for delete-all test" @@ -481,7 +485,7 @@ echo "$DEFAULTS_BEFORE" echo "" echo "Test 6.3: Add box to saved boxes first" -lager boxes add --name "test_default_box" --ip "$BOX" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "test_default_box" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 6.4: Set default box" @@ -499,7 +503,7 @@ fi echo "" echo "Test 6.6: Set default with non-existent box (error case)" -if lager defaults add --box "INVALID_BOX_12345" 2>&1 | grep -qi "does not exist in saved boxes"; then +if lager defaults add --box "INVALID_BOX_12345" 2>&1 | grep -qi "not found in saved boxes"; then track_test "pass" else track_test "fail" @@ -553,7 +557,7 @@ done echo "" echo "Test 7.5: Set both box and serial port at once" -lager boxes add --name "test_combo_default" --ip "192.168.1.123" --yes >/dev/null 2>&1 +lager boxes add --name "test_combo_default" --ip "192.168.1.123" --user testuser --yes >/dev/null 2>&1 if lager defaults add --box "test_combo_default" --serial-port "/dev/ttyUSB2" 2>&1 | grep -q "Set defaults"; then track_test "pass" else @@ -572,7 +576,7 @@ echo "========================================================================" echo "" echo "Test 8.1: Add test box for persistence tests" -lager boxes add --name "test_persist_box" --ip "$BOX" --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" +lager boxes add --name "test_persist_box" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 && track_test "pass" || track_test "fail" echo "" echo "Test 8.2: Set default box and verify persistence" @@ -614,7 +618,7 @@ echo "========================================================================" echo "" echo "Test 9.1: Empty box name in lager boxes add" -if lager boxes add --name "" --ip "192.168.1.100" --yes 2>&1 | grep -qi "name cannot be empty"; then +if lager boxes add --name "" --ip "192.168.1.100" --user testuser --yes 2>&1 | grep -qi "name cannot be empty"; then track_test "pass" else track_test "fail" @@ -622,7 +626,7 @@ fi echo "" echo "Test 9.2: Whitespace-only box name" -if lager boxes add --name " " --ip "192.168.1.100" --yes 2>&1 | grep -qi "name cannot be empty"; then +if lager boxes add --name " " --ip "192.168.1.100" --user testuser --yes 2>&1 | grep -qi "name cannot be empty"; then track_test "pass" else track_test "fail" @@ -630,39 +634,47 @@ fi echo "" echo "Test 9.3: Empty box name in lager hello" -lager hello --box "" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 9.4: Very long box name" LONG_BOX_NAME=$(printf 'a%.0s' {1..500}) -lager hello --box "$LONG_BOX_NAME" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "$LONG_BOX_NAME" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 9.5: Box name with special characters" -lager hello --box "test@#$%^&*()" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "test@#$%^&*()" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" echo "Test 9.6: Box name with spaces" -lager hello --box "test box with spaces" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "test box with spaces" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" # The name mixes BMP (CJK) and non-BMP (U+1D400, 4-byte UTF-8) codepoints on # purpose: they exercise different encoding paths, and a name that is only CJK # would not catch a surrogate-pair bug. echo "Test 9.7: Box name with Unicode characters" -lager hello --box "test_设备_𝐀" 2>&1 | grep -qi "error" && track_test "pass" || track_test "pass" +lager hello --box "test_设备_𝐀" 2>&1 | grep -qi "error" && track_test "pass" || track_test "fail" echo "" -echo "Test 9.8: Invalid IP address format in box add" -if lager boxes add --name "test_invalid_ip" --ip "999.999.999.999" --yes 2>&1 | grep -qi "not a valid IP"; then +# --ip takes an IP address OR a DNS hostname (cli/address_utils.py +# validate_ip_or_hostname), so anything shaped like a hostname is accepted by +# design: "999.999.999.999", "192.168.1.256" and "string" are all syntactically +# valid hostname labels and are NOT rejected. These six checks therefore cover +# the inputs the validator genuinely refuses. Do not "restore" assertions that +# expect an IP-only contract -- that contract does not exist, and asserting it +# only produces failures that say nothing about the product. + +echo "Test 9.8: URL rejected in box add" +if lager boxes add --name "test_url_ip" --ip "http://box.example.com" --user testuser --yes 2>&1 | grep -qi "looks like a URL"; then track_test "pass" else track_test "fail" fi echo "" -echo "Test 9.9: Malformed IP address" -if lager boxes add --name "test_malformed" --ip "not.an.ip.address" --yes 2>&1 | grep -qi "not a valid IP"; then +echo "Test 9.9: Address with a path rejected" +if lager boxes add --name "test_path_ip" --ip "box.example.com/path" --user testuser --yes 2>&1 | grep -qi "contains a path"; then track_test "pass" else track_test "fail" @@ -670,7 +682,7 @@ fi echo "" echo "Test 9.10: Empty IP address" -if lager boxes add --name "test_empty_ip" --ip "" --yes 2>&1 | grep -qi "IP.*cannot be empty"; then +if lager boxes add --name "test_empty_ip" --ip "" --user testuser --yes 2>&1 | grep -qi "cannot be empty"; then track_test "pass" else track_test "fail" @@ -678,23 +690,23 @@ fi echo "" echo "Test 9.11: IP address with whitespace" -if lager boxes add --name "test_whitespace_ip" --ip " " --yes 2>&1 | grep -qi "IP.*cannot be empty"; then +if lager boxes add --name "test_whitespace_ip" --ip " " --user testuser --yes 2>&1 | grep -qi "cannot be empty"; then track_test "pass" else track_test "fail" fi echo "" -echo "Test 9.12: String instead of IP address" -if lager boxes add --name "test_string_ip" --ip "string" --yes 2>&1 | grep -qi "not a valid IP"; then +echo "Test 9.12: Address carrying an explicit port rejected" +if lager boxes add --name "test_port_ip" --ip "192.168.1.100:8080" --user testuser --yes 2>&1 | grep -qi "contains a port"; then track_test "pass" else track_test "fail" fi echo "" -echo "Test 9.13: IP address with invalid octets" -if lager boxes add --name "test_octet_ip" --ip "192.168.1.256" --yes 2>&1 | grep -qi "not a valid IP"; then +echo "Test 9.13: Invalid hostname label rejected" +if lager boxes add --name "test_label_ip" --ip "bad_label.example.com" --user testuser --yes 2>&1 | grep -qi "invalid hostname label"; then track_test "pass" else track_test "fail" @@ -702,7 +714,7 @@ fi echo "" echo "Test 9.14: Box operations without required parameters" -lager boxes add 2>&1 | grep -qi "error\|missing" && track_test "pass" || track_test "pass" +lager boxes add 2>&1 | grep -qi "error\|missing" && track_test "pass" || track_test "fail" echo "" echo "Test 9.15: Defaults add without any parameters" @@ -714,7 +726,7 @@ fi echo "" echo "Test 9.16: Valid IPv4 address acceptance" -if lager boxes add --name "test_valid_ipv4" --ip "192.168.1.100" --yes 2>&1 | grep -q "Added box"; then +if lager boxes add --name "test_valid_ipv4" --ip "192.168.1.100" --user testuser --yes 2>&1 | grep -q "Added box"; then track_test "pass" lager boxes delete --name "test_valid_ipv4" --yes >/dev/null 2>&1 || true else @@ -723,7 +735,7 @@ fi echo "" echo "Test 9.17: Valid IPv6 address acceptance" -if lager boxes add --name "test_valid_ipv6" --ip "2001:0db8:85a3:0000:0000:8a2e:0370:7334" --yes 2>&1 | grep -q "Added box"; then +if lager boxes add --name "test_valid_ipv6" --ip "2001:0db8:85a3:0000:0000:8a2e:0370:7334" --user testuser --yes 2>&1 | grep -q "Added box"; then track_test "pass" lager boxes delete --name "test_valid_ipv6" --yes >/dev/null 2>&1 || true else @@ -732,7 +744,7 @@ fi echo "" echo "Test 9.18: Localhost IP acceptance" -if lager boxes add --name "test_localhost" --ip "127.0.0.1" --yes 2>&1 | grep -q "Added box"; then +if lager boxes add --name "test_localhost" --ip "127.0.0.1" --user testuser --yes 2>&1 | grep -q "Added box"; then track_test "pass" lager boxes delete --name "test_localhost" --yes >/dev/null 2>&1 || true else @@ -759,7 +771,7 @@ else TEST_COMBO_IP="192.168.1.100" fi FAILED=0 -lager boxes add --name "${TEST_BOX_NAME}_combo" --ip "$TEST_COMBO_IP" --yes >/dev/null 2>&1 || FAILED=1 +lager boxes add --name "${TEST_BOX_NAME}_combo" --ip "$TEST_COMBO_IP" --user testuser --yes >/dev/null 2>&1 || FAILED=1 # Only test set/hello if we used the real BOX IP if [ "$TEST_COMBO_IP" = "$BOX" ]; then lager defaults add --box "${TEST_BOX_NAME}_combo" >/dev/null 2>&1 || FAILED=1 @@ -770,7 +782,7 @@ lager boxes delete --name "${TEST_BOX_NAME}_combo" --yes >/dev/null 2>&1 || FAIL echo "" echo "Test 10.2: Interleaved list/add operations" -lager boxes add --name "test_interleave" --ip "$BOX" --yes >/dev/null 2>&1 +lager boxes add --name "test_interleave" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 FAILED=0 lager defaults >/dev/null 2>&1 || FAILED=1 lager defaults add --box "test_interleave" >/dev/null 2>&1 || FAILED=1 @@ -838,7 +850,7 @@ echo "" echo "Test 11.4: Rapid box add/delete cycles (20 iterations)" FAILED=0 for i in {1..20}; do - lager boxes add --name "stress_box_${i}" --ip "192.168.100.${i}" --yes >/dev/null 2>&1 || FAILED=1 + lager boxes add --name "stress_box_${i}" --ip "192.168.100.${i}" --user testuser --yes >/dev/null 2>&1 || FAILED=1 lager boxes delete --name "stress_box_${i}" --yes >/dev/null 2>&1 || FAILED=1 done [ $FAILED -eq 0 ] && track_test "pass" || track_test "fail" @@ -878,7 +890,7 @@ lager hello --box "$BOX" >/dev/null && track_test "pass" || track_test "fail" echo "" echo "Test 12.3: Verify box list after failed add" -lager boxes add --name "" --ip "" --yes >/dev/null 2>&1 || true +lager boxes add --name "" --ip "" --user testuser --yes >/dev/null 2>&1 || true lager boxes list >/dev/null && track_test "pass" || track_test "fail" echo "" @@ -888,7 +900,7 @@ lager defaults >/dev/null && track_test "pass" || track_test "fail" echo "" echo "Test 12.5: Verify configuration consistency after multiple operations" -lager boxes add --name "test_regression" --ip "$BOX" --yes >/dev/null 2>&1 +lager boxes add --name "test_regression" --ip "$BOX" --user lagerdata --yes >/dev/null 2>&1 DEFAULTS_START=$(lager defaults 2>&1) lager defaults add --box "test_regression" >/dev/null 2>&1 || true lager hello --box "$BOX" >/dev/null 2>&1 diff --git a/test/integration/infrastructure/nets.sh b/test/integration/infrastructure/nets.sh index b092d225..b1985ab4 100755 --- a/test/integration/infrastructure/nets.sh +++ b/test/integration/infrastructure/nets.sh @@ -101,7 +101,11 @@ fi echo "" echo "Test 1.2: Nets command help" -if lager nets --help 2>&1 | grep -q 'List all saved nets'; then +# Match the durable part of the group help ("... saved nets"), not the full +# sentence: the wording has already drifted once (it now reads "List and +# manage saved nets") and a whole-sentence match makes this a copy-editing +# tripwire rather than a test that the command exists and describes itself. +if lager nets --help 2>&1 | grep -q 'saved nets'; then echo -e "${GREEN}[OK]${NC}" track_test "pass" else diff --git a/test/integration/infrastructure/python.sh b/test/integration/infrastructure/python.sh index e70016a0..4a47dd72 100755 --- a/test/integration/infrastructure/python.sh +++ b/test/integration/infrastructure/python.sh @@ -12,7 +12,11 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" source "${SCRIPT_DIR}/../../framework/colors.sh" source "${SCRIPT_DIR}/../../framework/harness.sh" -DOCKER_IMAGE="${DOCKER_IMAGE:?Set DOCKER_IMAGE to your lagerbox image tag}" +# Optional: only the two `lager python --image` checks need it, and a hard +# `:?` guard here aborted the whole suite before a single test ran -- which is +# why none of the ~100 checks below had ever executed anywhere. Unset means +# those two checks report as skipped; everything else runs. +DOCKER_IMAGE="${DOCKER_IMAGE:-}" set +e # DON'T exit on error - we want to track failures @@ -22,6 +26,41 @@ init_harness # Safety delay between tests (seconds) TEST_DELAY=0.5 +# Kill checks below use `--kill-all`, not `--kill`. `--kill` takes a process +# ID (`--kill TEXT`); passing it none made click consume the following +# `--signal` as its argument and then reject the signal name as a script path. +# Every kill check in this suite was malformed that way, which only became +# visible once the suite actually ran. +# +# Both the launch and the kill are bounded. The first CI run of this section +# spent over nine minutes here and was killed by the step timeout with the +# infinite loop it had started still running on the box; the suites after it +# then ran against a box that was still churning. Bounding turns that into a +# visible result for the check that caused it instead of damage charged to +# whatever ran next. +KILL_TIMEOUT=45 +LAUNCH_TIMEOUT=30 + +# Bound EVERY lager call in this suite, not just the ones already known to +# hang. Two separate paths have now stalled indefinitely on the bench -- +# a detached launch that never returned, and `--timeout 3` against a 30s +# script that ran for seventeen minutes -- and each one cost a full bench +# round to find because an unbounded hang looks identical to a slow suite +# until the step timeout kills it. With this wrapper a hang becomes a failed +# check that names itself, and the suites after this one stop inheriting a +# box that is still busy. +# +# Resolve the real binary first: calling `lager` inside the function would +# recurse. The ceiling is far above any legitimate operation here (the +# longest deliberate wait in this file is --timeout 10). +LAGER_CALL_TIMEOUT="${LAGER_CALL_TIMEOUT:-90}" +LAGER_BIN="$(command -v lager)" +if [ -z "$LAGER_BIN" ]; then + echo "lager CLI not found on PATH; install it first (pip install -e cli/)" >&2 + exit 1 +fi +lager() { timeout "$LAGER_CALL_TIMEOUT" "$LAGER_BIN" "$@"; } + # Check if required arguments are provided if [ $# -lt 1 ]; then echo "Usage: $0 " @@ -168,7 +207,7 @@ if lager python --box $BOX --image "nonexistent:tag999" -c "print('test')" 2>&1 track_test "pass" else echo -e "${YELLOW}[WARNING] Invalid image may not be validated upfront${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -185,7 +224,7 @@ echo "" sleep $TEST_DELAY echo "Test 2.6: Invalid --signal value" -if lager python --box $BOX --kill --signal invalid_signal 2>&1 | grep -qi "error\|invalid"; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal invalid_signal 2>&1 | grep -qi "error\|invalid"; then echo -e "${GREEN}[OK] Invalid signal caught${NC}" track_test "pass" else @@ -201,18 +240,18 @@ if lager python --box $BOX -p "invalid:port:format" -c "print('test')" 2>&1 | gr track_test "pass" else echo -e "${YELLOW}[WARNING] Port format may not be validated upfront${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY -echo "Test 2.8: --kill without running script" -if lager python --box $BOX --kill 2>&1 | grep -qi "error\|not running\|no script"; then +echo "Test 2.8: --kill-all without running script" +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all 2>&1 | grep -qi "error\|not running\|no script"; then echo -e "${GREEN}[OK] Kill without script handled${NC}" track_test "pass" else echo -e "${YELLOW}[WARNING] Kill without script may be silently ignored${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -542,7 +581,7 @@ if lager python "$TEST_DIR/no_overwrite.py" --box $BOX --download no_overwrite.t track_test "pass" else echo -e "${YELLOW}[WARNING] Overwrite may not be prevented${NC}" - track_test "pass" + track_test "skip" fi rm -f no_overwrite.txt echo "" @@ -585,7 +624,10 @@ echo "" sleep $TEST_DELAY echo "Test 6.2: Specify custom Docker image (using default)" -if lager python --box $BOX --image "$DOCKER_IMAGE" -c "print('Custom image')" 2>&1 | grep -q "Custom image"; then +if [ -z "$DOCKER_IMAGE" ]; then + echo -e "${YELLOW}[SKIP] DOCKER_IMAGE not set${NC}" + track_test "skip" +elif lager python --box $BOX --image "$DOCKER_IMAGE" -c "print('Custom image')" 2>&1 | grep -q "Custom image"; then echo -e "${GREEN}[OK] Custom image specified${NC}" track_test "pass" else @@ -676,7 +718,7 @@ if lager python "$TEST_DIR/long_running.py" --box $BOX -d 2>&1 | grep -qi "detac track_test "pass" else echo -e "${YELLOW}[SKIP] Detached mode may not show confirmation${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -697,6 +739,11 @@ echo "========================================================================" echo "" sleep $TEST_DELAY +# This section starts an INFINITE loop on the box and then asks the CLI to +# kill it, so it is where an unbounded call does the most damage. Timeouts +# come from the top of the file; the cleanup at the end of the section does +# not depend on `--kill` having worked. + echo "Test 9.1: Kill running script (--kill with default signal)" cat > "$TEST_DIR/infinite.py" <<'EOF' import time @@ -705,57 +752,63 @@ while True: time.sleep(1) EOF # Start in background -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill 2>&1 | grep -qi "killed\|terminated\|stopped"; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all 2>&1 | grep -qi "killed\|terminated\|stopped"; then echo -e "${GREEN}[OK] Script killed${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] Kill may have succeeded without confirmation${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] Kill did not confirm within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY echo "Test 9.2: Kill with SIGINT signal" -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill --signal sigint 2>&1; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigint 2>&1; then echo -e "${GREEN}[OK] SIGINT sent${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] SIGINT may have been sent${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] SIGINT not confirmed within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY echo "Test 9.3: Kill with SIGTERM signal" -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill --signal sigterm 2>&1; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigterm 2>&1; then echo -e "${GREEN}[OK] SIGTERM sent${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] SIGTERM may have been sent${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] SIGTERM not confirmed within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY echo "Test 9.4: Kill with SIGKILL signal" -lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 +timeout $LAUNCH_TIMEOUT lager python "$TEST_DIR/infinite.py" --box $BOX -d >/dev/null 2>&1 sleep 2 -if lager python --box $BOX --kill --signal sigkill 2>&1; then +if timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigkill 2>&1; then echo -e "${GREEN}[OK] SIGKILL sent${NC}" track_test "pass" else - echo -e "${YELLOW}[SKIP] SIGKILL may have been sent${NC}" - track_test "pass" + echo -e "${YELLOW}[SKIP] SIGKILL not confirmed within ${KILL_TIMEOUT}s${NC}" + track_test "skip" fi echo "" sleep $TEST_DELAY +# Leave no infinite loop behind, whatever the four checks above concluded. +# Not a tracked check: this is cleanup, and its failure is reported by the +# suites that follow rather than counted here. +echo "Section 9 cleanup: stop any script still running on the box" +timeout $KILL_TIMEOUT lager python --box $BOX --kill-all --signal sigkill >/dev/null 2>&1 || true + # ============================================================ # SECTION 10: TIMEOUT OPERATIONS # ============================================================ @@ -789,7 +842,7 @@ if lager python "$TEST_DIR/slow.py" --box $BOX --timeout 3 2>&1 | grep -qi "time track_test "pass" else echo -e "${YELLOW}[SKIP] Timeout may have triggered without message${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -800,7 +853,7 @@ if lager python --box $BOX --timeout 1 -c "import time; time.sleep(10); print('s track_test "pass" else echo -e "${YELLOW}[SKIP] Short timeout may have triggered${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -811,7 +864,7 @@ if lager python --box $BOX --timeout 0 -c "print('test')" 2>&1 | grep -qi "error track_test "pass" else echo -e "${YELLOW}[SKIP] Zero timeout may use default${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -822,7 +875,7 @@ if lager python --box $BOX --timeout -1 -c "print('test')" 2>&1 | grep -qi "erro track_test "pass" else echo -e "${YELLOW}[SKIP] Negative timeout may be rejected${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -907,7 +960,10 @@ echo "" sleep $TEST_DELAY echo "Test 11.5: Custom image with environment variables" -if lager python --box $BOX --image "$DOCKER_IMAGE" --env CUSTOM=test -c "import os; print(os.environ.get('CUSTOM'))" 2>&1 | grep -q "test"; then +if [ -z "$DOCKER_IMAGE" ]; then + echo -e "${YELLOW}[SKIP] DOCKER_IMAGE not set${NC}" + track_test "skip" +elif lager python --box $BOX --image "$DOCKER_IMAGE" --env CUSTOM=test -c "import os; print(os.environ.get('CUSTOM'))" 2>&1 | grep -q "test"; then echo -e "${GREEN}[OK] Custom image + env works${NC}" track_test "pass" else @@ -998,7 +1054,10 @@ echo "" sleep $TEST_DELAY echo "Test 12.5: Unicode and UTF-8" -if lager python --box $BOX -c "print('Hello 世界 🌍')" 2>&1 | grep -q "世界"; then +# Mixes BMP (CJK, 3-byte) and non-BMP (U+1D400, 4-byte) codepoints, the same +# pair generic.sh and nets.sh use: they exercise different encoding paths, and +# a string that is only CJK would not catch a surrogate-pair bug. +if lager python --box $BOX -c "print('Hello 世界 𝐀')" 2>&1 | grep -q "世界"; then echo -e "${GREEN}[OK] Unicode support works${NC}" track_test "pass" else @@ -1051,7 +1110,7 @@ if lager python --box $BOX --add-file "$TEST_DIR/large_file.bin" -c "import os; track_test "pass" else echo -e "${YELLOW}[WARNING] Large file may not have uploaded correctly${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1250,7 +1309,7 @@ if lager python --box $BOX -c "" 2>&1; then track_test "pass" else echo -e "${YELLOW}[SKIP] Empty script may cause error${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1261,7 +1320,7 @@ if lager python --box $BOX -c " " 2>&1; then track_test "pass" else echo -e "${YELLOW}[SKIP] Whitespace-only script may cause error${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1288,7 +1347,7 @@ if lager python --box $BOX -c "$LONG_SCRIPT" >/dev/null 2>&1; then track_test "pass" else echo -e "${YELLOW}[SKIP] Very long command may be truncated${NC}" - track_test "pass" + track_test "skip" fi echo "" sleep $TEST_DELAY @@ -1358,7 +1417,7 @@ echo "" sleep $TEST_DELAY # Make sure no scripts are still running -lager python --box $BOX --kill >/dev/null 2>&1 || true +timeout $KILL_TIMEOUT lager python --box $BOX --kill-all >/dev/null 2>&1 || true # ============================================================ # TEST SUMMARY diff --git a/tools/bench_suite_gate.sh b/tools/bench_suite_gate.sh new file mode 100644 index 00000000..3ddd4df9 --- /dev/null +++ b/tools/bench_suite_gate.sh @@ -0,0 +1,121 @@ +#!/bin/bash +# Copyright 2024-2026 Lager Data +# SPDX-License-Identifier: Apache-2.0 +# +# Two-sided baseline ratchet for one bench suite. +# +# The bench integration suites are wired into CI carrying known failures. +# A suite that simply "must pass" would either sit permanently red (a check +# people learn to ignore) or force the failures to be softened away. Instead +# each suite declares how many checks are expected to fail today, and this +# gate gets stricter automatically: +# +# failures > baseline -> ERROR, exit 1. New breakage beyond the known gap. +# failures == baseline -> warning. Known gap, run stays green. +# failures < baseline -> notice telling the reader to lower the baseline +# and lock the improvement in. +# +# Every non-zero baseline must name an owning issue where it is set. A number +# with no issue behind it is softening, not baselining. +# +# Usage: bench_suite_gate.sh