Fix station bootstrap and defaults; harden application monitoring readiness

.github/workflows/ci.yml

@@ -24,9 +24,22 @@ jobs:
           python3 -I -B tools/fetch_caddy_fixture.py --output "$RUNNER_TEMP/dragontools-caddy-2.11.4"
           python3 -I -B tests/agent_ingestion_test.py --caddy "$RUNNER_TEMP/dragontools-caddy-2.11.4"
       - run: python3 -I -B tests/release_test.py
+      - name: Doers signals, alerts and outage recovery (isolated Linux processes)
+        if: runner.os == 'Linux'
+        timeout-minutes: 12
+        run: |
+          fixture="$RUNNER_TEMP/dragontools-doers"
+          python3 -I -B tests/integration/render_doers.py "$fixture"
+          python3 -I -B tools/fetch_doers_fixture.py --output "$fixture"
+          cp zig-out/bin/dragontool-agent "$fixture/dragontool-agent"
+          python3 -I -B tests/integration/agent_ingestion_pipeline.py --prepare-credentials "$fixture"
+          sudo unshare --net sh -c 'ip link set lo up; exec python3 -I -B tests/integration/doers_runtime.py "$1" --outage' fixture "$fixture"
       - name: Linux helper filesystem lifecycle (temporary paths only)
         if: runner.os == 'Linux'
         run: sudo python3 -I -B tests/helper_install_test.py --fixture "$PWD/zig-out/bin/dragontool-pki-fixture" --agent "$PWD/zig-out/bin/dragontool-agent"
+      - name: Native station bootstrap (isolated filesystem, exact production ownership)
+        if: runner.os == 'Linux'
+        run: sudo python3 -I -B tests/station_bootstrap_test.py --agent "$PWD/zig-out/bin/dragontool-agent"
   cross-build:
     runs-on: ubuntu-24.04
     strategy:

AGENTS.md

@@ -7,6 +7,11 @@ configuration and login shell. `wizard` and `completion` are local UX entry poin
 No generic resource DSL, shell hooks or provider framework.
 Read README.md, architecture.md and design.md before changing workflow behavior.
 
+Station install/verify/status/notify-test default only to CWD `./station.toml`;
+explicit `--config` replaces it and CLI fields override it. `[station].hostname`
+is canonical; conflicting v1 `[ingress].hostname` aliases fail. Application
+commands never load station config. Plans resolve no secrets.
+
 The primary application workflow is `monitoring apply` with strict version-1
 `./monitoring.toml`, or one explicit `--config`. Keep application configuration
 separate from central station configuration and secrets. Application/environment

CHANGELOG.md

@@ -2,6 +2,34 @@
 
 ## 0.1.0-dev — unreleased
 
+- Default station install/verify/status/notify-test to CWD `./station.toml`, with
+  explicit config replacement and CLI precedence. Add canonical `[station].hostname`,
+  retain the compatible v1 `[ingress].hostname` alias and reject conflicts. Plans
+  show the SSH connection and TLS hostname/9443/9444 without resolving secrets;
+  application commands retain their separate `./monitoring.toml` default.
+- Fix Linux native bootstrap's free-lock failure: Zig 0.16 `O_PATH` descriptors
+  cannot be flocked. Use a readable directory descriptor with unchanged exclusive,
+  nonblocking semantics; report contention as OperationBusy/96. Add granular safe
+  filesystem/CA/server checks and preserve redaction and CA maintenance semantics.
+- Converge absent/empty/interrupted station PKI state, recover proven unpublished
+  candidates, reuse published CA/server identity and restrict registry repair to
+  the known 0700-to-0750 generation. Add portable lifecycle/lock tests and a real
+  helper chroot regression with exact root/dt-ingest production ownership and no-op
+  rerun. No application registration, ambient OpenSSL or architecture changes.
+
+- Accept fresh vmagent failed-scrape telemetry when the application is down;
+  require fresh non-scrape payload on successful targets and reject stale/missing
+  data. Report pipeline readiness independently of application availability.
+- Read the post-handshake TLS client-auth alert before sending HTTP, avoiding
+  OpenSSL write-side EOF races without accepting EOF/timeouts as proof of mTLS.
+- Give app publication, scraper reload and finalization fixed semantic check IDs;
+  do not suggest reinstalling ingress for app namespace/rule failures.
+- Add the production-rendered Doers reference process gate: zero-app station,
+  real pinned Caddy/Vector/vmagent/blackbox/VM/VL/vmalert, quiet logs and trusted
+  labels, the real two-minute alert hold and recovery, scoped probe removal and
+  unchanged publication. Add the isolated Linux gate to CI, retaining shared
+  Linux/macOS tests and a fixture-only early HTTP rejection portability fix.
+
 - Move station CA/server PKI, Caddy and private ingress authorization ownership to
   `monitoring install`. Add `--ingress-hostname` / `[ingress].hostname`, independent
   of SSH; reuse an existing managed endpoint when omitted. Install/verify/status

README.md

@@ -6,7 +6,8 @@ Install the monitoring station once, then keep each application's monitoring
 contract in its own repository:
 
 ```bash
-dragontool monitoring install --config station.toml
+cd monitoring-infra # Directory containing station.toml
+dragontool monitoring install
 
 cd my-application
 dragontool monitoring apply --plan
@@ -15,6 +16,7 @@ dragontool monitoring app-verify
 dragontool monitoring apply # Deliberate unchanged rerun
 ```
 
+Station `install`, `verify`, `status` and `notify-test` read `./station.toml`.
 Application commands read only `./monitoring.toml` by default, or one explicit
 `--config PATH`. They configure host metrics, selected journal logs, optional
 application metrics, station HTTP probes and application alerts. The strict
@@ -129,27 +131,35 @@ while anonymous access, auth proxy and signup stay disabled.
 
 ## Monitoring configuration and Grafana credentials
 
-`--config` reads one explicit, small TOML file for monitoring `install`, `verify`,
-`status`, and `notify-test`. There is no implicit file discovery. The version-1
+`monitoring install`, `verify`, `status` and `notify-test` load **`./station.toml`**
+when present; install `--plan` uses it too. `--config PATH` selects one different
+file instead of the default. Only the current working directory is used: no parent,
+home, XDG or `/etc` search. Application commands use only `./monitoring.toml`. The version-1
 schema accepts an OpenSSH alias, an ingress TLS DNS hostname, Grafana/Telegram secret references and bounded
 named HTTP/HTTPS probes; component tuning, literal passwords, unknown keys and
 duplicate keys are rejected.
 
-The checked-in [examples/monitoring.toml](examples/monitoring.toml) contains these
-**example** references. Replace the alias and vault/item/field paths with your own:
+Save central station intent as `station.toml`, separately from application
+repositories. [examples/station.toml](examples/station.toml) is a minimal template.
+The following hostname, SSH alias and optional secret references are examples;
+replace them with your own:
 
 ```toml
 version = 1
 
 [connection]
 ssh_host = "monitoring"
 
-[ingress]
-hostname = "monitoring.example.com"
+[station]
+hostname = "monitoring.baptizeddragon.com"
 
 [grafana]
 username = { op = "op://BaptizedDragon/Grafana/username" }
 password = { op = "op://BaptizedDragon/Grafana/password" }
+
+[telegram]
+bot_token = { op = "op://BaptizedDragon/DragonTools/alarms-telegram-bot-token" }
+chat_id = { op = "op://BaptizedDragon/DragonTools/alarms-telegram-chat-id" }
 ```
 
 A reference contains no resolved secret and is safe to keep in configuration or
@@ -164,17 +174,44 @@ its CLI, or its session credentials.
 op signin
 
 zig build -Doptimize=ReleaseSafe
-./zig-out/bin/dragontool monitoring install --config examples/monitoring.toml --plan
-./zig-out/bin/dragontool monitoring install --config examples/monitoring.toml
-./zig-out/bin/dragontool monitoring verify --config examples/monitoring.toml
-./zig-out/bin/dragontool monitoring status --config examples/monitoring.toml
+./zig-out/bin/dragontool monitoring install --plan
+./zig-out/bin/dragontool monitoring install
+./zig-out/bin/dragontool monitoring verify
+./zig-out/bin/dragontool monitoring status
 
 # Desired credentials already work: no password reset or service restart.
-./zig-out/bin/dragontool monitoring install --config examples/monitoring.toml
+./zig-out/bin/dragontool monitoring install
+```
+
+Precedence is explicit CLI flags, then the selected file, then CLI defaults. An
+explicit `--config other.toml` ignores `./station.toml` completely. A missing
+default file still permits CLI-only usage; without a connection, the error points
+to `./station.toml`, `--config` and CLI options. Present malformed files fail
+without exposing their contents.
+
+`connection.ssh_host` is an administrative OpenSSH alias. `station.hostname` is
+the independent DNS/mTLS identity, overridden by `--ingress-hostname`. It must be
+a DNS name without a scheme, port or path. Version 1 continues accepting deprecated
+`[ingress].hostname`; specifying both forms with different values fails. Plan shows
+the connection, hostname and metrics/logs ports without resolving any secrets.
+
+For example, from a directory without `station.toml`, CLI-only operation is:
+
+```bash
+dragontool monitoring install --ssh-host monitoring \
+  --ingress-hostname monitoring.baptizeddragon.com
 ```
 
-Explicit CLI values override the corresponding configuration values. The direct
-CLI equivalent is:
+`station.toml` describes the station; `monitoring.toml` describes one application:
+
+```bash
+cd monitoring-infra
+dragontool monitoring install
+cd ../doers
+dragontool monitoring apply
+```
+
+The Grafana CLI equivalent is:
 
 ```bash
 ./zig-out/bin/dragontool monitoring install \
@@ -569,14 +606,14 @@ Run with the same enrolled SSH alias throughout:
 
 ```bash
 zig build -Doptimize=ReleaseSafe
-./zig-out/bin/dragontool monitoring install --config monitoring.toml --plan
-./zig-out/bin/dragontool monitoring install --config monitoring.toml
-./zig-out/bin/dragontool monitoring verify --config monitoring.toml
-./zig-out/bin/dragontool monitoring status --config monitoring.toml
+./zig-out/bin/dragontool monitoring install --config station.toml --plan
+./zig-out/bin/dragontool monitoring install --config station.toml
+./zig-out/bin/dragontool monitoring verify --config station.toml
+./zig-out/bin/dragontool monitoring status --config station.toml
 # Deliberate unchanged rerun:
-./zig-out/bin/dragontool monitoring install --config monitoring.toml
+./zig-out/bin/dragontool monitoring install --config station.toml
 # Explicitly sends a test alert through Alertmanager to its configured receiver:
-./zig-out/bin/dragontool monitoring notify-test --config monitoring.toml
+./zig-out/bin/dragontool monitoring notify-test --config station.toml
 ```
 
 Only GET with expected HTTP 2xx is supported. Probe names are unique, at most 63
@@ -875,6 +912,13 @@ app rule glob, and zero rule samples are valid. `monitoring apply` and
 expected managed rules to be loaded and healthy. Rule readiness uses bounded
 retries after restart; it does not require an alert expression to match samples.
 A failing HTTP target is valid monitoring data; a broken probe pipeline fails.
+The same distinction applies to vmagent: fresh `up=0` proves scrape reporting and
+remote delivery while the target is down. Success reports pipeline readiness,
+not application availability. An `up=1` target still needs fresh application payload.
+The [Doers reference validation](tests/integration/doers-validation.md) records
+isolated signal, alert recovery, namespace and rerun evidence and remaining host
+checks. Publication/reload failures identify `application_publish`,
+`application_scrape_reload` or `application_finalize` without raw remote output.
 Verify/status never resolve station secrets or send test notifications. Live
 alert evaluation remains active independently.
 
@@ -997,7 +1041,7 @@ for local test evidence and remaining deployment checks.
 
 `monitoring install` owns all ten station services, including Caddy and private
 registry authorization, plus the native helper and station PKI. A fresh station
-requires `--ingress-hostname DNS` or `[ingress].hostname` in the central station
+requires `--ingress-hostname DNS` or `[station].hostname` in the central station
 config. The CLI flag overrides the file. Without either, an existing exactly
 managed server bundle supplies its saved identity; a fresh station fails with
 `ingress_hostname_required`. The SSH alias never supplies the TLS hostname.
@@ -1218,6 +1262,25 @@ CA state. Correct the reported cause and rerun station install for base PKI/ingr
 or the same application config for client enrollment;
 empty managed bootstrap parents remain safe to reuse.
 
+Station install accepts an absent ingestion tree, the empty root-owned
+`pki/clients/registry` skeleton and interrupted unpublished bundles. It validates
+native Mbed TLS CA/server candidates before atomic publication and fsync, reuses
+an already published valid CA, and never rotates an invalid existing CA silently.
+Only recognized private staging with known names, marker and metadata is removed;
+unrecognized files and symlinks are preserved and refused. The known
+root:dt-ingest registry mode `0700` migrates to `0750`; other ownership/type or
+unknown mode conflicts fail. No app directory or registration is required.
+
+Fixed checks now distinguish `ingestion_root_invalid`, `pki_directory_invalid`,
+`clients_directory_invalid`, `registry_directory_invalid`, `state_directory_invalid`,
+`unexpected_managed_file`, `unexpected_symlink`, `ca_bundle_invalid` and
+`server_identity_invalid`. Missing CA in an uninitialized tree is the internal
+`ca_missing_bootstrap_allowed` state, not corruption. A held operation lock reports
+`operation_busy` (native exit 96); it never reports a filesystem contradiction.
+The Linux lock regression used Zig 0.16's `O_PATH` directory descriptor, which
+`flock` rejects even without contention. Using a readable directory handle preserves
+the same exclusive, nonblocking advisory lock and read-only verification behavior.
+
 This is a private ingestion channel, so DragonTools intentionally uses its own
 CA rather than Let's Encrypt. The station certificate includes its configured
 DNS/IP SAN. Vector checks both certificate and hostname, and vmagent retains
@@ -1262,8 +1325,10 @@ and the main journald configuration remain untouched. Conflicting later override
 are refused. These bounds do not cover applications writing their own log files.
 
 Install verifies service/configuration/hardening, the secured endpoint, recent
-host metrics, every selected log stream, and each app target's successful scrape
-and recent non-scrape metric. Install/verify require samples newer than the current
+host metrics, every selected log stream, and fresh scrape telemetry for each app
+target. A target reporting `up=1` must also supply a recent non-scrape metric.
+Fresh `up=0` is valid monitoring while the application is down; missing/stale
+`up`, missing payload from an `up=1` target, or an unreachable station still fail. Install/verify require samples newer than the current
 agent process start as well as their freshness windows (90 seconds for metrics,
 two minutes for logs), preventing old data from proving a changed URL works. Keep
 both hosts' clocks synchronized because Vector timestamps originate on the agent.

architecture.md

@@ -14,7 +14,12 @@ provider abstraction, arbitrary shell hooks, or general plugin framework.
 Application repositories use strict `monitoring.toml` v1 and `monitoring apply`.
 The controller reads only the current directory's file or an explicit `--config`.
 `app-verify` and `app-status` are read-only application commands; station
-`install/verify/status` retain their separate central configuration and secrets.
+`install/verify/status/notify-test` load optional `./station.toml`, separately from
+application configuration and secrets. Explicit `--config` replaces the default.
+Neither workflow searches parents, home, XDG or `/etc`; CLI-only station operation
+remains valid when the default is absent. `[station].hostname` supplies TLS identity
+independent of the SSH alias; the deprecated v1 `[ingress].hostname` alias cannot
+contradict it. Plans show the resolved connection/hostname and never resolve secrets.
 
 `cli/parse.zig` merges explicit monitoring configuration and validates all supplied
 inputs before SSH. CLI values override file values; the small version-1 TOML
@@ -471,7 +476,10 @@ The shared Vector host rules preserve application/environment/host grouping;
 application log rules use exact scoped fields and bounded counts/windows. Each
 probe has one default alert or one explicit override. Native probe telemetry with
 `probe_success=0` is a successful monitoring mechanism, not apply failure.
-No notification tests or synthetic application errors run during apply/verify.
+Fresh vmagent `up=0` likewise proves delivery while the target is down; `up=1`
+requires both a fresh scrape result and application payload. Missing/stale samples
+and station outages fail bounded readiness. No notification tests or synthetic
+application errors run during apply/verify.
 
 Dashboards remain unimplemented. Future generated dashboards must use folder
 `DragonTools / <application>` and deterministic UIDs, with explicit ownership.
@@ -538,7 +546,7 @@ readiness budget, not fixed sleeps. Missing signals fail installation.
 
 Station `monitoring install` provisions the ingestion/Caddy accounts, native helper,
 CA/server bundle, Caddy binary/config/unit and private authorization helper. Explicit
-`[ingress].hostname` or `--ingress-hostname` supplies the TLS DNS identity on first
+`[station].hostname` or `--ingress-hostname` supplies the TLS DNS identity on first
 install; later runs may reuse the managed server endpoint. No SSH alias inference
 or application registration is involved. Station verify checks PKI and transport,
 including mandatory client authentication with an empty registry; it never enrolls