Integration test suite for the `run` command (simnet MVP, CI-runnable)

[emlautarom1-agent]

Summary

Add a minimal, CI-runnable integration test suite that validates the run command works end-to-end under normal conditions. The goal is not feature coverage — it is an MVP that proves a run cluster boots, forms a network, and completes duties, validated by launching the real binary and observing it from the outside, the same way scripts/dkg-runner/run.sh validates DKG.

Charon is the source of truth for what needs to be tested. Its integration suite lives in testutil/integration/ (gated by a runtime -integration flag).

Prerequisites (must exist before any test runs)

1. A run binary with a simnet mode. Charon exposes --simnet-beacon-mock and --simnet-validator-mock on charon run: the node runs its own in-process beacon-node mock (ticks slots, emits deterministic attester/proposer/sync duties) and its own validator-client mock (fetches, signs, submits). Without equivalent flags there is nothing deterministic to black-box.
2. On-host transports for a single-machine cluster. Charon's simnet runs consensus over libp2p via a relay and exchanges partial signatures in-memory. The node must talk to peers without a real beacon chain: a reachable relay + a partial-sig path that works cluster-locally.
3. Cluster artifact generation. A way to lay down N node directories (cluster lock + p2p key + BLS key shares) on disk — via the create cluster / create dkg CLI (already produced by the DKG runner).
4. A local relay the nodes can register with (Charon starts one in-process; a CI script can start a relay binary, or reuse a hosted one as the DKG runner does).
5. An external success signal. In-process, Charon asserts via a broadcast callback. For a black-box binary, the harness needs to read success from outside: a readiness/health/monitoring endpoint (peer count, beacon sync, duties seen) and/or structured logs it can grep for "duty broadcast".
6. A shell runner skeleton mirroring the DKG runner phases (setup → start-nodes → monitor → collect → verify), CI-gated.

The three high-value tests (Charon as source of truth)

Duty workflow (port of TestSimnetDuties)

• What Charon proves: a 3-node cluster runs the full pipeline per duty type — schedule → fetch → consensus → sign → partial-sig exchange → threshold-aggregate → broadcast. The load-bearing assertion is agreement: every node produces the same aggregated signature for a given duty, and each expected duty type fires within a couple of slots.
• MVP scope: the attester path alone proves the whole pipeline. Proposer / sync-committee / builder-registration are the "matrix" — defer to a nightly variant.
• Black-box realization: launch N run nodes in simnet mode against their beacon mocks, wait a few slots, assert each node reports a successfully broadcast attestation and that the signatures agree.

P2P connectivity (port of TestPingCluster)

• What Charon proves: nodes discover each other, connect through the relay, upgrade to direct connections, and reach a full ping mesh (variants: direct, relay, wrong-external-IP fallback).
• MVP scope: just that the cluster forms a fully-connected mesh — a precondition for the duty-workflow test, so it is cheap to assert first.
• Black-box realization: poll each node's monitoring endpoint until connected-peer count reaches N-1 within a timeout.

Black-box CLI (port of the compose TestSmoke)

• What Charon proves: the actual charon run binary, in simnet mode, reaches a healthy state across small topologies.
• Key insight: this is not a separate test so much as the harness that the duty-workflow and connectivity tests run inside. Booting the real binary in simnet mode, confirming readiness, running one full duty cycle, and shutting down cleanly is the smoke test — and it maps directly onto the DKG-runner model.

In Charon these are three separate in-process Go tests. For a CI-first Pluto MVP they collapse into one black-box harness: the black-box CLI harness is the scaffolding (boot + ready + clean exit), the connectivity test is an assertion on the running cluster (mesh formed), and the duty-workflow test is an assertion on the same cluster (duties flow and agree).

CI execution model (mirror scripts/dkg-runner/run.sh)

A sibling runner (e.g. scripts/simnet-runner/) with the same phase structure and ergonomics (PID/pgid tracking, TIMEOUT, CI-aware logging, work dir preserved on failure):

Phase	DKG-runner analog	Simnet runner does
setup	setup.sh	generate N node dirs (lock + p2p key + shares) via CLI
start-nodes	start-nodes.sh	launch N run --simnet-* processes against a local relay
monitor	monitor.sh	poll health API within TIMEOUT: mesh formed → nodes ready → ≥1 duty broadcast
collect	collect.sh	gather per-node logs + a health snapshot
verify	verify-*.sh	assert mesh size, duty broadcast, signature agreement across nodes, clean exit

Gate it behind a CI flag; run the attester-only 3-node case on every PR (fast, deterministic) and the full duty matrix / fault variants nightly.

Suggested sequencing

1. Prereqs — simnet flags on run (beacon + validator mock), cluster-artifact generation, a local relay, and a readiness/health signal to poll.
2. Black-box CLI skeleton — the runner boots a 3-node simnet cluster, confirms all become ready, then shuts down cleanly. Immediate "does run boot" signal and the scaffold everything layers onto.
3. Connectivity assertion — extend the monitor phase to require the full peer mesh before proceeding.
4. Duty workflow (attester) — assert at least the attester duty flows end-to-end and all nodes agree on the aggregated signature. This is the core correctness win.
5. Expand (nightly) — proposer / sync-committee / builder-registration duty matrix, then a t-of-n fault case with one node down.

Charon references

• testutil/integration/simnet_test.go — TestSimnetDuties (duty workflow)
• testutil/integration/ping_test.go — TestPingCluster (P2P connectivity)
• testutil/compose/smoke/smoke_test.go — TestSmoke (black-box CLI)
• app/app.go — TestConfig / SimnetBMock / SimnetVMock seams
• testutil/beaconmock/, testutil/validatormock/, cluster.NewForT, testutil/relay/ — simnet building blocks