Sync upstream DNS enforcement and sandbox runtime updates

README.md

@@ -250,7 +250,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit
 * [Egress Traffic](docs/egress-traffic.md): Which protocols an Actor may reach the outside world with and which are blocked.
 * [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](docs/egress-trust-bundle.md): Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception.
 * [Request Parking](docs/request-parking.md): How the router parks requests through transient worker-pool saturation.
-* [Rolling Upgrade Runbook](docs/upgrade.md): Upgrade a running substrate node by node without losing actor state.
+* [Upgrade Runbook](docs/upgrade.md): Upgrade a running install to a newer supported release, and roll back if needed.
 * [Security Guide](docs/security.md): Security overview, security principles, and threat modeling.
 * [Roadmap](docs/roadmap.md): Current limitations and what is planned next.
 * [Benchmarking Guide](benchmarking/README.md): Locust-based load tests, monitoring stack, and the orchestrated benchmark harness.

benchmarking/README.md

@@ -111,7 +111,7 @@ Four flags control the optional post-run measurements described in
   Defaults to the in-cluster service installed by
   [Optional: Prometheus + Grafana](#optional-prometheus--grafana).
 * `--atelet-lag-s`: how long to wait after the run before reading the
-  atelet's snapshot metrics. Defaults to 70.
+  atelet's snapshot and active actor metrics. Defaults to 70.
 * `--phase-breakdown` / `--no-phase-breakdown`: read the atelet and worker pod
   logs once the run ends and add the suspend/resume phase percentiles to
   `stats.jsonl`. On by default; needs `pods/log` access in `ate-system` and
@@ -522,7 +522,9 @@ them are checked into the repository.
   those two keys, because it is what CI orchestration reads to decide whether a
   trial ran at all.
 * `stats.csv`, `stats_history.csv`, `failures.csv`, `exceptions.csv`: Locust's
-  own CSV output.
+  own CSV output. `stats_history.csv` has a row per operation every second as
+  well as the `Aggregated` row, so each operation's latency can be charted
+  over the run; filter to `Aggregated` for run-wide numbers.
 * `logs.txt`, `traces.txt`: the runner log, and the trace IDs seen during the run.
 * `stats.jsonl`: one JSON object per line, one per metric. Every row carries
   the same five keys: `timestamp`, `tag`, `test_name`, `metric`, and a flat
@@ -565,7 +567,9 @@ comparing numbers across runs:
 * **Actors are derived, not counted.** Locust only sees virtual users, so the
   numerator is the peak user count times `--actors-per-user`. No server-side
   gauge counts resident actors: `ate.actor.stats.sampled_actors` drops any
-  actor without a live resource measurement, so suspended ones fall out.
+  actor without a live resource measurement, so suspended ones fall out. It
+  is recorded as `active_actors` in `server_summary.json`, a running-actor
+  count rather than a resident one.
 * **The denominators are read once, after the run.** A cluster that autoscaled
   mid-run is measured at its final size, so the ratio pairs a peak from one
   moment with a capacity from another.
@@ -591,11 +595,29 @@ actually did, independent of what the load generator reported.
 * `snapshots.size_p50_mb` through `size_p99_mb`: actor memory image sizes.
 * `snapshots.size_avg_mb`: mean memory image size.
 * `snapshots.restore_p50_s` through `restore_p99_s`, `restore_mean_s`, and the
-  same for `checkpoint_*`: atelet restore and checkpoint latency.
+  same for `checkpoint_*`: how long the atelet took to resume or suspend an
+  actor (`ate_actor_{restore,checkpoint}_duration`, `total` phase). Every
+  snapshot kind counts, as in the sizes: the actor's own snapshot, a
+  template's first start from its golden snapshot, and a pause. The buckets
+  reach 60s, so a slower operation reads as 60s.
 * `snapshots.checkpoints_in_window`, `checkpoints_cumulative`: checkpoint
   volume over the steady-state window, and since the atelet started.
 * `snapshots.checkpoint_mb_s`: bytes written per second spent checkpointing,
   not per second of wall clock.
+* `active_actors`: running actors (`ate_actor_stats_sampled_actors`), as a
+  percentile `summary` of the cluster-wide count, `atelets` seen, and a
+  `timeseries` every 10s over the whole run (ramp-up included). Each point has
+  the cluster-wide count and `active_atelets` hosting an actor. As the
+  assumptions above note, the gauge drops actors without a live measurement,
+  so suspended ones fall out and the count dips while actors are suspended.
+  The atelet stops exporting when a node has no running actor, so at a step
+  where at least one atelet reported, a missing one counts as 0. A step where
+  none reported is skipped, not 0, since a failed scrape looks the same; this
+  mostly hits 1-user runs. If the metric never appeared during the run, every
+  field is `null`. `atelets` counts atelet processes seen in the run, not
+  nodes: the collector keeps an exited atelet's last value for about 5
+  minutes, so a restart can briefly count twice, and an atelet scraped both
+  directly and through a collector counts twice.
 
 Every distribution reports p50, p90, p95 and p99 over the steady-state window.
 The steady-state window runs from the first to the last Locust sample at 90% or
@@ -608,13 +630,13 @@ every 10s.
 
 The atelet exports before Prometheus scrapes it, so the harvest waits
 `--atelet-lag-s` seconds (default 70, enough for the OTel SDK's 60s default
-export and a 10s scrape)
-and reads the snapshot window half that late. The window ends at the last
-full-load sample, so teardown suspends are left out.
+export and a 10s scrape) and reads the snapshot and `active_actors` windows
+half that late, stamping `active_actors` points back by the same amount. The
+window ends at the last full-load sample, so teardown suspends are left out.
 
-`metadata.start_ts` and `end_ts` bound the whole run, which the packing
-`timeseries` covers. `steady_start_ts` and `steady_end_ts` bound the
-steady-state window. A flat subset of the numbers also goes into a
+`metadata.start_ts` and `end_ts` bound the whole run, which the packing and
+`active_actors` timeseries cover. `steady_start_ts` and `steady_end_ts` bound
+the steady-state window. A flat subset of the numbers also goes into a
 `server_summary` row in `stats.jsonl`.
 
 Neither the Kubernetes API nor Prometheus is required. If either is unreachable,

benchmarking/automation/manifests/nighthawk-ingress-runner-job.yaml.tmpl

@@ -20,6 +20,9 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: benchmarking
+  labels:
+    # ate-system admits API clients only from namespaces with this label.
+    ate.dev/api-access: "true"
 ---
 apiVersion: v1
 kind: ServiceAccount

benchmarking/automation/manifests/runner-job.yaml.tmpl

@@ -20,6 +20,9 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: benchmarking
+  labels:
+    # ate-system admits API clients only from namespaces with this label.
+    ate.dev/api-access: "true"
 ---
 apiVersion: v1
 kind: ServiceAccount

benchmarking/automation/test_orchestrator.py

@@ -135,6 +135,39 @@ def test_no_placeholder_survives(self):
         self.assertNotIn("${", orchestrator.render_template(self.TMPL, subs))
 
 
+class RunnerNamespaceTest(unittest.TestCase):
+    """Every runner Job's namespace carries the label ate-system's
+    NetworkPolicy admits API clients by; without it the runner cannot reach
+    ate-api-server."""
+
+    HERE = os.path.dirname(__file__)
+    POLICIES = os.path.join(HERE, "..", "..", "manifests", "ate-install", "ate-system-networkpolicies.yaml")
+
+    def api_client_labels(self):
+        for doc in yaml.safe_load_all(open(self.POLICIES)):
+            if not doc or doc["metadata"]["name"] != "ate-api-server":
+                continue
+            for rule in doc["spec"]["ingress"]:
+                for peer in rule.get("from", []):
+                    if "namespaceSelector" in peer:
+                        return peer["namespaceSelector"]["matchLabels"]
+        self.fail("ate-api-server NetworkPolicy admits no namespace by label")
+
+    def test_namespace_has_api_access_label(self):
+        want = self.api_client_labels()
+        for name in ("runner-job.yaml.tmpl", "nighthawk-ingress-runner-job.yaml.tmpl"):
+            with self.subTest(template=name):
+                text = open(os.path.join(self.HERE, "manifests", name)).read()
+                ns = next(
+                    yaml.safe_load(d)
+                    for d in text.split("\n---\n")
+                    if "kind: Namespace" in d
+                )
+                self.assertEqual(ns["metadata"]["name"], "benchmarking")
+                for k, v in want.items():
+                    self.assertEqual(ns["metadata"].get("labels", {}).get(k), v)
+
+
 class JobNameTest(unittest.TestCase):
     COMMIT = "ac41c06deadbeef"
 

benchmarking/locust/runner.py

@@ -161,10 +161,11 @@ def parse_args() -> argparse.Namespace:
         type=int,
         default=70,
         help=(
-            "Seconds to wait after the run, and to shift the snapshot window "
-            "by, so the atelet's last export is scraped. Must cover the "
-            "atelet's OTEL_METRIC_EXPORT_INTERVAL plus one Prometheus scrape; "
-            "the default fits the OTel SDK's 60s default export and a 10s scrape"
+            "Seconds to wait after the run, so the atelet's last export is "
+            "scraped; its snapshot and active actor windows are read half this "
+            "late. Must cover the atelet's OTEL_METRIC_EXPORT_INTERVAL plus one "
+            "Prometheus scrape; the default fits the OTel SDK's 60s default export "
+            "and a 10s scrape"
         ),
     )
     p.add_argument(
@@ -344,6 +345,10 @@ def run_test(args: argparse.Namespace, csv_prefix: Path, logs: TextIO, traces: T
         "-t", args.duration,
         "-u", str(args.users),
         "--csv", str(csv_prefix),
+        # Every operation's row in stats_history.csv, not just Aggregated, so
+        # each one's latency can be charted over the run. Readers of the file
+        # here keep to the Aggregated rows.
+        "--csv-full-history",
     ]
     if with_boomer:
         # Master mode so boomer can connect as a worker on localhost:5557.