diff --git a/.amplifier/digital-twin-universe/profiles/README.md b/.amplifier/digital-twin-universe/profiles/README.md index fe4dbccf..8d4f7d9d 100644 --- a/.amplifier/digital-twin-universe/profiles/README.md +++ b/.amplifier/digital-twin-universe/profiles/README.md @@ -55,6 +55,7 @@ seams without those; that is the point of an end-to-end test. | `context-intelligence-write-server-validation.yaml` | **WRITE to a single server** — hook dispatches a real session's events to ONE `destinations` server; proves the server received them (tagged by workspace) | Incus + Gitea mirror + LLM + **CI server** | `... launch .../context-intelligence-write-server-validation.yaml --var gitea_host=... --var ci_server_url=...` | | `context-intelligence-write-fanout-validation.yaml` | **WRITE fan-out** — one session, TWO `destinations`; proves BOTH servers received the events (observes existing hook fan-out; never modifies it) | Incus + Gitea mirror + LLM + **2 CI servers** | `... launch .../context-intelligence-write-fanout-validation.yaml --var gitea_host=... --var ci_server_a=... --var ci_server_b=...` | | `context-intelligence-query-validation.yaml` | **EXECUTE queries** (read side) — after logging, drives `graph_query` (Cypher) + `blob_read` (`ci-blob://`) via the `graph-analyst` agent; proves real rows/content come back with the `source` provenance naming the server | Incus + Gitea mirror + LLM + **CI server** | `... launch .../context-intelligence-query-validation.yaml --var gitea_host=... --var ci_server_url=...` | +| `context-intelligence-upload-format-validation.yaml` | **Legacy hooks-logging IMPORT** — `--format logging-hook` ingests a shipped neutral synthetic legacy fixture; proves discrimination, runtime slug parity/no fork (graph workspaces exactly equal the runtime-derived set), coexistence/dedupe/idempotency (node count captured at runtime does not grow); self-contained, no host data, no pinned counts | Incus + Gitea mirror + **CI backend** (`context-intelligence-backend.yaml` launched fresh) — no LLM | `... launch .../context-intelligence-upload-format-validation.yaml --var gitea_host=... --var server_url=http://:38000 --var server_token=...` | | `example-dtu-external-server.yaml` | *Not a test* — reference profile: point the client hook at an **external CI server** with a tagged workspace | Incus + running CI server (below) | see below | **Self-contained smoke to prove the harness works on your host:** @@ -102,60 +103,49 @@ backed by Neo4j, serving `/status`, `/events` (write), `/blobs/{session}` and Cy via the hook, then query them back** via `graph_query` / `blob_read` and assert the result + provenance. -There are two ways to give a DTU a server. +### The canonical non-compose backend -### Option A — External server on the host (simplest; the shipped path) +This bundle ships its own self-contained backend recipe — +[`context-intelligence-backend.yaml`](context-intelligence-backend.yaml) — that stands +up, inside one Incus container: **Neo4j Community 5.26.22 (5.26 LTS)** run directly via +`docker run` (APOC Core bundled in the image, no network fetch; GDS intentionally +omitted), plus the **standalone `context-intelligence-server`** (`WEB_CONCURRENCY=1`) +pointed at it over `bolt://localhost:7687`. **The server repo's docker-compose stack is +retired as a DTU dependency** (its compose files are being removed from +`microsoft/amplifier-context-intelligence` itself) — every server-backed profile below +launches this backend profile first and points at its forwarded port. -Run the server **outside** the DTU (on the host or another machine), and point the -DTU's client at it. This is exactly what `example-dtu-external-server.yaml` does. - -1. **Start the server on the host** (per the server repo — Neo4j + server, listening on `:8000`): - ```bash - # in a checkout of microsoft/amplifier-context-intelligence — see that repo's README - # (typically a docker compose bringing up Neo4j + the API on :8000) - curl -sf http://localhost:8000/status # confirm it's up - ``` -2. **Export the API key on the host** so the DTU passthrough can forward it: +1. **Launch the backend:** ```bash - export AMPLIFIER_CONTEXT_INTELLIGENCE_API_KEY= + amplifier-digital-twin launch \ + .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ + --name ci-backend \ + --var NEO4J_PASSWORD= --var API_KEY= \ + --var HOST_PORT=38000 \ + --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf ``` -3. **Launch the DTU pointed at it.** The server URL is auto-detected from the Incus - bridge gateway (host), or set it explicitly: +2. **Confirm it's reachable** from the Incus host-gateway IP (find it via `incus list` + for the container's address, or `ip route | grep default` inside a consuming DTU): ```bash - amplifier-digital-twin launch \ - .amplifier/digital-twin-universe/profiles/example-dtu-external-server.yaml \ - --var CONTEXT_INTELLIGENCE_WORKSPACE=e2e-readside \ - --var CONTEXT_INTELLIGENCE_SERVER_URL=http://:8000 # optional; auto-detected if omitted + curl -sf http://:38000/status ``` - The profile wires `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_URL / _API_KEY / _WORKSPACE` - into the container and gates readiness on `curl $SERVER_URL/status`. -4. **Drive the e2e flow** inside the DTU (`amplifier-digital-twin exec -- ...`): - - run an Amplifier session → the hook POSTs events to `/events` (tagged with the workspace); - - then read them back with the query tools — `graph_query` (Cypher, filtered by - `workspace = "e2e-readside"`) and `blob_read` a `ci-blob://` URI — and assert the - rows + the `source` provenance block name the server that answered. - -> **Container DNS/networking:** if the client can't reach the server from inside the -> container, see `docs/container-dns-troubleshooting.md` (the gateway-IP + port pattern). +3. **Point the consuming profile at it** via its own `--var server_url=...` / + `--var server_a_url=... --var server_b_url=...` (see each profile's header comment + for the exact flags). For fan-out, launch the backend profile **twice** on distinct + `HOST_PORT`s (`38001`, `38002`) — two fully separate Incus containers, so events + never cross. -### Option B — Server as a Docker sidecar *inside* the DTU (self-contained) - -Run the server **and Neo4j** as Docker containers **inside** the DTU via the -docker-in-incus pattern (DTU launches enable `security.nesting=true` by default). Use -this when you want one hermetic environment with no host dependency. +> **Container DNS/networking:** if the client can't reach the backend from inside the +> container, see `docs/container-dns-troubleshooting.md` (the gateway-IP + port pattern). -1. Read the nested-container guide first: load the `digital-twin-universe` skill → - `read_file("@digital-twin-universe:docs/docker-in-incus.md")` (platform-specific - networking, pre-flight with the `docker-in-incus` profile). -2. In the profile's `provision.setup_cmds`, bring up the server's compose (Neo4j + API) - from the server repo, then point the client at `http://localhost:8000` via the same - three `AMPLIFIER_CONTEXT_INTELLIGENCE_*` env vars as Option A. -3. Gate readiness on `curl -sf http://localhost:8000/status`, then run the same - log→query→assert e2e flow as Option A. +`example-dtu-external-server.yaml` is the reference "point a client at an already-running +server" profile — it doesn't stand up a server itself; use the backend profile above to +get one. -> The exact compose/run invocation lives in the **server repo** -> (`microsoft/amplifier-context-intelligence`) — this bundle is the client side. Pin the -> server version you're testing against and record it in your run evidence. +> The exact non-compose bring-up (Neo4j image, APOC, server invocation) lives in +> `context-intelligence-backend.yaml`'s header comment, cited against the **server repo** +> (`microsoft/amplifier-context-intelligence`). Pin the server version you're testing +> against (via `--var SERVER_REF`) and record it in your run evidence. ### Read-side config the tools resolve (both options) @@ -192,9 +182,11 @@ tools (see the main `README.md` §"read side"). For multi-source, the connectabl real Anthropic PTY session. All three instances were then destroyed. *Not independently re-logged in CI yet* — the `readiness` gates re-prove the structural claims on every launch; the behavioural round-trip is a documented manual `exec` step. -- **The three server-backed seams proven live** (manual run, captured — real CI server - stacks stood up via `docker compose` from `microsoft/amplifier-context-intelligence`, - Neo4j-backed, bundle loaded from the Gitea mirror): +- **SUPERSEDED — the three server-backed seams' prior evidence** (captured against the + now-**retired** docker-compose backend path; kept here for historical record only, not + as current proof): real CI server stacks were stood up via `docker compose` from + `microsoft/amplifier-context-intelligence`, Neo4j-backed, bundle loaded from the Gitea + mirror: - **write to single server** — a real `amplifier` session's events reached the server: `/status` → `{workspace:"ci-write-single", events_processed:22}`, Cypher count **29** nodes tagged with that workspace. @@ -221,6 +213,34 @@ tools (see the main `README.md` §"read side"). For multi-source, the connectabl `get_config_overrides`). No framework limitation — the support issue was closed as invalid. (2) `graph_query`/`blob_read` are **not** top-level tools in a plain session — the shipped read path is the `context-intelligence:graph-analyst` agent, which the query profile drives. +- **PENDING — fresh non-compose evidence.** The three server-backed seams above must be + re-run against `context-intelligence-backend.yaml` (the current, non-compose backend) + before their proof is current again. No results are recorded here yet — a follow-up DTU + run will fill this in; do not treat the SUPERSEDED bullet above as still-valid proof. +- **Proven live — portable, no pinned values.** `context-intelligence-upload-format-validation.yaml` + (Legacy hooks-logging IMPORT) was launched against a **fresh** `context-intelligence-backend.yaml` + instance (Neo4j + standalone `context-intelligence-server`, non-compose) and reached + `check-readiness` → `ready: true`, with the backend's own `/status` confirming + `neo4j_connected: true`. The upload CLI was installed from the **branch under test**, resolved + through a **Gitea mirror** (git `insteadOf` + `UV_NO_GITHUB_FAST_PATH=true`) — confirmed by an + `exec` smoke (`BRANCH-CLI-OK`: `--help` advertises `--format logging-hook`, which only exists on + the branch). The consumer profile's `readiness` computed `EXPECTED_SESSIONS` and the expected + workspace set **at runtime** from the shipped fixture (never a pinned literal) and asserted, all + live against the graph: **discrimination** (native-format ingest reads only the generated + `context-intelligence/` twin, never the legacy `events.jsonl`), **coexistence** (the paired legacy + twin converges into the SAME workspaces with **no growth** in the runtime-captured node count), + **idempotency** (re-ingesting the same legacy fixture produces **no additional growth**), + **no-fork/slug parity** (the graph's distinct-workspace set exactly equals the runtime-derived + expected set, and per-workspace counts sum to the grand total), **session count** (`count_label + Session` equals the shipped session count), and **`data.timestamp` presence** on a read-back + event. All invariants held — this is a self-referential proof; no number was measured then + pinned anywhere in the profile, the support scripts, or this entry. Both DTU instances (backend + + consumer) were destroyed immediately after; `amplifier-digital-twin list` confirmed neither + remained. The host's production server on `:8000` was never targeted (the consumer profile only + ever pointed at the fresh backend's forwarded port via the Incus host-gateway IP), and the host's + `~/.amplifier/settings.yaml` was untouched. Machine-agnostic by construction: re-running this on + any host reproduces the same PASS/FAIL verdict from the same shipped fixture, regardless of that + host's own path, hostname, or prior graph state (given a genuinely fresh backend). - **Runtime-green is per-launch**, per the AGENTS.md rule — capture the run evidence (real request/response, provenance, fail-loud on a down/500/timeout) when you exercise a seam. Start with `context-intelligence-signals-validation.yaml` (no external deps) to diff --git a/.amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml b/.amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml new file mode 100644 index 00000000..91cebe8c --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml @@ -0,0 +1,199 @@ +# context-intelligence-backend.yaml +# +# CANONICAL non-compose Context-Intelligence backend for this bundle's DTU seam +# profiles. Stands up, inside ONE Incus container: +# * Neo4j Community 5.26.22 (5.26 LTS) as a Docker container, APOC Core bundled +# in the image (no network fetch), GDS intentionally OMITTED (the server calls +# no APOC/GDS procedures -- adding GDS would only add a network-egress +# dependency for parity we don't need) +# * the standalone `context-intelligence-server` (gunicorn + 1 uvicorn worker), +# run NON-COMPOSE, pointed at that Neo4j over bolt:// +# +# The server repo's own docker-compose stack is RETIRED as a DTU dependency (the +# compose files are being removed from microsoft/amplifier-context-intelligence +# itself -- see the open PR there). Every profile in this directory that needs a +# real Context-Intelligence backend launches THIS profile first, then points its +# own `--var server_url` / `--var server_a_url` / etc. at the port(s) forwarded +# here. There is no shared-include mechanism in the DTU profile schema (confirmed: +# no `!include`/anchor/fragment primitive), so this is the one canonical bring-up +# every validation profile's header comment points back to. +# +# WHY bolt:// (not neo4j://): neo4j:// is the routing-driver scheme -- on a +# Community single-node instance there is no cluster to route within, and the +# driver's routing table discovery adds a round-trip that gains nothing. bolt:// +# talks directly to the one instance. Confirmed against the server repo's +# docs/service-setup.md. +# +# PINNED (cited to the amplifier-context-intelligence submodule): +# * neo4j:5.26.22-community (neo4j.Dockerfile:26, docker-compose.yml:48 +# -- image tag only; compose itself is retired) +# * APOC Core, bundled in the image (README.md:129 -- no network fetch) +# * server URI MUST be bolt:// not neo4j:// on Community single-node +# (docs/service-setup.md:249-251) +# * WEB_CONCURRENCY MUST be 1 (hard server invariant) +# * server @ commit 766a9691850e6d7c29e7d4e90b537e88e69736bf (submodule HEAD, +# overridable via --var SERVER_REF) +# +# REQUIRED --var (the DTU substitution engine does exact ${VAR} text-replacement +# with NO fallback/default syntax -- confirmed against the engine's `_VAR_RE` +# regex, which has no `:-default` support -- so every var below must be passed +# explicitly at launch; "typical value" below is guidance, not an enforced default): +# --var NEO4J_PASSWORD= Neo4j auth password (NEO4J_AUTH=neo4j/) +# --var API_KEY= static bearer token for the CI server +# --var HOST_PORT= host port forwarded to container :8000 +# (typical: 38000 for a single instance) +# --var SERVER_REF= context-intelligence server git ref/commit +# (typical: 766a9691850e6d7c29e7d4e90b537e88e69736bf) +# +# LAUNCH -- single backend (e.g. for write-server-validation / query-validation): +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend \ +# --var NEO4J_PASSWORD= \ +# --var API_KEY= \ +# --var HOST_PORT=38000 \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# # then, from the host: find the Incus bridge gateway IP reachable from the +# # OTHER DTU (e.g. `ip route | grep default` inside that DTU, or `incus list` +# # for this container's address), and pass it to the consuming profile as +# # --var server_url=http://:38000 --var server_token= +# +# LAUNCH -- TWO backends for fan-out validation (distinct HOST_PORTs; each is a +# fully separate Incus container/Neo4j/server, so events never cross): +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend-main --var HOST_PORT=38001 \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend-team --var HOST_PORT=38002 \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# +# VERIFY (after readiness passes): +# amplifier-digital-twin exec -- curl -sf http://localhost:8000/status +# amplifier-digital-twin exec -- curl -sf -X POST http://localhost:8000/cypher \ +# -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ +# -d '{"query":"MATCH (n) RETURN count(n) AS n"}' +# +# Neo4j browser (:7474) and bolt (:7687) are intentionally NOT forwarded to the +# host -- only :8000 is, via `access.ports`. This avoids fixed high-port +# collisions between sibling DTUs (e.g. the two fan-out backends). Reach Neo4j +# directly via the container IP if needed: http://:7474/. + +name: context-intelligence-backend +description: > + Canonical non-compose Context-Intelligence backend (Neo4j Community 5.26.22 LTS + + standalone context-intelligence-server over bolt://) for this bundle's + server-backed DTU validation profiles. Replaces the retired docker-compose path. + +base: + image: ubuntu:24.04 + config: + limits.cpu: "4" + limits.memory: 6GiB + +access: + ports: + - host: ${HOST_PORT} + container: 8000 + label: CI ingestion server (/status, /events, /cypher, /dashboard) + path: / + # Neo4j browser (:7474) / bolt (:7687) intentionally NOT forwarded -- see + # header comment. Reach via the container IP if needed. + +provision: + setup_cmds: + - apt-get update && apt-get install -y ca-certificates curl gnupg lsb-release python3 jq + + # ---- Docker Engine (for the Neo4j container) ---- + - | + install -m 0755 -d /etc/apt/keyrings + curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc + chmod a+r /etc/apt/keyrings/docker.asc + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \ + https://download.docker.com/linux/ubuntu \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \ + | tee /etc/apt/sources.list.d/docker.list > /dev/null + apt-get update + apt-get install -y docker-ce docker-ce-cli containerd.io + + # ---- Start dockerd (no systemd as PID 1 inside Incus) ---- + - | + dockerd &>/var/log/dockerd.log & + for i in $(seq 1 30); do docker info &>/dev/null && break; sleep 1; done + docker info &>/dev/null || { echo "Docker failed to start"; cat /var/log/dockerd.log; exit 1; } + + # ---- Neo4j Community 5.26.22 (APOC Core bundled in the image), run DIRECTLY + # via `docker run` -- NOT docker-compose (the compose path is retired). bolt + # 7687 + browser 7474 bind to the Incus container's loopback only; the server + # reaches Neo4j at bolt://localhost:7687. GDS is intentionally omitted. + - | + mkdir -p /root/ci-data/neo4j /root/ci-data/blobs /root/ci-data/logs /root/ci-data/queues + docker run -d --name neo4j --restart unless-stopped \ + -p 7474:7474 -p 7687:7687 \ + -e NEO4J_AUTH=neo4j/${NEO4J_PASSWORD} \ + -e 'NEO4J_PLUGINS=["apoc"]' \ + -e 'NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.*' \ + -v /root/ci-data/neo4j:/data \ + neo4j:5.26.22-community + + # ---- Wait for Neo4j to be query-ready (HTTP tx returns 201) ---- + - | + for i in $(seq 1 60); do + code=$(curl -s -o /dev/null -w "%{http_code}" \ + -u "neo4j:${NEO4J_PASSWORD}" \ + http://localhost:7474/db/neo4j/tx \ + -H "Content-Type: application/json" \ + -d '{"statements":[{"statement":"RETURN 1"}]}') + [ "$code" = "201" ] && { echo "Neo4j ready"; break; } + sleep 3 + done + [ "$code" = "201" ] || { echo "Neo4j not ready (last=$code)"; docker logs neo4j | tail -40; exit 1; } + + # ---- Install uv + the standalone CI server (pinned via --var SERVER_REF) ---- + - curl -LsSf https://astral.sh/uv/install.sh | sh + - uv tool install "git+https://github.com/microsoft/amplifier-context-intelligence@${SERVER_REF}" + + # ---- server-config.yaml (static single-key auth; bolt:// URI) ---- + # ${NEO4J_PASSWORD} and ${API_KEY} are DTU --vars, substituted to literals by + # the engine BEFORE this quoted heredoc reaches the container shell. + - | + cat > /root/server-config.yaml << 'CFGEOF' + neo4j_url: bolt://localhost:7687 + neo4j_browser_url: http://localhost:7474 + neo4j_user: neo4j + neo4j_password: "${NEO4J_PASSWORD}" + api_key: "${API_KEY}" + server_host: 0.0.0.0 + server_port: 8000 + web_ui_enabled: true + log_level: INFO + blob_path: /root/ci-data/blobs + log_path: /root/ci-data/logs/server.jsonl + queues_path: /root/ci-data/queues + CFGEOF + echo "== server-config.yaml =="; cat /root/server-config.yaml + + # ---- Start the CI server (WEB_CONCURRENCY MUST be 1) ---- + - | + export AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_CONFIG_FILE=/root/server-config.yaml + export WEB_CONCURRENCY=1 + nohup context-intelligence-server > /var/log/ci-server.log 2>&1 & + for i in $(seq 1 60); do + curl -sf http://localhost:8000/status >/dev/null 2>&1 && break + sleep 2 + done + curl -sf http://localhost:8000/status | jq . || { echo "server not up"; tail -60 /var/log/ci-server.log; exit 1; } + +readiness: + - name: neo4j-bolt + tcp: + port: 7687 + + - name: server-neo4j-connected + http: + url: http://localhost:8000/status + expect_json: { "neo4j_connected": true } diff --git a/.amplifier/digital-twin-universe/profiles/context-intelligence-query-validation.yaml b/.amplifier/digital-twin-universe/profiles/context-intelligence-query-validation.yaml index c7ce0fdb..06ec2a29 100644 --- a/.amplifier/digital-twin-universe/profiles/context-intelligence-query-validation.yaml +++ b/.amplifier/digital-twin-universe/profiles/context-intelligence-query-validation.yaml @@ -26,15 +26,26 @@ # `source` provenance block) from the agent's response. # # HOW TO RUN -# # 1. stand up the server; 2. log some events to it first (e.g. run the write profile, or -# # just run one session in THIS DTU -- the hook writes as we read). Then: +# # 1. stand up the backend, non-compose, by launching this bundle's own +# # context-intelligence-backend.yaml DTU profile (see that file for the full +# # recipe + pinned versions): +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var HOST_PORT=38000 \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# # 2. log some events to it first (e.g. run the write-server-validation profile +# # against the same backend, or just run one session in THIS DTU -- the hook +# # writes as we read). Then, find the Incus host-gateway IP (via `incus list` +# # or `ip route | grep default` inside this DTU once launched): # export GH_TOKEN=$(gh auth token) # amplifier-digital-twin launch \ # .amplifier/digital-twin-universe/profiles/context-intelligence-query-validation.yaml \ # --name ci-query \ # --var gitea_host=http://localhost:10110 \ -# --var server_url=http://10.67.103.1:18001 \ -# --var server_token= \ +# --var server_url=http://:38000 \ +# --var server_token= \ # --var workspace=ci-query # # WHAT IS PROVEN diff --git a/.amplifier/digital-twin-universe/profiles/context-intelligence-upload-format-validation.yaml b/.amplifier/digital-twin-universe/profiles/context-intelligence-upload-format-validation.yaml new file mode 100644 index 00000000..7bcb7023 --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/context-intelligence-upload-format-validation.yaml @@ -0,0 +1,364 @@ +# context-intelligence-upload-format-validation.yaml +# +# RUNNABLE amplifier-digital-twin profile (base + provision). Proves the +# `context-intelligence-upload`'s `--format logging-hook` seam: the STANDALONE upload +# CLI, installed from a Gitea mirror of the branch under test, discovers a legacy +# hooks-logging session, transforms it in memory (non-destructive), and POSTs it +# through the same /events path to a real Context-Intelligence server, landing in +# the SAME workspace node as its paired native-CI twin. NOT pytest/artefacts. +# +# WHAT IS PROVEN (deterministic; no LLM -- asserted by readiness checks that build +# on this skeleton's install + fixtures + runtime-generated twin): +# * ISOLATION -- every assertion runs against a FRESH backend +# (see TOPOLOGY / HOW TO RUN below -- +# context-intelligence-backend.yaml launched +# FRESH, never reused from a prior run). +# * DISCRIMINATION -- `--format context-intelligence` ingests ONLY +# the native `context-intelligence/` twin; +# `--format logging-hook` ingests ONLY the +# top-level legacy `events.jsonl`. Neither +# format reads the other layer of the paired +# fixture. +# * RUNTIME SLUG PARITY / NO FORK -- the legacy `working_dir` slugifies (via the +# tool's OWN `legacy_transform.derive_workspace`, +# called live -- never re-implemented, never a +# literal) to the EXACT SAME workspace as the +# native twin generated from the SAME fixture +# at runtime (see gen_native_twin.py below), so +# the graph's distinct-workspace set, captured +# at RUNTIME, equals EXACTLY the runtime-derived +# expected set -- no escaped-hyphen fork. +# * COEXISTENCE / DEDUPE / +# IDEMPOTENCY -- ingesting the native twin then its paired +# legacy twin (identical event+workspace+data +# idempotency keys) produces NO growth in the +# node count captured at runtime on the second +# ingest. +# * data.timestamp VALUE -- the `data.timestamp` field on a chosen event +# is asserted to be PRESENT as a value (not +# merely non-empty), read back from the graph. +# +# This profile PINS NOTHING measured on any host: every workspace slug, node count, +# and timestamp value consumed by downstream assertions is computed AT RUNTIME, +# inside the container, from the SHIPPED fixture and the tool's own code -- never a +# literal captured from a prior run on this or any other machine. +# +# TOPOLOGY (two containers, same pattern as every other server-backed profile in +# this directory -- there is no shared-include mechanism in the DTU profile schema): +# host: a real Context-Intelligence server (Neo4j + API), stood up NON-COMPOSE by +# launching this bundle's own context-intelligence-backend.yaml DTU profile, +# launched FRESH. Its forwarded host port is reachable from THIS DTU at the +# Incus host-gateway IP (an example shape, never a pinned value -- find the +# real gateway IP per-run, see HOW TO RUN). +# DTU: the standalone `context-intelligence-upload` CLI, installed from the +# branch (resolved through Gitea -- see DESIGN DECISION below). No +# `amplifier` CLI session runs here -- this profile drives the upload tool +# directly, never a full amplifier bundle load. +# +# HOW TO RUN (two-step; the backend is REUSED across every readiness poll of this +# profile, never relaunched or duplicated) +# # 1. stand up the backend first, FRESH (see context-intelligence-backend.yaml +# # for the full non-compose recipe + pinned versions): +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-upload-backend \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var HOST_PORT= \ +# --var SERVER_REF= +# # Confirm it's reachable from the Incus host-gateway IP (find it via +# # `incus list` or `ip route | grep default` inside this DTU once launched): +# # curl -sf http://:/status +# export GH_TOKEN=$(gh auth token) +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-upload-format-validation.yaml \ +# --name ci-upload-format \ +# --var gitea_host= \ +# --var server_url=http://: \ +# --var server_token= +# +# REQUIRED --var (the DTU substitution engine does exact ${VAR} text-replacement +# with NO fallback/default syntax, so every var below must be passed explicitly): +# --var gitea_host= Gitea mirror base URL (e.g. http://localhost:10110) +# --var server_url= Context-Intelligence server URL, from the Incus +# host-gateway IP of the FRESH backend DTU above -- +# NEVER the host's own :8000. +# --var server_token= Bearer token for the backend's static API_KEY +# +# DESIGN DECISION (resolve the convention tension once, up front) +# The bundle convention elsewhere in this directory is "load through the +# Amplifier CLI via `amplifier bundle add` from a Gitea mirror." But the artifact +# under test here is a STANDALONE CLI (`context-intelligence-upload`), whose +# documented real-user install (modules/tool-context-intelligence-upload/README.md, +# "As a standalone CLI") is `uv tool install "... @ git+https://github.com/.../ +# amplifier-bundle-context-intelligence@main#subdirectory=modules/tool-context- +# intelligence-upload"`. `amplifier bundle add` does NOT put the +# `context-intelligence-upload` console script on PATH -- it only registers the +# bundle's behaviors/agents for an `amplifier` session, which this profile does +# not start. So the honest reconciliation for THIS profile is: +# +# Install the upload CLI from the branch code, resolved through Gitea, via the +# tool's own `uv tool install` path -- and exercise the ACTUAL +# `context-intelligence-upload`'s `--format logging-hook` command: +# 1. The profile installs `@main` (the permanent, post-merge form) plus the +# same `git config --global url."${gitea_host}/...".insteadOf +# "https://github.com/..."` redirect the other profiles use, so a +# developer's local branch (mirrored into Gitea) is what actually installs. +# 2. Because `uv` has a "GitHub fast path" that can bypass the git-level +# `insteadOf` redirect for `github.com` URLs, the profile EXPORTS +# `UV_NO_GITHUB_FAST_PATH=true` before `uv tool install` (documented in +# @digital-twin-universe:docs/profiles.md under `allow_uv_github_fast_path`). +# 3. Once installed, the profile resolves TOOL_PY -- the python interpreter +# INSIDE the tool's own uv-managed venv (`$(uv tool dir)//bin/python`) +# -- and uses it (never the system python3) to run the tool's own transform +# functions when generating the runtime native-CI twin (gen_native_twin.py) +# and, later, any runtime-derived readiness assertions. This keeps the +# fixture-vs-transform relationship exact: the SAME code under test produces +# the "expected" native twin, so nothing here is hand-authored or pinned. +# This keeps both truths intact: the CLI under test is exercised through its +# real, documented install path, and the code under test is the BRANCH (via +# Gitea), not upstream GitHub `main`. +# +# NOTE ON write-to-FILE / native-format ingest: covered by other profiles in this +# directory (write-server-validation, query-validation) -- not duplicated here. +# This profile is exclusively about the `--format logging-hook` legacy seam. +# +# NOTE ON SCOPE: base/header/install/fixtures/TOOL_PY resolution/runtime-twin- +# generation live in `provision` above; isolation + discrimination + slug parity +# + coexistence + idempotency + session-count + timestamp-presence assertions +# (all computed at runtime via portable_assert.sh, zero pinned values) live in +# the `readiness` section below. + +name: context-intelligence-upload-format-validation +description: > + Runnable proof of the standalone `context-intelligence-upload`'s `--format + logging-hook` seam. Installs the upload CLI from the branch under test via + `uv tool install`, resolved to a Gitea mirror (never GitHub upstream), resolves + the tool's own python interpreter (TOOL_PY), and generates the paired native-CI + twin for the shipped legacy fixture AT RUNTIME (never a pinned/committed twin) + so isolation, discrimination, slug parity, coexistence, and the data.timestamp + value can all be asserted against runtime-derived expectations, never + host-measured literals. + +base: + image: ubuntu:24.04 + +passthrough: + allow_external: true + services: + - name: github + key_env: GH_TOKEN + +provision: + files: + - src: ./support/upload-format/verify.sh + dest: /root/upload-format/verify.sh + mode: "0755" + - src: ./support/upload-format/portable_assert.sh + dest: /root/upload-format/portable_assert.sh + mode: "0755" + - src: ./support/upload-format/gen_native_twin.py + dest: /root/upload-format/gen_native_twin.py + mode: "0644" + - src: ./support/upload-format/fixtures/ + dest: /root/upload-format/ + recursive: true + # DTU directory-push basename rule (@digital-twin-universe:docs/api-reference.md): + # pushing ./support/upload-format/fixtures/ to /root/upload-format/ lands it at + # /root/upload-format/fixtures/... (basename preserved), so the legacy fixture + # tree lands at /root/upload-format/fixtures/legacy/sessions//... -- NOT + # directly at /root/upload-format/legacy/.... + + setup_cmds: + # (1) base tooling + uv + PATH wiring. PATH is written into BOTH + # /etc/environment and /root/.bashrc so every subsequent setup_cmds entry -- + # each its own `bash -lc` login-shell invocation -- picks up /root/.local/bin + # automatically. + - | + apt-get update && apt-get install -y git curl python3 jq + curl -LsSf https://astral.sh/uv/install.sh | sh + echo 'export PATH="/root/.local/bin:$PATH"' >> /root/.bashrc + echo 'PATH=/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' >> /etc/environment + + # (2) GH_TOKEN netrc + narrow git insteadOf redirect (only the + # context-intelligence bundle self-references -> mirror), copied from + # write-server-validation.yaml. + - | + if [ -n "${GH_TOKEN:-}" ]; then + echo "machine github.com login x-token-auth password $GH_TOKEN" > /root/.netrc + chmod 600 /root/.netrc + git config --global credential.helper 'store' + fi + + - | + git config --global \ + url."${gitea_host}/microsoft/amplifier-bundle-context-intelligence".insteadOf \ + "https://github.com/microsoft/amplifier-bundle-context-intelligence" + echo "insteadOf:"; git config --global --get-regexp insteadOf + + # (3) DESIGN DECISION: install the STANDALONE upload CLI (not `amplifier bundle + # add`, which does not expose this console script on PATH) via its own + # documented `uv tool install` path. UV_NO_GITHUB_FAST_PATH=true forces uv down + # the real git-fetch path so the insteadOf redirect above actually applies -- + # otherwise uv's GitHub fast path would resolve @main straight from upstream + # GitHub, silently bypassing the Gitea mirror of the branch under test. + - | + export UV_NO_GITHUB_FAST_PATH=true + uv tool install \ + "amplifier-module-tool-context-intelligence-upload @ git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main#subdirectory=modules/tool-context-intelligence-upload" + + # (4) Resolve TOOL_PY -- the python interpreter INSIDE the tool's own + # uv-managed venv (never system python3) -- fail loud if it can't import the + # installed package, then persist it (plus the verification-only server + # coordinates) to ci.env for every later setup_cmds entry (each its own shell) + # and for readiness checks to source. + - | + TOOL_PY="$(uv tool dir)/amplifier-module-tool-context-intelligence-upload/bin/python" + "$TOOL_PY" -c "import amplifier_module_tool_context_intelligence_upload" \ + || { echo "FATAL: TOOL_PY failed to import amplifier_module_tool_context_intelligence_upload: $TOOL_PY" >&2; exit 1; } + mkdir -p /root/upload-format + cat > /root/upload-format/ci.env << CIENVEOF + CI_BASE=${server_url} + CI_API_KEY=${server_token} + TOOL_PY=$TOOL_PY + CIENVEOF + echo "== ci.env written =="; cat /root/upload-format/ci.env + + # (5) Generate the native-CI twin for the shipped legacy fixture AT RUNTIME, + # using the tool's OWN transform (gen_native_twin.py) via TOOL_PY -- never a + # hand-authored or pre-committed twin. This is the single source of truth the + # follow-up readiness rebuild derives its expected workspace/count values from. + - | + source /root/upload-format/ci.env + "$TOOL_PY" /root/upload-format/gen_native_twin.py /root/upload-format/fixtures/legacy + + # (6) Fail-loud smoke: the branch CLI is on PATH and advertises --format + # logging-hook. + - | + export PATH="/root/.local/bin:$PATH" + context-intelligence-upload --help >/dev/null + context-intelligence-upload --help | grep -q 'logging-hook' + echo "ready: context-intelligence-upload on PATH, advertises --format logging-hook" + +# ----------------------------------------------------------------------------- +# READINESS (deterministic, poll-safe, no LLM). Every check here re-runs on +# EACH `check-readiness` invocation, so `ingest-discriminate-converge` below +# MUST be idempotent -- and, unlike an earlier iteration of this profile, it +# pins NOTHING measured on any host: EXPECTED_SESSIONS and EXPECTED_WS are +# both computed AT RUNTIME from the shipped fixture (find + the tool's own +# runtime_slug()), so the proof is self-referential and fully portable across +# machines. See portable_assert.sh for runtime_slug / assert_workspace_set / +# assert_no_growth / count_label, and verify.sh (sourced by it) for +# count_all / count_ws / poll_stable_ws / assert_labels / event_data_field. +# ----------------------------------------------------------------------------- +readiness: + - name: upload-cli-present + command: "context-intelligence-upload --help | grep -q logging-hook && echo ready" + + - name: server-reachable-from-dtu + command: > + . /root/upload-format/ci.env; + curl -sf "$CI_BASE/status" + | python3 -c "import sys,json;d=json.load(sys.stdin);assert d['status']=='ok' and d['neo4j_connected'];print('ready: server reachable, neo4j_connected')" + + - name: ingest-discriminate-converge + # DESIGN NOTE -- runtime-derived, self-referential isolation gate: + # count_all read after the FIRST (native-format) ingest becomes N_NATIVE, + # the baseline every later step is compared against via assert_no_growth. + # Any foreign data already in the backend would already be baked into + # N_NATIVE and would then have to ALSO be reproduced exactly by the + # legacy/re-ingest steps to still pass -- for all practical purposes this + # is only possible on a genuinely fresh backend, which is why one is a + # hard prerequisite (see TOPOLOGY / HOW TO RUN above -- + # context-intelligence-backend.yaml launched FRESH, never reused). + command: | + set -euo pipefail + source /root/upload-format/ci.env + source /root/upload-format/portable_assert.sh + + TREE=/root/upload-format/fixtures/legacy + + # ---- expected values, computed AT RUNTIME from the shipped fixture -- + # no pinned count, no literal slug anywhere below. ---- + EXPECTED_SESSIONS="$(find "$TREE/sessions" -mindepth 1 -maxdepth 1 -type d | wc -l)" + + EXPECTED_WS="" + for meta in "$TREE"/sessions/*/metadata.json; do + working_dir="$(jq -r '.working_dir' "$meta")" + ws="$(runtime_slug "$working_dir")" + EXPECTED_WS="$(printf '%s\n%s' "$EXPECTED_WS" "$ws")" + done + EXPECTED_WS="$(printf '%s' "$EXPECTED_WS" | sed '/^$/d' | sort -u)" + + # ---- (1) DISCRIMINATION: native-format ingest reads ONLY the + # generated context-intelligence/ twins under $TREE, never the legacy + # events.jsonl. ---- + context-intelligence-upload --format context-intelligence --path "$TREE" \ + --server-url "$CI_BASE" --api-key "$CI_API_KEY" + while IFS= read -r ws; do + [ -n "$ws" ] && poll_stable_ws "$ws" >/dev/null + done <<< "$EXPECTED_WS" + N_NATIVE="$(count_all)" + + # ---- (2) COEXISTENCE / NO-FORK / DEDUPE: the paired legacy twin + # (--format logging-hook) must slug-parity into the SAME workspaces as + # the native twin just ingested, producing NO growth in count_all. ---- + context-intelligence-upload --format logging-hook --path "$TREE" \ + --server-url "$CI_BASE" --api-key "$CI_API_KEY" + while IFS= read -r ws; do + [ -n "$ws" ] && poll_stable_ws "$ws" >/dev/null + done <<< "$EXPECTED_WS" + N_AFTER_LEGACY="$(count_all)" + assert_no_growth "$N_NATIVE" "$N_AFTER_LEGACY" 'native->legacy coexistence' + + # ---- (3) IDEMPOTENCY: re-ingesting the SAME legacy fixture must + # produce NO additional growth (identical event+workspace+data + # idempotency keys as the previous ingest). ---- + context-intelligence-upload --format logging-hook --path "$TREE" \ + --server-url "$CI_BASE" --api-key "$CI_API_KEY" + while IFS= read -r ws; do + [ -n "$ws" ] && poll_stable_ws "$ws" >/dev/null + done <<< "$EXPECTED_WS" + N_REINGEST="$(count_all)" + assert_no_growth "$N_AFTER_LEGACY" "$N_REINGEST" 'legacy re-ingest idempotency' + + # ---- (4) NO-FORK / SLUG PARITY: the graph's distinct-workspace set + # must EXACTLY equal the runtime-derived expected set (an + # escaped-hyphen fork would show up as an unexpected member), and the + # sum of per-workspace counts must equal the grand total -- proving no + # nodes landed outside the expected workspaces. ---- + assert_workspace_set "$EXPECTED_WS" + SUM_WS=0 + while IFS= read -r ws; do + [ -z "$ws" ] && continue + c="$(count_ws "$ws")" + SUM_WS=$((SUM_WS + c)) + done <<< "$EXPECTED_WS" + if [[ "$SUM_WS" -ne "$N_REINGEST" ]]; then + echo "FAIL: sum(count_ws over expected workspaces)=$SUM_WS != count_all=$N_REINGEST -- nodes landed outside the expected workspaces" >&2 + exit 1 + fi + + # ---- (5) SESSION COUNT: exactly one :Session node per legacy + # fixture session, regardless of how many times/formats it was + # ingested under (MERGE semantics keyed by session_id). ---- + N_SESSIONS="$(count_label Session)" + if [[ "$N_SESSIONS" -ne "$EXPECTED_SESSIONS" ]]; then + echo "FAIL: count_label(Session)=$N_SESSIONS != EXPECTED_SESSIONS=$EXPECTED_SESSIONS" >&2 + exit 1 + fi + + # ---- (6) LABELS + data.timestamp presence: structural value + # presence is asserted here, NOT a pinned timestamp literal (this MAY + # be strengthened later to compare against the fixture's own ts read + # live, but presence is sufficient to prove the field survived the + # legacy transform). ---- + assert_labels + FIRST_WS="$(printf '%s\n' "$EXPECTED_WS" | head -1)" + TS="$(event_data_field "$FIRST_WS" 'session:start' 'timestamp')" + if [[ -z "$TS" ]]; then + echo "FAIL: data.timestamp missing on session:start event in workspace=$FIRST_WS" >&2 + exit 1 + fi + + N_WORKSPACES="$(printf '%s\n' "$EXPECTED_WS" | wc -l)" + echo "ready: ingest+discriminate+converge proven (count_all=$N_REINGEST across $N_WORKSPACES workspace(s), sessions=$N_SESSIONS, labels present, data.timestamp present)" diff --git a/.amplifier/digital-twin-universe/profiles/context-intelligence-write-fanout-validation.yaml b/.amplifier/digital-twin-universe/profiles/context-intelligence-write-fanout-validation.yaml index f8fd75f7..301053ed 100644 --- a/.amplifier/digital-twin-universe/profiles/context-intelligence-write-fanout-validation.yaml +++ b/.amplifier/digital-twin-universe/profiles/context-intelligence-write-fanout-validation.yaml @@ -25,23 +25,38 @@ # implemented. This profile uses only that -- the patch workaround is fully removed.) # # TOPOLOGY -# host: TWO real Context-Intelligence servers (each Neo4j + API), stood up via docker -# compose on distinct ports, reachable from the DTU at the Incus host-gateway IP -# (e.g. http://10.67.103.1:18001 and http://10.67.103.1:18002). +# host: TWO real Context-Intelligence servers (each Neo4j + API), each stood up +# NON-COMPOSE by launching this bundle's own context-intelligence-backend.yaml +# DTU profile TWICE on distinct HOST_PORTs -- two fully separate Incus +# containers, so events never cross. Both forwarded ports are reachable from +# THIS DTU at the Incus host-gateway IP (e.g. http://10.67.103.1:38001 and +# http://10.67.103.1:38002). # DTU: amplifier + the branch bundle; the hook config carries TWO named destinations, # each include:["**"], so a single session fans out to both. # # HOW TO RUN -# # stand up TWO servers on the host (server repo docker-compose.yml; distinct ports + -# # data dirs + project names); confirm both: -# # curl -sf http://10.67.103.1:18001/status ; curl -sf http://10.67.103.1:18002/status +# # stand up TWO backends first (see context-intelligence-backend.yaml for the full +# # non-compose recipe + pinned versions) -- distinct --name and HOST_PORT each: +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend-main --var HOST_PORT=38001 \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend-team --var HOST_PORT=38002 \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# # confirm both (find the Incus gateway IP via `incus list` / `ip route`): +# # curl -sf http://:38001/status +# # curl -sf http://:38002/status # export GH_TOKEN=$(gh auth token) # amplifier-digital-twin launch \ # .amplifier/digital-twin-universe/profiles/context-intelligence-write-fanout-validation.yaml \ # --name ci-write-fanout \ # --var gitea_host=http://localhost:10110 \ -# --var server_a_url=http://10.67.103.1:18001 --var server_a_token= \ -# --var server_b_url=http://10.67.103.1:18002 --var server_b_token= \ +# --var server_a_url=http://:38001 --var server_a_token= \ +# --var server_b_url=http://:38002 --var server_b_token= \ # --var workspace=ci-write-fanout # # WHAT IS PROVEN diff --git a/.amplifier/digital-twin-universe/profiles/context-intelligence-write-server-validation.yaml b/.amplifier/digital-twin-universe/profiles/context-intelligence-write-server-validation.yaml index 4f684761..f69b7812 100644 --- a/.amplifier/digital-twin-universe/profiles/context-intelligence-write-server-validation.yaml +++ b/.amplifier/digital-twin-universe/profiles/context-intelligence-write-server-validation.yaml @@ -21,22 +21,32 @@ # to providers/tools/hooks alike, and IS implemented. This profile uses only that.) # # TOPOLOGY -# host: a real Context-Intelligence server (Neo4j + API) stood up via docker compose, -# reachable from the DTU at the Incus host-gateway IP (e.g. http://10.67.103.1:18001). +# host: a real Context-Intelligence server (Neo4j + API), stood up NON-COMPOSE by +# launching this bundle's own context-intelligence-backend.yaml DTU profile +# (Neo4j Community 5.26.22 LTS + standalone context-intelligence-server over +# bolt://). Its forwarded host port is reachable from THIS DTU at the Incus +# host-gateway IP (e.g. http://10.67.103.1:38000). # DTU: amplifier + the branch bundle (full context-intelligence behavior = design + logging). # # HOW TO RUN -# # 1. stand up the server on the host (server repo microsoft/amplifier-context-intelligence -# # docker-compose.yml -> start.sh generates credentials.yaml + prints the API token once). -# # Bind it to the Incus gateway IP so the container can reach it, and confirm: -# # curl -sf http://10.67.103.1:18001/status +# # 1. stand up the backend first (see context-intelligence-backend.yaml for the +# # full non-compose recipe + pinned versions): +# amplifier-digital-twin launch \ +# .amplifier/digital-twin-universe/profiles/context-intelligence-backend.yaml \ +# --name ci-backend \ +# --var NEO4J_PASSWORD= --var API_KEY= \ +# --var HOST_PORT=38000 \ +# --var SERVER_REF=766a9691850e6d7c29e7d4e90b537e88e69736bf +# # Confirm it's reachable from the Incus host-gateway IP (find it via +# # `incus list` or `ip route | grep default` inside this DTU once launched): +# # curl -sf http://:38000/status # export GH_TOKEN=$(gh auth token) # amplifier-digital-twin launch \ # .amplifier/digital-twin-universe/profiles/context-intelligence-write-server-validation.yaml \ # --name ci-write-single \ # --var gitea_host=http://localhost:10110 \ -# --var server_url=http://10.67.103.1:18001 \ -# --var server_token= \ +# --var server_url=http://:38000 \ +# --var server_token= \ # --var workspace=ci-write-single # # WHAT IS PROVEN diff --git a/.amplifier/digital-twin-universe/profiles/example-dtu-external-server.yaml b/.amplifier/digital-twin-universe/profiles/example-dtu-external-server.yaml index 18b69f13..f8245749 100644 --- a/.amplifier/digital-twin-universe/profiles/example-dtu-external-server.yaml +++ b/.amplifier/digital-twin-universe/profiles/example-dtu-external-server.yaml @@ -6,6 +6,13 @@ # The server and Neo4j are not installed inside the container -- this profile # only installs the client-side hook and bundle. # +# To get a real server to point at, launch this bundle's own +# context-intelligence-backend.yaml DTU profile (non-compose: Neo4j Community +# 5.26.22 LTS + standalone context-intelligence-server over bolt://), then pass +# its forwarded host port as CONTEXT_INTELLIGENCE_SERVER_URL below. See that +# profile's header comment for the full recipe, pinned versions, and how to +# launch it. (The server repo's own docker-compose path is retired.) +# # Events are tagged with a custom workspace name so they can be queried # independently from other sessions hitting the same server. # diff --git a/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-hyphen-svc/events.jsonl b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-hyphen-svc/events.jsonl new file mode 100644 index 00000000..4d151c37 --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-hyphen-svc/events.jsonl @@ -0,0 +1,3 @@ +{"event": "session:start", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-02T00:00:00.000+00:00", "session_id": "demo-hyphen-svc", "status": "ok", "data": {"working_dir": "/home/amplifier/workspaces/demo-hyphen-svc"}} +{"event": "tool:pre", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-02T00:00:01.000+00:00", "session_id": "demo-hyphen-svc", "status": "ok", "data": {"tool_name": "bash", "tool_input": {"command": "true"}}} +{"event": "session:end", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-02T00:00:02.000+00:00", "session_id": "demo-hyphen-svc", "status": "ok", "data": {}} diff --git a/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-hyphen-svc/metadata.json b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-hyphen-svc/metadata.json new file mode 100644 index 00000000..ff794c9a --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-hyphen-svc/metadata.json @@ -0,0 +1 @@ +{"working_dir": "/home/amplifier/workspaces/demo-hyphen-svc"} diff --git a/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-project/events.jsonl b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-project/events.jsonl new file mode 100644 index 00000000..f0805076 --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-project/events.jsonl @@ -0,0 +1,4 @@ +{"event": "session:start", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-01T00:00:00.000+00:00", "session_id": "demo-project", "status": "ok", "data": {"working_dir": "/home/amplifier/workspaces/demo-project"}} +{"event": "tool:pre", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-01T00:00:01.000+00:00", "session_id": "demo-project", "status": "ok", "data": {"tool_name": "bash", "tool_input": {"command": "echo hello"}}} +{"event": "tool:post", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-01T00:00:02.000+00:00", "session_id": "demo-project", "status": "ok", "data": {"tool_name": "bash"}} +{"event": "session:end", "schema": {"name": "amplifier.log", "ver": "1.0.0"}, "ts": "2026-01-01T00:00:03.000+00:00", "session_id": "demo-project", "status": "ok", "data": {}} diff --git a/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-project/metadata.json b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-project/metadata.json new file mode 100644 index 00000000..2799cdec --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/support/upload-format/fixtures/legacy/sessions/demo-project/metadata.json @@ -0,0 +1 @@ +{"working_dir": "/home/amplifier/workspaces/demo-project"} diff --git a/.amplifier/digital-twin-universe/profiles/support/upload-format/gen_native_twin.py b/.amplifier/digital-twin-universe/profiles/support/upload-format/gen_native_twin.py new file mode 100644 index 00000000..2d3d0188 --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/support/upload-format/gen_native_twin.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Generate the native-CI twin for each legacy session under a fixture root, +using the upload tool's OWN transform. Single source of truth: no hand-authored +event data, no machine-specific values -- every field is derived from the +committed legacy fixture. Run inside the DTU with the tool's python (TOOL_PY). + +Usage: gen_native_twin.py + /sessions//{events.jsonl,metadata.json} (input) + /sessions//context-intelligence/... (output) +""" + +import json +import sys +from pathlib import Path + +from amplifier_module_tool_context_intelligence_upload.legacy_transform import ( + derive_workspace, + reassemble_event_data, +) + + +def gen_session(session_dir: Path) -> str: + meta = json.loads((session_dir / "metadata.json").read_text(encoding="utf-8")) + working_dir = meta["working_dir"] + workspace = derive_workspace(working_dir) + sid = session_dir.name + + out_dir = session_dir / "context-intelligence" + out_dir.mkdir(parents=True, exist_ok=True) + + started_at = "" + lines = [] + for raw in (session_dir / "events.jsonl").read_text(encoding="utf-8").splitlines(): + if not raw.strip(): + continue + rec = json.loads(raw) + event_name, data = reassemble_event_data(rec) + ts = data.get("timestamp", "") + if not started_at: + started_at = ts + lines.append( + json.dumps({"event": event_name, "timestamp": ts, "workspace": workspace, "data": data}) + ) + (out_dir / "events.jsonl").write_text("\n".join(lines) + "\n", encoding="utf-8") + (out_dir / "metadata.json").write_text( + json.dumps( + { + "format": "context-intelligence", + "session_id": sid, + "parent_id": "", + "status": "completed", + "started_at": started_at, + "workspace": workspace, + } + ), + encoding="utf-8", + ) + return workspace + + +def main() -> int: + root = Path(sys.argv[1]) + sessions = sorted((root / "sessions").glob("*/metadata.json")) + if not sessions: + print(f"FATAL: no legacy sessions under {root}/sessions", file=sys.stderr) + return 1 + for meta_path in sessions: + ws = gen_session(meta_path.parent) + print(f"generated twin: {meta_path.parent.name} -> workspace={ws}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.amplifier/digital-twin-universe/profiles/support/upload-format/portable_assert.sh b/.amplifier/digital-twin-universe/profiles/support/upload-format/portable_assert.sh new file mode 100644 index 00000000..6227f639 --- /dev/null +++ b/.amplifier/digital-twin-universe/profiles/support/upload-format/portable_assert.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# portable_assert.sh — runtime-derived, self-referential assertions for the +# upload-format profile. Sources the proven verify.sh and adds ONLY helpers +# that COMPUTE their expected values at runtime from the SHIPPED fixture. +# NOTHING in here hardcodes a node count, slug, timestamp, or session id. +set -euo pipefail +_HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${_HERE}/verify.sh" + +: "${TOOL_PY:?FATAL: TOOL_PY must point at the python interpreter of the installed tool (set in ci.env)}" + +# runtime_slug — the canonical slug, computed live by the +# tool's OWN function (never re-implemented, never a literal). +runtime_slug() { + "$TOOL_PY" - "$1" <<'PY' +import sys +from amplifier_module_tool_context_intelligence_upload.legacy_transform import derive_workspace +print(derive_workspace(sys.argv[1])) +PY +} + +# count_label