feat(upload): add --format logging-hook legacy import (non-destructive, dedup-on) + portable DTU harness#75
Merged
Conversation
…e, dedup-on) + portable DTU harness
Add a new `--format {context-intelligence(default)|logging-hook}` flag to the
context-intelligence upload tool so operators can import legacy logging-hook
sessions into the same property graph as native context-intelligence data,
without any destructive rewrite of on-disk session files.
What changed:
- CLI: `--format` selects the discovery + transform path. `context-intelligence`
is the default and unchanged; `logging-hook` enables legacy import.
- Discovery discriminates by the `format` field in each session's metadata.json,
so legacy and native trees are classified deterministically (no misclassification).
- Legacy transform reuses the shared `build_payload` and the CI hook's own
`workspace_slug` / `config_resolver._slugify_path`, so migrated legacy data is
byte-identical to native: same event `data`, same workspace slug (including
Windows-origin paths), no workspace fork.
- Dedup is always-on for legacy import (idempotent re-ingest produces no node
growth). `--no-replay` combined with `--format logging-hook` is rejected with
exit code 2 (legacy import always dedups; the flag is meaningless there).
- Operator reconciliation summary plus zero-discovery, live-skip, and
partial-success signals for a clear import report.
- DTU profile alignment to the non-compose Neo4j-community backend.
- Ship a portable, self-contained `context-intelligence-upload-format-validation`
DTU regression harness: neutral synthetic fixture, runtime-derived assertions,
zero pinned values (no host paths, no measured node counts, no fixed slugs), so
it runs on any machine for future regression testing.
Docs & catalogue:
- Document `--format logging-hook` in the upload tool README + a pointer in the
bundle README; catalogue the new DTU profile in the profiles README inventory
and the AGENTS profile roster.
Scope / guardrails:
- Write-side hook fan-out (fanout.py / logging_handler) is untouched.
- Transform reuses shared `build_payload` + the hook's `workspace_slug` so the
two formats cannot diverge.
- Retiring amplifier-ci-migrate is OUT OF SCOPE.
🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)
Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
…y drop Review of PR #75 surfaced two issues; both fixed here: 1. Write-side was NOT untouched as claimed. Relative to merge-base, the branch had removed the 404/410 "endpoint not found" diagnostic branch from _DestinationDispatcher._worker, deleted its 5 regression tests (TestPermanentMessagePolarity), and rolled uv.lock back past the Entra token fix (dropping azure-identity). Restored all three write-side hook files to merge-base state so the module is genuinely untouched and the PR's read/import-side-only claim holds. Hook module: 576 passed. 2. Silent data loss in legacy discovery: a genuinely-legacy session whose first >5 records are corrupt failed the bounded sniff and was dropped with no counter — invisible in the reconciliation summary. Now surfaced as a counted, fail-loud `unclassified` signal in the operator summary, with a regression test. Upload module: 337 passed. ruff + ruff format clean; pyright 0 errors.
colombod
added a commit
that referenced
this pull request
Jul 16, 2026
…on-blank read lines Two follow-up fixes from the PR #75 review: 1. Symlink parity: the live CI hook stamps a session's workspace via config_resolver._slug_from_working_dir(), which resolves symlinks (Path(working_dir).resolve()) before slugifying. The legacy import's derive_workspace() delegated to the hook's abspath-only low-level slugifier instead, so a working_dir traversing a symlink would import data into a DIFFERENT workspace than the hook wrote — latent drift the 0/3051 corpus sweep never hit because none of those sessions crossed a symlink. Align _slugify_path() to resolve first (preserving the empty -string default-fallback case), update the 4 parity tests to the resolved oracle, and add a real-symlink regression test. 2. Reconciliation arithmetic: _count_lines() counted all physical lines including blanks, but the operator summary's `read` field is documented as a non-blank event-line count and the upload loop skips blanks — so `read` could exceed ingested+skipped+unmapped from trailing blank lines. Count only non-blank lines so the summary reconciles by construction. Verified: 342 passed (ground-truth sweep 3286 ws_diffs=0, 400 data-parity 0 defects); ruff + ruff format clean; pyright 0 errors. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
--format {context-intelligence(default)|logging-hook}flag to the context-intelligence upload tool so operators can import legacy logging-hook sessions into the same property graph as native context-intelligence data, non-destructively. Legacy discovery discriminates by each session's metadata.jsonformatfield; the legacy transform reuses the sharedbuild_payloadand the CI hook's ownworkspace_slug/config_resolver._slugify_path, so the payload envelope and workspace slug are structurally identical to native (same workspace slug incl. Windows-origin paths, no workspace fork); the legacy→CI eventdatareassembly is guarded against drift by a real-data ground-truth parity oracle (0/3051 workspace diffs), not by structural sharing. Dedup is always-on for legacy import (idempotent re-ingest → no node growth);--no-replay+--format logging-hookis rejected with exit 2. Ships an operator reconciliation summary plus zero-discovery / live-skip / partial-success signals, aligns the DTU profiles to the non-compose Neo4j-community backend, and adds a portable, self-containedcontext-intelligence-upload-format-validationDTU regression harness (neutral synthetic fixture, runtime-derived assertions, zero pinned values) that anyone can run on any machine.Scope / guardrails
fanout.py/_DestinationDispatcher/logging_handler) is not modified — this change is read/import-side only.build_payloadand the hook's ownworkspace_slug/config_resolver._slugify_path, so the payload envelope and workspace slug cannot drift structurally. The legacy→CIdatareassembly (PROMOTED_KEYS) is hand-mirrored and guarded against drift by the ground-truth parity oracle (test-asserted), not structurally impossible. Windows-origin path parity delegated to the same hook leaf function.amplifier-ci-migrateis deliberately not part of this PR.Verification
modules/tool-context-intelligence-upload→ 336 passed (incl. ground-truth parity sweep: 0/3051 workspace diffs, 0 reassembly defects)tests/→ 753 passedruff check+ruff format --checkcleanpyrightclean — 0 errors, 0 warningsscripts/validate-full.sh→validation_mode: full, overall PASS, 10/10 bundles good,bundle.dotfresh. The lone mode-advertising ERROR is the DOCUMENTED false positive per AGENTS.md — not "fixed".Real evidence on seams (not mock-only)
The
--format logging-hookimport crosses the client↔server upload seam, so it is proven against a real backend, not a mock:The shipped portable
context-intelligence-upload-format-validationDTU profile was launched against a fresh, isolated Neo4j + context-intelligence server backend, and all runtime-derived invariants held:data.timestampvalue presentIsolation verified: backend reached via the Incus gateway IP (never host
:8000); no leftover containers; hostsettings.yamlunpolluted. Additional real evidence: coexistence proof on real paired sessions — 474 == 474 nodes, single workspace; ground-truth parity sweep 0/3051 across the real corpus. Every assertion is runtime-derived — the harness pins no measured node count, slug, or host path, so it is valid on any machine for future regression testing.Docs & diagrams
bundle.dot/bundle.png— regenerated/confirmed fresh byvalidate-full.sh(no structural change; validator does not flagBUNDLE_DOT_STALE)--format logging-hookdocumented in the upload tool README (import path, dedup-always-on,--no-replayexit 2, discrimination, exit codes) + a pointer in the bundle README; the new DTU profile catalogued in.amplifier/digital-twin-universe/profiles/README.mdinventory and the AGENTS profile rosterNotes / follow-ups
amplifier-ci-migrateis out of scope for this PR and tracked separately.🤖 Generated with Amplifier