Added the intelligence CLI: Intelligence Packages, trusted registries, a lockfile and npm distribution (0.11.0)

[dzykovic]

Summary

The product interface becomes a CLI. npm i -g @ainova-systems/intelligence installs one intelligence command owning the whole lifecycle — init, add, remove, install, update, upgrade, list, search, sync, doctor, status, registry, migrate — with versioned Intelligence Packages (git-tag semver), a committed intelligence.lock, and name resolution only through the project's trusted registries (no built-in catalog, no name→github guessing; vendor defaults exist solely as manifest lines init seeds — visible, reviewable, deletable). The engine's own content ships as the @ainova-systems/sync package (auto-selected at init, exact-pinned to the engine, materialized from the npm bundle offline); the distribution's identity is data in cli/engine-package.yaml, so a fork edits one file and no code. The vendored (v1) flow keeps working unchanged and byte-identically — it only gains a one-line stderr recommendation to migrate; intelligence migrate converts a vendored project transactionally with --dry-run.

Full record: the ## [0.11.0] section in CHANGELOG.md.

Type of change

• New capability (CLI, packages, registries, lockfile, npm distribution) — minor, 0.11.0
• Documentation (docs/CLI.md + vendored mirror, CONVENTIONS/ADAPTERS token and CLI-mode sections, README quickstart, ROADMAP)

Verification

• legacy-golden CI job: vendored outputs for the smoke examples are byte-identical vs main across every engine change; the one intentional content change (mode-aware meta-skills, <manifest> token) is declared with [golden-skip] and recorded in the CHANGELOG
• 5 hermetic suites (unit-semver, unit-manifest, e2e-packages, e2e-lifecycle, e2e-negative — 180+ cases incl. migrate rollback, frozen-integrity, dangerous-name guards) green on ubuntu + macos; npm-pack-smoke green on ubuntu + macos + windows (spaced global prefix through the Node shim)
• Live end-to-end on the published prerelease (0.11.0-rc.6, dist-tag next): init → add @ainova-systems/core (resolved via the seeded registry from the real intelligence-dev-packs) → sync → fresh-clone install --frozen → doctor
• CHANGELOG.md has the dated ## [0.11.0] section; lockstep bump done: scripts/VERSION, the sync_version example in INIT.md, every examples/*/config.yaml, and examples/cli-project/intelligence.yaml is included in the stamp check
• shellcheck --severity=warning clean for the engine and cli/

Notes for reviewers

• Compatibility contract: old downstream update.sh pulls main HEAD and copies only intelligence/sync/* — that path is untouched and remains mandatory-complete; the engine's CLI mode is entirely env-gated (IS_CLI=1), with every variable unset behavior is bit-identical (CI-enforced).
• After merge (house release style, directly on main): tag v0.11.0 + GitHub release, then dispatch release-npm with channel=latest from the tag — that publishes the stable npm version and permanently takes over the latest dist-tag.
• The adversarial design record (why the engine-content package is bundle-seeded and exact-pinned rather than a plain ranged package) is summarized in the CHANGELOG entry and docs/CLI.md.

[Copilot]

Pull request overview

This pull request turns intelligence-sync into a CLI-driven product distributed via npm, while preserving the legacy “vendored engine in-repo” flow as a compatibility contract. It introduces a v2 project layout (intelligence.yaml + intelligence.lock + .intelligence/ store), a package/registry/lockfile lifecycle, and an engine “CLI mode” (IS_CLI=1) so the same sync engine can run from outside the repo.

Changes:

• Added the intelligence CLI (bash dispatcher + per-command scripts) with package resolution, trusted registries, and a committed lockfile.
• Extended the existing sync engine to support CLI-driven execution (env contract, new layout tokens like <sync-cmd> / <manifest>, and output-path protections for root-manifest projects).
• Added CI workflows and npm packaging tooling to build, smoke-test, and publish @ainova-systems/intelligence.

Reviewed changes

Copilot reviewed 65 out of 67 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
ROADMAP.md	Documents the product concept and future ordered work.
README.md	Adds CLI quickstart and positions CLI as primary interface.
npm/README.md	Package README for npm distribution, CLI usage overview.
npm/package.json	Defines the npm package metadata and intelligence bin entry.
npm/build.sh	Assembles npm/dist from repo sources and injects version.
npm/bin/intelligence.js	Node shim that locates bash (esp. Windows) and execs the CLI.
intelligence/sync/skills/intelligence-update/SKILL.md	Updates update guidance; adds CLI-setup redirect guidance.
intelligence/sync/skills/intelligence-uninstall-adapter/SKILL.md	Parameterizes paths/tokens for both vendored + CLI modes.
intelligence/sync/skills/intelligence-sync/SKILL.md	Updates sync instructions to use <sync-cmd> and <manifest>.
intelligence/sync/skills/intelligence-review-skills/SKILL.md	Updates path resolution guidance to <umbrella>/<module>/<manifest> model.
intelligence/sync/skills/intelligence-learn-from-context/SKILL.md	Updates layout discovery language to tokenized paths.
intelligence/sync/skills/intelligence-install-adapter/SKILL.md	Updates adapter install steps to tokenized manifest/paths.
intelligence/sync/skills/intelligence-extract-skill/SKILL.md	Updates authoring references to <umbrella> / <module>.
intelligence/sync/skills/intelligence-add-skill/SKILL.md	Updates authoring steps to use <umbrella> and <manifest>.
intelligence/sync/skills/intelligence-add-rule/SKILL.md	Updates rule authoring steps to use <umbrella> and <manifest>.
intelligence/sync/skills/intelligence-add-agent/SKILL.md	Updates agent authoring steps and tokenized rule references.
intelligence/sync/scripts/VERSION	Bumps engine version to 0.11.0.
intelligence/sync/scripts/update.sh	Adds stderr-only note recommending CLI migration.
intelligence/sync/scripts/sync.sh	Adds CLI mode gate/contract and vendored-mode recommendation note.
intelligence/sync/scripts/lib/common.sh	Adds <sync-cmd>/<manifest> token expansion and CLI-mode output protection.
intelligence/sync/scripts/adapters/agents.sh	Updates AGENTS.md header to use CLI-provided sync command when available.
intelligence/sync/rules/intelligence-authoring.md	Updates authoring rule to use <manifest> and <sync-cmd> tokens.
intelligence/sync/INIT.md	Recommends CLI for new projects; bumps example sync_version.
intelligence/sync/docs/CONVENTIONS.md	Documents new tokens and the CLI-mode env contract.
intelligence/sync/docs/CLI.md	Adds CLI documentation (vendored mirror).
intelligence/sync/docs/ADAPTERS.md	Documents tokens including <sync-cmd>/<manifest> and CLI-mode module path.
intelligence/sync/agents/intelligence-operator.md	Updates build/verify guidance to <sync-cmd>.
intelligence/sync/agents/intelligence-architect.md	Updates build/verify guidance to <sync-cmd>.
examples/with-remote-skills/config.yaml	Lockstep bump to sync_version: 0.11.0.
examples/platform-with-submodules/config.yaml	Lockstep bump to sync_version: 0.11.0.
examples/go-api/config.yaml	Lockstep bump to sync_version: 0.11.0.
examples/go-api-with-pi-and-codex/config.yaml	Lockstep bump to sync_version: 0.11.0.
examples/go-api-with-opencode/config.yaml	Lockstep bump to sync_version: 0.11.0.
examples/dotnet-api-with-react-frontend/config.yaml	Lockstep bump to sync_version: 0.11.0.
examples/cli-project/intelligence.yaml	Adds a v2 CLI-layout example manifest.
docs/CONVENTIONS.md	Updates tokens and documents CLI mode (repo-root docs).
docs/CLI.md	Adds CLI documentation (repo-root docs).
docs/ADAPTERS.md	Updates adapter docs for new tokens + CLI mode (repo-root docs).
cli/tests/unit-semver.sh	Adds hermetic unit tests for semver-over-tags.
cli/tests/unit-manifest.sh	Adds hermetic unit tests for manifest/lockfile editors.
cli/tests/e2e-packages.sh	Adds hermetic e2e tests for add/install/update/remove/registry.
cli/tests/e2e-negative.sh	Adds negative-path e2e tests (refusals, rollback, frozen integrity).
cli/tests/e2e-lifecycle.sh	Adds e2e tests for init/migrate/doctor/upgrade workflows.
cli/lib/semver.sh	Implements semver comparison and git-tag range resolution.
cli/lib/registry.sh	Implements registry trust-list resolution and package fetch.
cli/lib/manifest.sh	Implements CLI-owned quoted-key YAML parsing/editing.
cli/lib/lockfile.sh	Implements intelligence.lock read/write helpers.
cli/lib/cli-common.sh	Shared CLI plumbing + engine-content package identity + env contract export.
cli/intelligence	CLI dispatcher that discovers bundled engine and execs commands.
cli/engine-package.yaml	Data-driven distribution identity (engine content package + default registry).
cli/commands/upgrade.sh	Implements CLI-side upgrade/migrations to align project with bundled engine.
cli/commands/update.sh	Re-resolves ranges and rewrites lockfile; skips engine-content package.
cli/commands/sync.sh	Runs sync in v2 mode (env contract) or delegates to vendored engine in legacy mode.
cli/commands/status.sh	Reports project mode and key versioning/installation state.
cli/commands/search.sh	Lists packages available via trusted registries and their local state.
cli/commands/remove.sh	Removes a package and unwires sources/lock/store (guarded for engine-content pkg).
cli/commands/registry.sh	Manages trusted registry URLs (list/add/remove; index.yaml verification).
cli/commands/migrate.sh	Transactional v1→v2 conversion (stage/verify/commit with rollback).
cli/commands/list.sh	Lists manifest packages with requested/locked/installed state.
cli/commands/install.sh	Restores .intelligence/ from lockfile (supports --frozen, staging, integrity checks).
cli/commands/init.sh	Creates a minimal v2 manifest, gitignore entry, optional engine-content package install.
cli/commands/doctor.sh	Read-only consistency checker; exits nonzero on problems.
cli/commands/add.sh	Adds packages via registry name or explicit source specs; wires sources + lock.
CHANGELOG.md	Adds 0.11.0 release notes describing CLI, packages, lockfile, and CI changes.
.gitignore	Ignores npm build artifacts (npm/dist/, *.tgz).
.github/workflows/release-npm.yml	Adds manual workflow to build, smoke-test, and publish to npm.
.github/workflows/ci.yml	Adds CLI shellcheck/e2e suites, npm pack smoke tests, and legacy golden-output check.

Suppressed comments (1)

CHANGELOG.md:37

• This “Fixed” entry says intelligence add fell through to a GitHub-convention URL when no registry declared a name, but the current implementation errors out when no trusted registry declares @scope/name. If the fallback was removed, the changelog should describe the real silent-failure scenario that was fixed (e.g., git ls-remote failures for explicit sources).

- **`intelligence add` died silently on a nonexistent conventional repo.** A name no registry declares falls through to `github.com/org/name`; when that repo did not exist, `git ls-remote` failed inside a `pipefail` command substitution and the command exited with no message. The source is now probed up front, and the error explains how the name resolved and suggests near-miss names from every trusted registry ("Did you mean: `intelligence add @ainova-systems/spec`").

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dzykovic]

Adversarial review — consensus only

Five independent reviewers. Items below were raised by at least two models and checked against the committed diff. Lone-model findings omitted. Vendored IS_CLI=0 looks structurally gated; the new CLI is not merge-ready.

Blockers

Merge-blocking. Do not land 0.11.0 with these open.

1. Package names from lock/manifest are used as paths without assert_valid_pkg_name

The validator exists because name segments feed rm -rf, but only add / remove / doctor / migrate call it. install and update take keys as-is:

# cli/commands/install.sh — same pattern in update.sh
rel=".intelligence/packages/$name"
rm -rf "${IP_ROOT:?}/$rel"

A committed lock/manifest key @x/../../../../OUTSIDE is enough: intelligence install after a fresh clone writes or deletes outside the store. Validate every key from qmap_keys / lock_to_tsv before any filesystem work (or reject inside those readers).

2. Git calls do not close the option list — URL starting with - is RCE

git ls-remote --tags "$url" (cli/lib/semver.sh) and git clone … "$url" / git checkout --quiet "$ref" (cli/lib/registry.sh) pass the value as the first positional. --upload-pack=<cmd> is executed. Reproduced with a committed packages."@acme/evil".url of --upload-pack=touch …;git-upload-pack under intelligence install.

Vectors: lock/manifest of a cloned repo (including install --frozen in CI) and url: in a registry index.yaml. The engine already allowlists schemes in fetch_remote_source; the CLI does not.

Fix: git … -- "$url", scheme allowlist (https/http/ssh/git/file), and a validated ref.

3. migrate mutates the live vendored project; rollback does not mean “untouched”

docs/CLI.md / the command header say any earlier failure leaves the project untouched. On the non-dry-run path, a schema gap runs run_migrations "$umbrella" + stamp_version against the live tree (cli/commands/migrate.sh). Dry-run already uses a staged copy.

rollback() only removes .intelligence, intelligence.yaml, intelligence.lock and restores .gitignore. A stamped/migrated config.yaml is left behind, and the “vendored setup is untouched” message is false.

Run the engine chain only on the staged copy (same as --dry-run); wire INT/TERM after the commit phase to rollback.

4. --frozen trusts an on-disk store that already exists

# cli/commands/install.sh
if [ -d "$IP_ROOT/$rel" ] && [ "$force" -eq 0 ]; then
    continue
fi

Lock sha is compared only on fetch. Under --frozen, a pre-existing (including substituted) store is synced as trusted. The comment in the same file already notes that the dir-exists skip bypasses verification. Under --frozen: always refetch + sha, or wipe the store first.

Lower priority — consider

Not merge-blocking. Fix after the blockers, or in a follow-up.

• update unwires sources before fetch_package. A failed clone leaves the package unwired; store/lock still claim it; doctor prints all-good; next sync silently drops the content. Fetch into staging, then unwire + commit.
• migrate pack names @$org/$repo are not uniqued. Two legacy packs with the same org/repo (or same basename on different hosts) overwrite one store dest and duplicate the lock key. Fail closed on collision.
• rm -rf "$root/$mirror" has no containment check. A packs.*.mirror of ../outside deletes outside the repo after a successful sync. Canonicalize and refuse .. before staging and before delete.
• Predictable temp paths: ${TMPDIR:-/tmp}/intelligence-cli-index-$$ (trusted on cache hit, never cleaned) and /tmp/intelligence-lock-$$.tsv (symlink write). Use mktemp -d / mktemp + EXIT trap.
• Lock/manifest writers do not YAML-escape " in url/path/sha. A quote in a registry URL breaks or aliases neighbouring fields.
• Dispatcher cli/intelligence: commands/$cmd.sh with no [a-z-]+ allowlist — intelligence ../../../tmp/x execs a neighbouring .sh.
• git+ grammar: git+git@host:org/repo.git (SCP, no / after @) is parsed as a ref; git+…@feature/foo (slash in the ref) is not parsed as a ref.
• legacy-golden is skipped for the whole branch via [golden-skip]. The content skip is declared; the cost is that unset-IS_CLI behaviour is no longer proven in CI. Split content vs env-gated behaviour.

Lower priority — nits

• sync.sh fallback hardcodes @ainova-systems/sync despite identity living in cli/engine-package.yaml.

[dzykovic]

Adversarial review — addressed in 5b9ea22

All four blockers fixed, plus every "consider" follow-up and the nit. A hostile-input CI scenario now exercises the vectors directly (RCE payload never runs, traversal key refused, dispatcher rejects path-shaped words).

Blockers

1. Untrusted names → rm -rf paths. install and update now call assert_valid_pkg_name on every key from qmap_keys/lock_to_tsv before any filesystem work; the validator also rejects empty/dot segments. @x/../../OUTSIDE is refused up front.
2. Argument-injection RCE. Every git call closes its option list (git … -- "$url", checkout "$ref" --), and assert_safe_source_url/assert_safe_ref enforce a scheme allowlist (https/http/ssh/git/file or scp user@host:path) and reject option-shaped/quoted values — at _fetch_index, list_remote_versions, remote_tag_for_version, fetch_package, add's probe, and registry add. A --upload-pack=… url is refused.
3. migrate mutating the live tree. The engine chain now always runs on a staged copy (never in place); an INT/TERM rollback trap is wired from the first moved file until the verified sync, then swapped to stage-cleanup. "Untouched on any earlier failure" holds literally.
4. --frozen trusting on-disk store. Under --frozen the store is always refetched and sha-verified (the dir-exists skip is bypassed).

Follow-ups

• update fetches into staging, then unwires + commits (failed clone leaves the install intact).
• migrate derived pack names uniqued — collision fails closed.
• mirror: containment-checked (reject /, ..) before rm -rf.
• Index cache + lock temp files use unpredictable mktemp paths (fixed the predictable -$$ and the symlink-write vector).
• Lock/manifest writers YAML-escape "/\.
• Dispatcher allowlists the command word ([a-z-]+) — intelligence ../../x is rejected.
• git+user@host:repo scp grammar parsed correctly (slash/colon in the tail is a host, not a ref).
• legacy-golden split: new always-on legacy-golden-code job holds content at main and swaps only intelligence/sync/scripts, proving engine code changes are byte-neutral for vendored output — independent of the content [golden-skip].

Nit

• CLI-mode IS_MODULE_REL carries no vendor default; a bare IS_CLI=1 sync now fails closed (identity comes only from the CLI / cli/engine-package.yaml).

CHANGELOG gains a ### Security section. All five suites green locally; CI running.

[dzykovic]

Superseded by the v1/v2 repository split — not merged here by design.

This work is now the main branch of https://github.com/ainova-systems/intelligence, with its full history: every commit on feature/intelligence-cli was pushed there as the starting point of the v2 product, and the repository has since moved to the v2 layout (cli/, engine/, packages/sync/).

Merging it here would have left this repository as v1 with v2 scaffolding grown into it — the wrong thing to keep as the vendored lineage. Instead the two lineages split at d124af6 (v0.10.1): this repository stays the vendored (v1) engine at the URL its clients already clone, so a frozen update.sh keeps working with no action from anyone.

Rationale and consequences: decisions/0001-split-v1-archive-from-v2-product.md. The review discussion on this PR stays readable here.