This is an outline of security-relevant statements across the project that overstate what the code actually does. Each needs a maintainer decision: either implement the stated behavior or soften the claim. No PR is attached for these on purpose — the right direction is a design/positioning call. (Related accuracy-only fixes already filed separately: pilot-protocol/website#138, #139, #140; pilot-protocol/release#36; #408 here.)
1. install.sh claims SLSA attestation + signed-manifest verification that the script does not perform
Where: release/install.sh — header comment (“Verifies SHA-256 of the tarball against checksums.txt AND the signed manifest”) and the auto-update consent text (“release checksums are verified against a GitHub SLSA provenance attestation (bound to the exact release tag) before anything is replaced”).
Reality: the script computes SHA-256 and compares it against (a) checksums.txt from the GitHub release and (b) the per-platform hash in latest.json served from pilotprotocol.network. There is no signature verification (no cosign/minisign/gpg) and no attestation verification (no slsa-verifier, no gh attestation verify) anywhere in the script. Both anchors are operator-controlled, so they are not independent against vendor-side compromise.
Decision: implement real attestation verification in the updater path, or reword to “SHA-256 checksum verification against the release manifest.” The checksum handling itself is genuinely fail-closed and worth keeping/advertising accurately.
2. Homepage: “block prompt-injection attacks before it ever reads them” (AEGIS)
Where: website homepage AEGIS copy.
Reality: the AEGIS blog post itself is honest — its threat table lists “Bypassed by: attacks that look like benign text (false negatives possible)” and its FAQ says defense-in-depth is required. Pattern/signature matching cannot deterministically block semantic prompt injection; an absolute “before it ever reads them” claim is not defensible and contradicts the product’s own documentation.
Decision: reword to a filtering/screening claim (e.g. “screens inbound messages against known injection patterns”) or gate the absolute claim on evidence.
3. “HIPAA-Compliant Agent Communication” title
Where: website blog post title/slug.
Reality: a transport protocol cannot “be HIPAA compliant” — compliance attaches to covered entities/business associates via BAAs, administrative and physical safeguards. The post’s own body concedes this (“does not provide everything needed for a fully compliant healthcare AI system”). Two body claims also overreach: “AES-256-GCM satisfies the transmission security requirement (164.312(e))” (that safeguard is addressable and organizational, not satisfied by a cipher choice), and “FIPS-adjacent … crypto/tls supports FIPS 140-2 mode” (Pilot’s tunnel uses AES-256-GCM/X25519 directly, not crypto/tls, so the FIPS-mode point does not apply to Pilot’s encryption path).
Decision: retitle (e.g. “Building toward HIPAA-eligible agent communication”) and correct the two body claims.
4. Docs: “random nonce prefix … to prevent replay”
Where: website /docs/concepts encryption section.
Reality: a per-connection random nonce prefix ensures AES-GCM nonce uniqueness (prevents catastrophic nonce reuse); it does not prevent replay. Replay protection requires sequence numbers with an anti-replay window (or equivalent).
Decision: if the session layer has an anti-replay window, document that as the replay defense; if it does not, this is a protocol gap worth fixing, not just a wording fix.
5. Uninstall leaves injected agent instructions behind
Where: release/install.sh uninstall path.
Reality: uninstall stops services and rm -rfs ~/.pilot (deleting pilotctl) but never runs pilotctl skills disable all, so injected skill/directive blocks persist in ~/.claude/, OpenClaw/OpenHands/etc. after removal — and the tool that could clean them is gone. For a feature whose consent story is “opt-out anytime,” removal should actually remove it.
Decision: run skill removal before deleting ~/.pilot in the uninstall path.
6. Minor: deployed docs drift
The deployed /docs/mcp-setup still shows “Full P2P; no third party sees metadata” for local mode (the registry/beacon necessarily see connection metadata; relayed paths carry traffic). The copy on website main is already honest — a redeploy would resolve it; flagging in case the deploy tracks another branch.
Filed after a full-site accuracy audit; happy to turn any accepted direction above into a PR.
🤖 Generated with Claude Code
https://claude.ai/code/session_01D2boz7GoTW34JmzymSjCpf
This is an outline of security-relevant statements across the project that overstate what the code actually does. Each needs a maintainer decision: either implement the stated behavior or soften the claim. No PR is attached for these on purpose — the right direction is a design/positioning call. (Related accuracy-only fixes already filed separately: pilot-protocol/website#138, #139, #140; pilot-protocol/release#36; #408 here.)
1. install.sh claims SLSA attestation + signed-manifest verification that the script does not perform
Where:
release/install.sh— header comment (“Verifies SHA-256 of the tarball against checksums.txt AND the signed manifest”) and the auto-update consent text (“release checksums are verified against a GitHub SLSA provenance attestation (bound to the exact release tag) before anything is replaced”).Reality: the script computes SHA-256 and compares it against (a)
checksums.txtfrom the GitHub release and (b) the per-platform hash inlatest.jsonserved from pilotprotocol.network. There is no signature verification (no cosign/minisign/gpg) and no attestation verification (noslsa-verifier, nogh attestation verify) anywhere in the script. Both anchors are operator-controlled, so they are not independent against vendor-side compromise.Decision: implement real attestation verification in the updater path, or reword to “SHA-256 checksum verification against the release manifest.” The checksum handling itself is genuinely fail-closed and worth keeping/advertising accurately.
2. Homepage: “block prompt-injection attacks before it ever reads them” (AEGIS)
Where: website homepage AEGIS copy.
Reality: the AEGIS blog post itself is honest — its threat table lists “Bypassed by: attacks that look like benign text (false negatives possible)” and its FAQ says defense-in-depth is required. Pattern/signature matching cannot deterministically block semantic prompt injection; an absolute “before it ever reads them” claim is not defensible and contradicts the product’s own documentation.
Decision: reword to a filtering/screening claim (e.g. “screens inbound messages against known injection patterns”) or gate the absolute claim on evidence.
3. “HIPAA-Compliant Agent Communication” title
Where: website blog post title/slug.
Reality: a transport protocol cannot “be HIPAA compliant” — compliance attaches to covered entities/business associates via BAAs, administrative and physical safeguards. The post’s own body concedes this (“does not provide everything needed for a fully compliant healthcare AI system”). Two body claims also overreach: “AES-256-GCM satisfies the transmission security requirement (164.312(e))” (that safeguard is addressable and organizational, not satisfied by a cipher choice), and “FIPS-adjacent … crypto/tls supports FIPS 140-2 mode” (Pilot’s tunnel uses AES-256-GCM/X25519 directly, not crypto/tls, so the FIPS-mode point does not apply to Pilot’s encryption path).
Decision: retitle (e.g. “Building toward HIPAA-eligible agent communication”) and correct the two body claims.
4. Docs: “random nonce prefix … to prevent replay”
Where: website
/docs/conceptsencryption section.Reality: a per-connection random nonce prefix ensures AES-GCM nonce uniqueness (prevents catastrophic nonce reuse); it does not prevent replay. Replay protection requires sequence numbers with an anti-replay window (or equivalent).
Decision: if the session layer has an anti-replay window, document that as the replay defense; if it does not, this is a protocol gap worth fixing, not just a wording fix.
5. Uninstall leaves injected agent instructions behind
Where:
release/install.shuninstall path.Reality: uninstall stops services and
rm -rfs~/.pilot(deletingpilotctl) but never runspilotctl skills disable all, so injected skill/directive blocks persist in~/.claude/, OpenClaw/OpenHands/etc. after removal — and the tool that could clean them is gone. For a feature whose consent story is “opt-out anytime,” removal should actually remove it.Decision: run skill removal before deleting
~/.pilotin the uninstall path.6. Minor: deployed docs drift
The deployed
/docs/mcp-setupstill shows “Full P2P; no third party sees metadata” for local mode (the registry/beacon necessarily see connection metadata; relayed paths carry traffic). The copy onwebsitemain is already honest — a redeploy would resolve it; flagging in case the deploy tracks another branch.Filed after a full-site accuracy audit; happy to turn any accepted direction above into a PR.
🤖 Generated with Claude Code
https://claude.ai/code/session_01D2boz7GoTW34JmzymSjCpf