diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a9c49c..5d06ea9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,19 @@ All notable changes follow [Keep a Changelog](https://keepachangelog.com/en/1.1. ## [Unreleased] +### Added + +- The unreleased package identity is now `0.2.0` for the claim/evidence export contract. +- `demo` and `run` can export a complete claim/evidence recurrence artifact with package + attribution, packet hashes, the full trace, and a named stopping decision. +- `verify-evidence` validates the unsigned outer digest, every evidence-packet hash, and derived + claim/decision summaries offline; conflicting artifact writes are refused. +- Bounded untrusted evidence to 1 MiB, 32 JSON levels, 50,000 nodes, and 1,024 packets; + malformed UTF-8, duplicate keys, conflicting summaries, and recomputed trace tampering fail + closed. +- Remote OpenAI-compatible credentials now require HTTPS, with a loopback-only HTTP exception; + credential-bearing URLs and unredacted transport errors are rejected. + ## [0.1.0] - 2026-08-06 ### Added diff --git a/CITATION.cff b/CITATION.cff index b69feea..bcccb28 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -2,8 +2,7 @@ cff-version: 1.2.0 message: "If you use VerifAxis, please cite the software." title: "VerifAxis" type: software -version: 0.1.0 -date-released: 2026-08-06 +version: 0.2.0 authors: - name: Ali repository-code: "https://github.com/aliengineering-byte/verifaxis" diff --git a/README.md b/README.md index 3854db1..bc67270 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,26 @@ $ verifaxis demo This is `smoke/demo` output, not a benchmark result. +Persist the complete claim, evidence chain, and named stopping decision, then validate it offline: + +```console +$ verifaxis demo --evidence-output demo-evidence.json +$ verifaxis verify-evidence demo-evidence.json +{ + "decision": "VERIFIED", + "evidence_packets": 2, + "status": "EVIDENCE ARTIFACT VERIFIED" +} +``` + +The artifact contains the final explicit claim, both the failing and passing verifier packets, +candidate and packet hashes, the full recurrence trace, `VERIFIED` stopping reason, package +attribution, and limitations. Its outer hash detects accidental or unrecomputed changes; packet +hashes and derived summaries are checked separately. This is self-consistency, not authentication: +an editor can recompute the unsigned hash, and hash validity does not make a verifier correct or +complete. The CLI accepts an identical existing artifact but refuses to overwrite different +content. + ## Architecture VerifAxis implements **Verifier-Conditioned External Recurrence (VCER)**: @@ -66,7 +86,7 @@ flowchart LR The loop persists candidates, concise structured state, evidence hashes, residuals, budgets, and termination decisions. It neither requests nor stores private chain-of-thought. LLM-generated criticism is always marked as LLM-produced and never silently treated as independent evidence. -The black-box adapter supports OpenAI-compatible HTTP endpoints, including compatible local servers. Open-weight latent recurrence is an interface-level future direction only; v0.1 makes no claim that it works. +The black-box adapter supports OpenAI-compatible HTTP endpoints, including compatible local servers. It rejects API keys and credential-like headers over plain HTTP unless the endpoint is explicitly loopback (`localhost` or a loopback IP), rejects credentials embedded in URLs, bounds and strictly parses responses, and does not include endpoint error details that may contain secrets. Open-weight latent recurrence is an interface-level future direction only; VerifAxis makes no claim that it works. ## What VerifAxis does not guarantee @@ -82,11 +102,21 @@ See the [threat model](docs/threat-model.md) and [novelty decision](docs/novelty ```bash verifaxis demo -verifaxis run examples/arithmetic.yaml +verifaxis run examples/arithmetic.yaml --evidence-output claim-evidence.json +verifaxis verify-evidence claim-evidence.json verifaxis bench --config configs/smoke.yaml verifaxis report runs/latest --format html ``` +Evidence output is complete by design: it contains the task, candidates, verifier packets, +counterexamples, and timestamps. Choose a sanitized input or protect the destination as sensitive +data. Output is no-clobber; because timestamps make a fresh run different, use a new filename (or +deliberately remove the old local artifact) when repeating a demo. Validate only artifacts from +trusted sources: `verify-evidence` treats files as untrusted strict UTF-8 JSON, rejects duplicate +keys, and limits the document to 1 MiB, 32 JSON levels, 50,000 JSON nodes, and 1,024 evidence +packets before deriving the claim, trace chain, and decision. These bounds mitigate local resource +exhaustion; they do not authenticate an unsigned artifact. + Run all offline checks: ```bash @@ -113,6 +143,7 @@ class MyVerifier: ``` Start with `src/verifaxis/verifiers/` and the security boundaries in [CONTRIBUTING.md](CONTRIBUTING.md). +Report a sanitized defect with the [bug form](https://github.com/aliengineering-byte/verifaxis/issues/new?template=bug.yml), or discuss a verifier/research question with the [research form](https://github.com/aliengineering-byte/verifaxis/issues/new?template=research.yml). Never attach private prompts, credentials, or unredacted evidence. ## Research status diff --git a/SECURITY.md b/SECURITY.md index b34af84..03aca9b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,7 +2,7 @@ ## Supported versions -Security fixes target the latest `0.1.x` revision on `main` while the project is pre-release. +Security fixes target the latest pre-release revision on `main`. ## Reporting @@ -10,6 +10,6 @@ Use GitHub's private vulnerability reporting for `aliengineering-byte/verifaxis` ## Security model -Model candidates, prompts, endpoint responses, verifier output, counterexamples, artifacts, and configuration files are untrusted. Default verifiers never run arbitrary candidate code. The math and restricted-function paths interpret allowlisted syntax only. The OpenAI-compatible adapter sends data only to the endpoint explicitly configured by the caller. +Model candidates, prompts, endpoint responses, verifier output, counterexamples, artifacts, and configuration files are untrusted. Default verifiers never run arbitrary candidate code. The math and restricted-function paths interpret allowlisted syntax only. The OpenAI-compatible adapter sends data only to the endpoint explicitly configured by the caller, requires HTTPS for remote credentials, allows credentialed HTTP only for loopback testing, and redacts transport error details. -Out of scope for v0.1: arbitrary-code sandboxes, multi-tenant hosting, authentication, secret storage, and network retrieval. See `docs/threat-model.md`. +Out of scope: arbitrary-code sandboxes, multi-tenant hosting, authentication, secret storage, and network retrieval. See `docs/threat-model.md`. diff --git a/docs/architecture.md b/docs/architecture.md index 14e6b09..64b14c8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -24,6 +24,7 @@ p_{z+1} = A_phi(p_z, h_z, Encode(e_z)) | `EvidenceResidual` | Explicit unresolved/failed/conflicting constraints | Data, not free-form hidden reasoning | | `VerificationController` | Continue, verify, abstain, or stop on a named failure mode | Cannot promote LLM criticism to independent proof | | `RunTrace` | Persist steps, accounting, evidence, residuals, termination | JSON-safe and auditable | +| Claim/evidence artifact | Bind an explicit final claim to the recurrence, packet hashes, and named stopping decision | Tamper evidence is not proof that the verifier is correct | Provider, verifier, controller, trace storage, and reporting boundaries remain separate so experiments can change one factor at a time. diff --git a/docs/threat-model.md b/docs/threat-model.md index f884dc2..340534e 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -21,9 +21,9 @@ Protect the host, secrets, local files, network, trace integrity, experimental v | False verifier pass | Independence/reliability metadata, conflict checks, fault testing, false-verification metric | A single trusted verifier can be wrong or incomplete | | LLM critique laundering | `llm_generated` is explicit; LLM-only evidence cannot verify | Incorrect integration metadata | | Replay/stale evidence | Candidate/claim binding and stale-evidence faults | Weak semantic claim binding | -| Resource exhaustion | Bounded model/verifier calls, iterations, tokens, and runtime accounting | HTTP endpoints enforce their own hard limits | -| Secret leakage | No keys in config/examples/traces; environment-based endpoint credentials; secret scans | Provider request logs and user-supplied prompts | -| Unsafe deserialization | JSON only for public config/trace paths | JSON size/depth denial of service without caller limits | +| Resource exhaustion | Bounded model/verifier calls, iterations, tokens, response bytes/JSON shape, and evidence file/JSON/packet counts | HTTP endpoints enforce their own server-side limits | +| Secret leakage | No keys in config/examples/traces; remote credentials require HTTPS; loopback-only HTTP credential exception; endpoint errors are redacted; secret scans | Provider request logs, caller-supplied custom header names, and user-supplied prompts | +| Unsafe deserialization | JSON only; evidence rejects malformed UTF-8, duplicate keys, files over 1 MiB, depth over 32, over 50,000 nodes, and over 1,024 packets | Callers that bypass the bounded file loader must impose equivalent transport limits | | Misleading research claim | Frozen contract, raw paired results, explicit smoke labels | Human interpretation and selective reporting | ## Arbitrary code diff --git a/paper/claims.md b/paper/claims.md index 5b4f6fe..da9480f 100644 --- a/paper/claims.md +++ b/paper/claims.md @@ -6,6 +6,7 @@ - Default arithmetic and restricted-function verifiers do not execute arbitrary candidate Python. - The controller exposes named success, budget, plateau, oscillation, conflict, unverifiable, model-error, and verifier-error outcomes. - The smoke harness can exercise named baselines and controlled fault types offline. +- The CLI can export and independently validate a tamper-evident claim/evidence recurrence artifact. These are engineering claims, not model-quality findings. diff --git a/pyproject.toml b/pyproject.toml index e89356b..4913e5c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,11 +4,11 @@ build-backend = "hatchling.build" [project] name = "verifaxis" -version = "0.1.0" +version = "0.2.0" description = "Verifier-conditioned recurrence for frozen language models, with executable evidence, adaptive stopping, and reproducible evaluation." readme = "README.md" requires-python = ">=3.11" -license = { text = "Apache-2.0" } +license = "Apache-2.0" authors = [{ name = "Ali" }] keywords = ["llm", "verification", "test-time-compute", "reproducible-research"] classifiers = [ diff --git a/src/verifaxis/__init__.py b/src/verifaxis/__init__.py index ba5c278..43b464e 100644 --- a/src/verifaxis/__init__.py +++ b/src/verifaxis/__init__.py @@ -1,6 +1,16 @@ """VerifAxis public API.""" +from importlib.metadata import version as package_version + +__version__ = package_version("verifaxis") + from .controller import VerificationController +from .evidence import ( + build_claim_evidence_artifact, + load_and_validate_claim_evidence_artifact, + validate_claim_evidence_artifact, + write_claim_evidence_artifact, +) from .interfaces import ModelAdapter, Verifier from .runtime import verify from .types import ( @@ -32,5 +42,10 @@ "VerificationController", "VerificationResult", "Verifier", + "__version__", + "build_claim_evidence_artifact", + "load_and_validate_claim_evidence_artifact", + "validate_claim_evidence_artifact", "verify", + "write_claim_evidence_artifact", ] diff --git a/src/verifaxis/cli.py b/src/verifaxis/cli.py index 6b23335..c5e43f3 100644 --- a/src/verifaxis/cli.py +++ b/src/verifaxis/cli.py @@ -9,10 +9,16 @@ from pathlib import Path from typing import Any +from . import __version__ from .bench import load_config, run_benchmark +from .evidence import ( + load_and_validate_claim_evidence_artifact, + write_claim_evidence_artifact, +) from .models import ReplayModel from .reporting import canonical_json, load_run, write_report from .runtime import verify +from .types import VerificationResult from .verifiers import SafeMathVerifier @@ -37,7 +43,7 @@ def _name(value: Any, *, default: str) -> str: return default -def _run_spec(path: str | Path) -> dict[str, Any]: +def _run_spec(path: str | Path) -> VerificationResult: spec = _json_file(path) allowed = {"schema_version", "task", "model", "verifiers", "max_iterations"} unknown = set(spec) - allowed @@ -62,10 +68,10 @@ def _run_spec(path: str | Path) -> dict[str, Any]: [SafeMathVerifier() for _ in verifier_names], max_iterations=max_iterations, ) - return result.to_dict() + return result -def _demo() -> int: +def _demo(args: argparse.Namespace) -> int: task = "What is 197 * 83?" result = verify(task, ReplayModel(), [SafeMathVerifier()], max_iterations=4) payload = { @@ -78,13 +84,28 @@ def _demo() -> int: "model_calls": result.trace.model_calls, "verifier_calls": result.trace.verifier_calls, } + if args.evidence_output is not None: + evidence_path = write_claim_evidence_artifact( + result, + args.evidence_output, + producer_version=__version__, + ) + payload["evidence_artifact"] = str(evidence_path) sys.stdout.write(canonical_json(payload)) return 0 if result.verified else 1 def _run(args: argparse.Namespace) -> int: result = _run_spec(args.config) - rendered = canonical_json(result) + payload = result.to_dict() + if args.evidence_output is not None: + evidence_path = write_claim_evidence_artifact( + result, + args.evidence_output, + producer_version=__version__, + ) + payload["evidence_artifact"] = str(evidence_path) + rendered = canonical_json(payload) if args.output is None: sys.stdout.write(rendered) else: @@ -95,6 +116,12 @@ def _run(args: argparse.Namespace) -> int: return 0 +def _verify_evidence(args: argparse.Namespace) -> int: + validation = load_and_validate_claim_evidence_artifact(args.artifact) + sys.stdout.write(canonical_json(validation)) + return 0 + + def _bench(args: argparse.Namespace) -> int: config = load_config(args.config) result = run_benchmark(config, args.output) @@ -121,11 +148,22 @@ def build_parser() -> argparse.ArgumentParser: ) subparsers = parser.add_subparsers(dest="command", required=True) - subparsers.add_parser("demo", help="run the deterministic arithmetic smoke demo") + demo_parser = subparsers.add_parser("demo", help="run the deterministic arithmetic smoke demo") + demo_parser.add_argument( + "--evidence-output", help="optional complete claim/evidence artifact destination" + ) run_parser = subparsers.add_parser("run", help="run a JSON-valid YAML task config") run_parser.add_argument("config", help="path to the run configuration") run_parser.add_argument("--output", help="optional JSON trace destination") + run_parser.add_argument( + "--evidence-output", help="optional complete claim/evidence artifact destination" + ) + + verify_evidence_parser = subparsers.add_parser( + "verify-evidence", help="validate a claim/evidence artifact offline" + ) + verify_evidence_parser.add_argument("artifact", help="artifact JSON path") bench_parser = subparsers.add_parser("bench", help="run deterministic smoke benchmarks") bench_parser.add_argument("--config", required=True, help="benchmark configuration path") @@ -145,13 +183,15 @@ def main(argv: Sequence[str] | None = None) -> int: args = parser.parse_args(argv) try: if args.command == "demo": - return _demo() + return _demo(args) if args.command == "run": return _run(args) if args.command == "bench": return _bench(args) if args.command == "report": return _report(args) + if args.command == "verify-evidence": + return _verify_evidence(args) except (OSError, ValueError) as error: parser.error(str(error)) parser.error(f"unknown command: {args.command}") diff --git a/src/verifaxis/evidence.py b/src/verifaxis/evidence.py new file mode 100644 index 0000000..30e1b74 --- /dev/null +++ b/src/verifaxis/evidence.py @@ -0,0 +1,263 @@ +"""Tamper-evident claim/evidence artifacts for one verification decision.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import cast + +from .types import JSONValue, VerificationResult, content_digest + +EVIDENCE_ARTIFACT_SCHEMA_VERSION = "1.0" +MAX_EVIDENCE_BYTES = 1_048_576 +MAX_EVIDENCE_JSON_DEPTH = 32 +MAX_EVIDENCE_JSON_NODES = 50_000 +MAX_EVIDENCE_PACKETS = 1_024 + + +def _as_object(value: object, context: str) -> dict[str, JSONValue]: + if not isinstance(value, dict) or not all(isinstance(key, str) for key in value): + raise ValueError(f"{context} must be a JSON object") + return cast(dict[str, JSONValue], value) + + +def _as_array(value: object, context: str) -> list[JSONValue]: + if not isinstance(value, list): + raise ValueError(f"{context} must be a JSON array") + return cast(list[JSONValue], value) + + +def _reject_duplicate_keys(pairs: list[tuple[str, JSONValue]]) -> dict[str, JSONValue]: + result: dict[str, JSONValue] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON key {key!r}") + result[key] = value + return result + + +def _enforce_json_limits(value: JSONValue) -> None: + stack: list[tuple[JSONValue, int]] = [(value, 1)] + nodes = 0 + while stack: + current, depth = stack.pop() + nodes += 1 + if nodes > MAX_EVIDENCE_JSON_NODES: + raise ValueError(f"evidence exceeds {MAX_EVIDENCE_JSON_NODES} JSON nodes") + if depth > MAX_EVIDENCE_JSON_DEPTH: + raise ValueError(f"evidence exceeds JSON depth {MAX_EVIDENCE_JSON_DEPTH}") + if isinstance(current, dict): + stack.extend((item, depth + 1) for item in current.values()) + elif isinstance(current, list): + stack.extend((item, depth + 1) for item in current) + + +def _packet_summary(packet: dict[str, JSONValue]) -> dict[str, JSONValue]: + return { + "verifier_type": packet.get("verifier_type"), + "verifier_version": packet.get("verifier_version"), + "status": packet.get("status"), + "checked_claim": packet.get("checked_claim"), + "independence": packet.get("independence"), + "llm_produced": packet.get("llm_produced"), + "content_hash": packet.get("content_hash"), + } + + +def build_claim_evidence_artifact( + result: VerificationResult, + *, + producer_version: str, +) -> dict[str, JSONValue]: + """Build a self-contained claim, evidence-chain, and stopping-decision record.""" + + if not producer_version.strip(): + raise ValueError("producer_version must not be empty") + candidate = result.candidate + evidence_chain: list[JSONValue] = [] + for step in result.trace.steps: + evidence_chain.append( + { + "iteration": step.iteration, + "candidate_fingerprint": step.candidate.fingerprint, + "packets": [_packet_summary(packet.to_dict()) for packet in step.evidence], + } + ) + payload: dict[str, JSONValue] = { + "schema_version": EVIDENCE_ARTIFACT_SCHEMA_VERSION, + "producer": { + "repository": "aliengineering-byte/verifaxis", + "version": producer_version, + "capability": "verifier-conditioned-claim-decision", + "documentation": "https://github.com/aliengineering-byte/verifaxis#before-and-after", + }, + "claim": { + "task": result.trace.task, + "candidate": None if candidate is None else candidate.content, + "candidate_fingerprint": None if candidate is None else candidate.fingerprint, + }, + "evidence_chain": evidence_chain, + "decision": { + "status": result.status.value, + "verified": result.verified, + "stopping_reason": result.status.value, + "iterations": len(result.trace.steps), + "model_calls": result.trace.model_calls, + "verifier_calls": result.trace.verifier_calls, + }, + "trace": result.trace.to_dict(), + "reproduction": { + "command_template": "verifaxis run --evidence-output ", + "validation_command_template": "verifaxis verify-evidence ", + }, + "limitations": [ + "A VERIFIED decision establishes only the checked properties within the " + "recorded verifiers' scope.", + "The unsigned artifact hash detects unrecomputed changes but is not authentication; " + "an editor can recompute it.", + "Self-consistency does not establish verifier correctness or external timestamp trust.", + ], + } + payload["artifact_hash"] = content_digest(payload) + return payload + + +def validate_claim_evidence_artifact(value: object) -> dict[str, JSONValue]: + """Validate the artifact hash, every packet hash, and derived decision summaries.""" + + artifact = _as_object(value, "evidence artifact") + _enforce_json_limits(artifact) + if artifact.get("schema_version") != EVIDENCE_ARTIFACT_SCHEMA_VERSION: + raise ValueError("unsupported evidence artifact schema_version") + producer = _as_object(artifact.get("producer"), "producer") + if producer.get("repository") != "aliengineering-byte/verifaxis": + raise ValueError("producer repository is not aliengineering-byte/verifaxis") + if producer.get("capability") != "verifier-conditioned-claim-decision": + raise ValueError("producer capability is not verifier-conditioned-claim-decision") + if not isinstance(producer.get("version"), str) or not producer["version"]: + raise ValueError("producer version must not be empty") + expected_hash = artifact.get("artifact_hash") + if not isinstance(expected_hash, str): + raise ValueError("evidence artifact requires artifact_hash") + unsigned = dict(artifact) + del unsigned["artifact_hash"] + actual_hash = content_digest(unsigned) + if actual_hash != expected_hash: + raise ValueError( + f"evidence artifact hash mismatch: expected {expected_hash}, got {actual_hash}" + ) + + trace = _as_object(artifact.get("trace"), "trace") + steps = _as_array(trace.get("steps"), "trace.steps") + derived_chain: list[JSONValue] = [] + packet_count = 0 + for index, raw_step in enumerate(steps): + step = _as_object(raw_step, f"trace.steps[{index}]") + candidate = _as_object(step.get("candidate"), f"trace.steps[{index}].candidate") + raw_packets = _as_array(step.get("evidence"), f"trace.steps[{index}].evidence") + if packet_count + len(raw_packets) > MAX_EVIDENCE_PACKETS: + raise ValueError(f"evidence exceeds {MAX_EVIDENCE_PACKETS} packets") + summaries: list[JSONValue] = [] + for packet_index, raw_packet in enumerate(raw_packets): + packet = _as_object( + raw_packet, + f"trace.steps[{index}].evidence[{packet_index}]", + ) + packet_hash = packet.get("content_hash") + if not isinstance(packet_hash, str): + raise ValueError("evidence packet requires content_hash") + unsigned_packet = dict(packet) + del unsigned_packet["content_hash"] + if content_digest(unsigned_packet) != packet_hash: + raise ValueError( + f"evidence packet hash mismatch at iteration {step.get('iteration')}" + ) + summaries.append(_packet_summary(packet)) + packet_count += 1 + derived_chain.append( + { + "iteration": step.get("iteration"), + "candidate_fingerprint": candidate.get("fingerprint"), + "packets": summaries, + } + ) + if artifact.get("evidence_chain") != derived_chain: + raise ValueError("evidence_chain does not match the embedded trace") + + claim = _as_object(artifact.get("claim"), "claim") + if steps: + final_step = _as_object(steps[-1], "trace final step") + final_candidate = _as_object(final_step.get("candidate"), "trace final candidate") + expected_claim = { + "task": trace.get("task"), + "candidate": final_candidate.get("content"), + "candidate_fingerprint": final_candidate.get("fingerprint"), + } + else: + expected_claim = { + "task": trace.get("task"), + "candidate": None, + "candidate_fingerprint": None, + } + if claim != expected_claim: + raise ValueError("claim does not match the embedded final trace candidate") + + decision = _as_object(artifact.get("decision"), "decision") + if decision.get("status") != trace.get("termination_reason"): + raise ValueError("decision status does not match trace termination_reason") + if decision.get("stopping_reason") != trace.get("termination_reason"): + raise ValueError("decision stopping_reason does not match trace termination_reason") + expected_verified = trace.get("termination_reason") == "VERIFIED" + if decision.get("verified") is not expected_verified: + raise ValueError("decision verified flag does not match trace termination_reason") + if decision.get("iterations") != len(steps): + raise ValueError("decision iteration count does not match trace") + if decision.get("model_calls") != trace.get("model_calls"): + raise ValueError("decision model_calls does not match trace") + if decision.get("verifier_calls") != trace.get("verifier_calls"): + raise ValueError("decision verifier_calls does not match trace") + return { + "schema_version": EVIDENCE_ARTIFACT_SCHEMA_VERSION, + "status": "EVIDENCE ARTIFACT VERIFIED", + "artifact_hash": expected_hash, + "decision": decision.get("status"), + "evidence_packets": packet_count, + } + + +def write_claim_evidence_artifact( + result: VerificationResult, + path: str | Path, + *, + producer_version: str, +) -> Path: + """Write one claim/evidence artifact to an explicit local destination.""" + + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + artifact = build_claim_evidence_artifact(result, producer_version=producer_version) + rendered = ( + json.dumps(artifact, indent=2, sort_keys=True, ensure_ascii=False, allow_nan=False) + "\n" + ) + if target.exists(): + if not target.is_file() or target.read_text(encoding="utf-8") != rendered: + raise FileExistsError(f"refusing to overwrite existing evidence artifact {target}") + else: + target.write_text(rendered, encoding="utf-8", newline="\n") + return target + + +def load_and_validate_claim_evidence_artifact(path: str | Path) -> dict[str, JSONValue]: + """Read and validate an artifact without executing a model or verifier.""" + + source = Path(path) + if not source.is_file(): + raise ValueError(f"evidence source must be a regular file: {source}") + if source.stat().st_size > MAX_EVIDENCE_BYTES: + raise ValueError(f"evidence exceeds {MAX_EVIDENCE_BYTES} bytes") + try: + text = source.read_bytes().decode("utf-8") + value: object = json.loads(text, object_pairs_hook=_reject_duplicate_keys) + except (UnicodeError, json.JSONDecodeError, RecursionError) as error: + raise ValueError(f"{source} is not strict UTF-8 JSON: {error}") from error + return validate_claim_evidence_artifact(value) diff --git a/src/verifaxis/models/openai_compatible.py b/src/verifaxis/models/openai_compatible.py index 79bc586..cacaf98 100644 --- a/src/verifaxis/models/openai_compatible.py +++ b/src/verifaxis/models/openai_compatible.py @@ -2,13 +2,61 @@ from __future__ import annotations +import ipaddress import json import urllib.error +import urllib.parse import urllib.request from collections.abc import Mapping from ..types import Candidate, JSONValue +MAX_RESPONSE_BYTES = 2_000_000 +MAX_RESPONSE_JSON_DEPTH = 32 +MAX_RESPONSE_JSON_NODES = 20_000 + + +def _is_loopback(hostname: str) -> bool: + if hostname.lower() == "localhost": + return True + try: + return ipaddress.ip_address(hostname).is_loopback + except ValueError: + return False + + +def _is_sensitive_header(name: str) -> bool: + normalized = name.lower().replace("_", "-") + return any( + marker in normalized + for marker in ("authorization", "api-key", "apikey", "token", "secret", "cookie") + ) + + +def _reject_duplicate_keys(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON key {key!r}") + result[key] = value + return result + + +def _enforce_response_limits(value: object) -> None: + stack: list[tuple[object, int]] = [(value, 1)] + nodes = 0 + while stack: + current, depth = stack.pop() + nodes += 1 + if nodes > MAX_RESPONSE_JSON_NODES: + raise ValueError("response has too many JSON nodes") + if depth > MAX_RESPONSE_JSON_DEPTH: + raise ValueError("response JSON is too deeply nested") + if isinstance(current, dict): + stack.extend((item, depth + 1) for item in current.values()) + elif isinstance(current, list): + stack.extend((item, depth + 1) for item in current) + class OpenAICompatibleModel: """Call a configured OpenAI-compatible endpoint using only the stdlib. @@ -30,17 +78,34 @@ def __init__( ) -> None: if not model: raise ValueError("model must not be empty") - if not base_url.startswith(("http://", "https://")): + parsed_url = urllib.parse.urlsplit(base_url) + if parsed_url.scheme not in {"http", "https"} or parsed_url.hostname is None: raise ValueError("base_url must be an HTTP(S) URL") + try: + _ = parsed_url.port + except ValueError as error: + raise ValueError("base_url contains an invalid port") from error + if parsed_url.username is not None or parsed_url.password is not None: + raise ValueError("base_url must not contain embedded credentials") + if parsed_url.query or parsed_url.fragment: + raise ValueError("base_url must not contain a query or fragment") if timeout_seconds <= 0: raise ValueError("timeout_seconds must be positive") + headers = dict(extra_headers or {}) + has_credentials = bool(api_key) or any(_is_sensitive_header(name) for name in headers) + if ( + parsed_url.scheme == "http" + and has_credentials + and not _is_loopback(parsed_url.hostname) + ): + raise ValueError("credentials require HTTPS except for an explicit loopback endpoint") self.model = model self.base_url = base_url.rstrip("/") self.api_key = api_key self.timeout_seconds = timeout_seconds self.temperature = temperature self.max_output_tokens = max_output_tokens - self.extra_headers = dict(extra_headers or {}) + self.extra_headers = headers @property def model_id(self) -> str: @@ -87,14 +152,15 @@ def generate(self, *, task: str, state: Mapping[str, JSONValue]) -> Candidate: ) as response: raw = response.read(2_000_001) except urllib.error.URLError as error: - raise RuntimeError(f"OpenAI-compatible endpoint failed: {error.reason}") from error - if len(raw) > 2_000_000: + raise RuntimeError("OpenAI-compatible endpoint request failed") from error + if len(raw) > MAX_RESPONSE_BYTES: raise RuntimeError("OpenAI-compatible response exceeds 2 MB") try: - parsed = json.loads(raw) + parsed = json.loads(raw.decode("utf-8"), object_pairs_hook=_reject_duplicate_keys) + _enforce_response_limits(parsed) content = parsed["choices"][0]["message"]["content"] - except (KeyError, IndexError, TypeError, json.JSONDecodeError) as error: + except (KeyError, IndexError, TypeError, UnicodeError, ValueError, RecursionError) as error: raise RuntimeError("OpenAI-compatible endpoint returned an invalid response") from error if not isinstance(content, str): raise RuntimeError("OpenAI-compatible endpoint returned non-text content") diff --git a/tests/test_cli.py b/tests/test_cli.py index 80161d2..d793b1f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -16,6 +16,31 @@ def test_demo_is_offline_and_reports_smoke(capsys: pytest.CaptureFixture[str]) - assert output["status"] == "VERIFIED" +def test_demo_writes_and_validates_claim_evidence( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + evidence_path = tmp_path / "demo-evidence.json" + assert main(["demo", "--evidence-output", str(evidence_path)]) == 0 + output = json.loads(capsys.readouterr().out) + assert output["evidence_artifact"] == str(evidence_path) + artifact = json.loads(evidence_path.read_text(encoding="utf-8")) + assert artifact["decision"]["stopping_reason"] == "VERIFIED" + assert [ + packet["status"] for step in artifact["evidence_chain"] for packet in step["packets"] + ] == ["FAIL", "PASS"] + + assert main(["verify-evidence", str(evidence_path)]) == 0 + validation = json.loads(capsys.readouterr().out) + assert validation["status"] == "EVIDENCE ARTIFACT VERIFIED" + assert validation["evidence_packets"] == 2 + + artifact["claim"]["candidate"] = "tampered" + evidence_path.write_text(json.dumps(artifact), encoding="utf-8") + with pytest.raises(SystemExit) as error: + main(["verify-evidence", str(evidence_path)]) + assert error.value.code == 2 + + def test_run_json_valid_yaml(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: config = tmp_path / "task.yaml" destination = tmp_path / "trace.json" diff --git a/tests/test_evidence.py b/tests/test_evidence.py new file mode 100644 index 0000000..c0cd5ea --- /dev/null +++ b/tests/test_evidence.py @@ -0,0 +1,164 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from verifaxis import ( + build_claim_evidence_artifact, + validate_claim_evidence_artifact, + verify, + write_claim_evidence_artifact, +) +from verifaxis.evidence import ( + MAX_EVIDENCE_BYTES, + MAX_EVIDENCE_PACKETS, + load_and_validate_claim_evidence_artifact, +) +from verifaxis.models import ReplayModel +from verifaxis.types import VerificationResult, content_digest +from verifaxis.verifiers import SafeMathVerifier + + +def _result() -> VerificationResult: + return verify( + "What is 197 * 83?", + ReplayModel(), + [SafeMathVerifier()], + max_iterations=4, + ) + + +def _artifact() -> dict[str, object]: + return build_claim_evidence_artifact(_result(), producer_version="0.2.0") + + +def _rehash(artifact: dict[str, object]) -> None: + unsigned = dict(artifact) + del unsigned["artifact_hash"] + artifact["artifact_hash"] = content_digest(unsigned) # type: ignore[arg-type] + + +def test_claim_evidence_artifact_exposes_recurrence_and_stopping_reason() -> None: + artifact = _artifact() + assert artifact["producer"] == { + "repository": "aliengineering-byte/verifaxis", + "version": "0.2.0", + "capability": "verifier-conditioned-claim-decision", + "documentation": "https://github.com/aliengineering-byte/verifaxis#before-and-after", + } + assert artifact["claim"]["candidate"] == "16351" # type: ignore[index] + assert artifact["decision"]["stopping_reason"] == "VERIFIED" # type: ignore[index] + chain = artifact["evidence_chain"] + assert isinstance(chain, list) + assert [step["packets"][0]["status"] for step in chain] == ["FAIL", "PASS"] + validation = validate_claim_evidence_artifact(artifact) + assert validation["status"] == "EVIDENCE ARTIFACT VERIFIED" + assert validation["evidence_packets"] == 2 + + +def test_outer_artifact_hash_detects_tampering() -> None: + artifact = _artifact() + decision = artifact["decision"] + assert isinstance(decision, dict) + decision["status"] = "UNVERIFIABLE" + with pytest.raises(ValueError, match="artifact hash mismatch"): + validate_claim_evidence_artifact(artifact) + + +def test_packet_hash_is_checked_even_if_outer_hash_is_recomputed() -> None: + artifact = _artifact() + trace = artifact["trace"] + assert isinstance(trace, dict) + steps = trace["steps"] + assert isinstance(steps, list) + packet = steps[0]["evidence"][0] + packet["checked_claim"] = "tampered claim" + _rehash(artifact) + with pytest.raises(ValueError, match="evidence packet hash mismatch"): + validate_claim_evidence_artifact(artifact) + + +def test_claim_and_decision_are_derived_from_trace_even_if_outer_hash_is_recomputed() -> None: + artifact = _artifact() + claim = artifact["claim"] + assert isinstance(claim, dict) + claim["candidate"] = "tampered" + _rehash(artifact) + with pytest.raises(ValueError, match="claim does not match"): + validate_claim_evidence_artifact(artifact) + + artifact = _artifact() + decision = artifact["decision"] + assert isinstance(decision, dict) + decision["verified"] = False + _rehash(artifact) + with pytest.raises(ValueError, match="verified flag"): + validate_claim_evidence_artifact(artifact) + + +def test_trace_chain_is_derived_even_if_outer_hash_is_recomputed() -> None: + artifact = _artifact() + chain = artifact["evidence_chain"] + assert isinstance(chain, list) + chain[0]["packets"] = [] + _rehash(artifact) + with pytest.raises(ValueError, match="evidence_chain does not match"): + validate_claim_evidence_artifact(artifact) + + +def test_evidence_rejects_excessive_depth_and_packet_count() -> None: + nested: object = None + for _ in range(33): + nested = [nested] + with pytest.raises(ValueError, match="JSON depth"): + validate_claim_evidence_artifact({"nested": nested}) + + artifact = _artifact() + trace = artifact["trace"] + assert isinstance(trace, dict) + steps = trace["steps"] + assert isinstance(steps, list) + packets = steps[0]["evidence"] + assert isinstance(packets, list) + packets[:] = [packets[0]] * (MAX_EVIDENCE_PACKETS + 1) + _rehash(artifact) + with pytest.raises(ValueError, match="evidence exceeds 1024 packets"): + validate_claim_evidence_artifact(artifact) + + +def test_evidence_file_size_boundary_duplicate_keys_and_malformed_unicode( + tmp_path: Path, +) -> None: + target = tmp_path / "untrusted-evidence.json" + raw = json.dumps(_artifact(), separators=(",", ":")).encode("utf-8") + target.write_bytes(raw + b" " * (MAX_EVIDENCE_BYTES - len(raw))) + assert load_and_validate_claim_evidence_artifact(target)["status"] == ( + "EVIDENCE ARTIFACT VERIFIED" + ) + target.write_bytes(raw + b" " * (MAX_EVIDENCE_BYTES - len(raw) + 1)) + with pytest.raises(ValueError, match="exceeds 1048576 bytes"): + load_and_validate_claim_evidence_artifact(target) + + target.write_text('{"schema_version":"1.0","schema_version":"1.0"}', encoding="utf-8") + with pytest.raises(ValueError, match="duplicate JSON key"): + load_and_validate_claim_evidence_artifact(target) + target.write_bytes(b"\xff") + with pytest.raises(ValueError, match="strict UTF-8 JSON"): + load_and_validate_claim_evidence_artifact(target) + + +def test_evidence_writer_accepts_identical_and_refuses_different_existing_file( + tmp_path: Path, +) -> None: + result = _result() + target = tmp_path / "evidence.json" + assert write_claim_evidence_artifact(result, target, producer_version="0.2.0") == target + original = target.read_bytes() + assert write_claim_evidence_artifact(result, target, producer_version="0.2.0") == target + assert target.read_bytes() == original + + target.write_text(json.dumps({"user": "content"}), encoding="utf-8") + with pytest.raises(FileExistsError, match="refusing to overwrite"): + write_claim_evidence_artifact(result, target, producer_version="0.2.0") diff --git a/tests/test_models.py b/tests/test_models.py index 4c58c7b..8933422 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -2,17 +2,21 @@ import io import json +import urllib.error from collections.abc import Mapping from unittest.mock import patch +import pytest + from verifaxis import Candidate, EvidenceStatus, JSONValue from verifaxis.models import OpenAICompatibleModel, ReplayModel from verifaxis.verifiers import SafeMathVerifier class FakeResponse: - def __init__(self, value: object) -> None: - self.buffer = io.BytesIO(json.dumps(value).encode()) + def __init__(self, value: object | bytes) -> None: + raw = value if isinstance(value, bytes) else json.dumps(value).encode() + self.buffer = io.BytesIO(raw) def __enter__(self) -> FakeResponse: return self @@ -62,6 +66,59 @@ def test_openai_compatible_adapter_parses_text_and_usage() -> None: assert request.full_url == "http://localhost:1234/v1/chat/completions" +def test_openai_adapter_rejects_remote_plain_http_credentials() -> None: + with pytest.raises(ValueError, match="credentials require HTTPS"): + OpenAICompatibleModel( + model="remote", + base_url="http://example.com/v1", + api_key="not-logged", + ) + with pytest.raises(ValueError, match="credentials require HTTPS"): + OpenAICompatibleModel( + model="remote", + base_url="http://example.com/v1", + extra_headers={"X-API-Key": "not-logged"}, + ) + + +def test_openai_adapter_allows_loopback_credentials_without_logging_them() -> None: + response = FakeResponse({"choices": [{"message": {"content": "ok"}}]}) + model = OpenAICompatibleModel( + model="local", + base_url="http://127.0.0.1:1234/v1", + api_key="not-logged", + ) + with patch("urllib.request.urlopen", return_value=response) as urlopen: + assert model.generate(task="answer", state={}).content == "ok" + request = urlopen.call_args.args[0] + assert request.get_header("Authorization") == "Bearer not-logged" + + remote = OpenAICompatibleModel( + model="remote", + base_url="https://example.com/v1", + api_key="not-logged", + ) + with ( + patch( + "urllib.request.urlopen", + side_effect=urllib.error.URLError("Bearer not-logged"), + ), + pytest.raises(RuntimeError, match="request failed") as error, + ): + remote.generate(task="answer", state={}) + assert "not-logged" not in str(error.value) + + +def test_openai_adapter_rejects_duplicate_response_keys() -> None: + response = FakeResponse(b'{"choices":[{"message":{"content":"ok"}}],"choices":[]}') + model = OpenAICompatibleModel(model="local", base_url="http://localhost:1234/v1") + with ( + patch("urllib.request.urlopen", return_value=response), + pytest.raises(RuntimeError, match="invalid response"), + ): + model.generate(task="answer", state={}) + + def test_openai_adapter_satisfies_generate_shape() -> None: model = ReplayModel() state: Mapping[str, JSONValue] = {"evidence": []} diff --git a/tests/test_types.py b/tests/test_types.py index 33919fb..837a881 100644 --- a/tests/test_types.py +++ b/tests/test_types.py @@ -1,6 +1,8 @@ from __future__ import annotations import json +import tomllib +from pathlib import Path import pytest @@ -10,9 +12,17 @@ EvidenceStatus, IndependenceClassification, TerminationReason, + __version__, ) +def test_runtime_version_matches_project_metadata() -> None: + project = tomllib.loads( + (Path(__file__).parents[1] / "pyproject.toml").read_text(encoding="utf-8") + ) + assert __version__ == project["project"]["version"] + + def packet(**changes: object) -> EvidencePacket: values = { "verifier_type": "test",