Skip to content

Merge the verification observer (verus#2965), and add coverage and proof-state observers - #52

Open
kiranandcode wants to merge 7 commits into
mainfrom
kg/verification-observer
Open

kiranandcode wants to merge 7 commits into
mainfrom
kg/verification-observer

Conversation

@kiranandcode

@kiranandcode kiranandcode commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Brings upstream's verification-observer interface (verus-lang#2965, still open upstream) into the fork, and builds two observers on it.

1. The observer interface (cherry-pick of 9bccfb4, author kept). VirObserver, AirObserver and QueryResultObserver are callbacks fired during VIR→AIR lowering, AIR lowering and after each query. A consumer selects one with -V observers=<name>. The conflicts with the fork are in var_to_const, smt_verify, context.rs, config.rs, verifier.rs and the test harness. Both sides were kept in each; the commit message lists how. Two changes to the PR's API:

  • The handles are Arc<Mutex<dyn Trait>> and the traits require Send, instead of Rc<RefCell<..>>. Resident sessions keep air::Context in a Mutex served from other threads, so an Rc in Context does not compile here.
  • VersionOrigin::{BranchMerge, BreakMerge} and VirObserver::on_break_merge are removed. They never fired. A merge takes the highest version a branch reached, and that branch already created it, so a merge makes no new version. on_break_merge had no caller.

2. A fix to the PR for cvc5. The Invalid callback's eval_bool_expr sends z3's (eval ...). cvc5 does not accept that, and the run dies with "Z3 reader thread failure". Under cvc5 it now asks (get-value (e)). eval_expr itself is unchanged; only the evaluator treats a multi-line answer as "not a boolean". Upstream's tests run z3 in internal test mode, so they never hit this. The PR's own test observer crashes on any failing query in the fork without this fix.

3. One version hook. The fork's own version recording in var_to_const (read by provenance, e-graph readings and speculation) now goes through the PR's notify_version_created, rather than a second path beside it. Behaviour is unchanged. Other fork lowering hooks were not ported: the goal scopes, and source names recorded at encode time. They feed the fork's own solver-side tools, and routing them through an external observer would add a layer without removing any fork code.

4. -V observers=coverage and -V observers=proof-state. One object, rust_verify/src/observers.rs, serves both names; they can be given together. Any other name, except the test suites' own observers used alone, is an argument error. The verifier takes each query's records out after the query and files them under the function, as it does for the other per-query solver reports. They are published in func-details, and so in --output-json and --report-json:

  • obligations (coverage): per query, every obligation Verus asserted, with its message, span and one of four statuses:

    • proved
    • failed
    • unknown: the solver gave up on the query
    • unchecked: the query stopped first, e.g. --multiple-errors 0 or the error budget ran out

    With -V axiom-usage-info, a proved query also lists its unsat-core axioms. This needed a fix to the fork: cvc5 prints the unsat core one name per line, and the one-line parser aborted the run under -V axiom-usage-info with or without an observer. This is coverage at the obligation level, the counterpart to verus-reach (Static reachability of verified code (verus --reach, tools/verus-reach) #12/verus-reach: static reachability coverage of verified code (HTML report, explicit roots, --connected, thresholds) #39), which covers which functions are used.

  • failing_asserts (proof-state): at each failing obligation, while cvc5's counterexample is live:

    • which conjuncts of the failing assertion are false, in source spelling
    • the counterexample's value for each variable those conjuncts read (the encoding's type ids and decorations are left out)

    For example, (i < 40) is false with i = 40. The split looks through two things:

    • assert(a && b) asserts a Verus temporary (tmp%1), which is read as its definition.
    • A failing call's precondition req%f(args) is read as f's requires at the call's arguments, taken from the req% axiom.

    A temporary or precondition whose definition is not available stays whole and is marked unexpanded.

Loop invariants and bit-vector asserts have no assert id, and a function's postconditions share one. So a failure is placed by its error's span, not on the first assertion with a matching id. A failure without a model has no span; the verifier passes its id, which may be none (a bit-vector assert). It is placed only when that id picks out exactly one undecided assertion.

Companion MCP PR: BasisResearch/verus-tools-mcp#60 (obligation_coverage, failing_assert_state).

Test plan

  • vargo build --release
  • vargo test --release -p rust_verify_test --test observer: the PR's 29 tests
  • CI: full rust_verify_test suite
  • vargo test --release -p rust_verify_test --test obligation_observer: 10 new tests, run under cvc5
    • statuses, an id-less invariant failure, --multiple-errors 0, false conjunct and values
    • assert(a && b) split, callee precondition split, no encoding constants among values
    • a failing bit-vector assert is failed
    • used_axioms under cvc5
    • unknown observer names rejected
  • cargo test --release -p rust_verify --lib observers (8), cargo test --release -p air --lib (218, including the multi-line unsat core parser and "merges announce no versions")
  • report_json (11), difficulty (3), resident (64) suites
  • cargo fmt -- --check, cargo clippy --all-targets -- -D warnings in source/ (as CI runs it)

🤖 Generated with Claude Code

matthewbdwyer and others added 7 commits September 26, 2026 21:08
…wering

Three observer traits each with default (no-op) methods let external
tooling observe the verification pipeline without modifying Verus:

- VirObserver (vir/src/vir_observer.rs): VIR->AIR lowering events — havoc,
  assign, branch/break merges, for-loops, quantifier and choose binders,
  reveal strings, function boundaries — and an optional assert-id supplier.
- AirObserver (air/src/air_observer.rs): AIR passes — the lowered query
  (after variable versioning, before block-to-assert), version creation with
  its origin, lambda/choose/axiom declarations.
- QueryResultObserver (air/src/query_result_observer.rs): the result of each
  solver call. On Invalid: the parsed counterexample model and a live boolean
  evaluator over the still-open solver model, plus the failing assert id. On
  Valid: the assert ids the query discharged and the solver's usage info
  (unsat core, when enabled). On Timeout: the assert id.

Each trait has a single optional registration slot; when nothing is
registered every hook is a no-op. The Invalid path is split so the
solver model stays open through the callback; the label-disable step
still runs afterwards with or without an observer.

block_to_assert::lower_query is made pub so a consumer can apply the
same final lowering the solver sees.

Test infrastructure (test-only, behind --observers=): recording
observers that emit an OBSERVER: note; a shared corpus of small programs
each exercising a verification-condition (VC) shape
(rust_verify_test/tests/vc_shapes); and an end-to-end suite that asserts
expected values for callback payloads.

In addition to coverage of VC shape, the test infrastructure varies
loop-isolation settings, checks payload delivery on both valid and
invalid verification results, and checks behavior neutrality by
verifying every corpus program with and without observers by requiring
identical outcomes.

Assisted-by: kiro_cli:claude-fable-5

[Cherry-picked from verus-lang#2965 (9bccfb4) onto the
BasisResearch fork. Conflicts resolved against the fork's own changes:
var_to_const::lower_query keeps the fork's version recording and goal
scopes and takes the observer as a third argument; the failing-assertion
label is still disabled after the Invalid callback, with the fork's
three-argument log_assert; config.rs, the test harness and verifier.rs
keep both sides' options and bucket teardown.

One change to the PR's interface: observer handles are
Arc<Mutex<dyn Trait>> (the AirObserverHandle, QueryResultObserverHandle
and VirObserverHandle aliases) and the traits require Send, instead of
Rc<RefCell<..>>. The fork's resident sessions keep air::Context values in
a Mutex served from other threads, so a Context holding an Rc does not
compile here.]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`QueryResultObserver::Invalid` hands observers `eval_bool_expr`, which
sends z3's `(eval ...)` extension. cvc5 has no `eval`: it answers with an
error the reader does not expect, and the run dies with "Got too many
empty lines" / "Z3 reader thread failure". The PR's own `test` observer
does this on any failing query here, since the fork runs cvc5 outside
internal test mode; upstream's tests pass because they run z3.

Under cvc5 `evaluate_bool` now asks `(get-value (e))` and reads the
`((e true))` / `((e false))` answer; z3 keeps `eval`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`lower_stmt` had two paths for a new variable version: the fork's
`record_versions` bookkeeping (read back by provenance, e-graph readings
and speculation) and the PR's `notify_version_created`, which announces
it to an `AirObserver`. They now share one: `notify_version_created`
records the version when asked and then notifies the observer, for
havoc, assign and merge versions alike. The declared version 0 is still
recorded where it is declared, which no statement announces.

A merge's version is always the highest a branch reached, so it was
already in the map; recording it again changes nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two observers built on the observation interface, served by one object
(rust_verify/src/observers.rs), so they may be named together.

coverage: per query, every obligation Verus asserted (an assert, a
postcondition, a loop invariant, a call's precondition, an overflow
check) with its message, span and status: proved, failed, unknown (the
solver gave up on the query) or unchecked (the query stopped before
deciding it). With -V axiom-usage-info a proved query also lists its
unsat-core axioms. Obligation-level coverage, beside verus-reach's
function-level reachability.

proof-state: at each failing obligation, while the counterexample is
live, the failing assertion's conjuncts with their value in it (false
first) and the counterexample value of each constant it reads, rendered
as source with the names the encoders recorded.

The observer sees queries but not functions, so the verifier drains it
after each query, files the records under the function like the other
per-query solver reports, and renders them at the end of the crate. They
appear in func-details as `obligations` and `failing_asserts`, and so in
--output-json and --report-json.

Assertion ids are not unique (loop invariants have none; a function's
postconditions share one), so a failure is placed by its error's span.
A failure that reaches the verifier without the Invalid callback (no
model) is still recorded from the ids the verifier saw fail, and when the
--multiple-errors budget runs out the obligations after the last failure
are unchecked rather than proved.

Also formats the observer handle types the previous commits introduced.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s, reject unknown names

Review fixes for -V observers=coverage / proof-state and the interface:

- proof-state reads an asserted Verus temporary (`assert(a && b)` asserts
  `tmp%1`, defined by `assume (= tmp%1 e)`) as its definition, and a
  failing call's precondition `req%f(args)` as f's requires at those
  arguments, from the `req%` axioms `on_axiom_decl` sees; the axiom
  location labels are looked through. A temporary or precondition whose
  definition is not available is kept whole and marked `unexpanded`.
  Values are taken from what the split conjuncts read, without the
  encoding's own constants (Type, Dcr, Fuel sorts: `INT`, `$`, `TYPE%..`).
- A failure without a model is placed by its id only when that id names
  exactly one undecided assertion (postconditions share ids). Such
  failures are passed as Option<AssertId>, so a bit-vector assert, which
  has no id, is `failed` rather than `unchecked`. Failures seen through
  the Invalid callback are no longer passed a second time.
- get-unsat-core: cvc5 prints the core one name per line; the reply is
  parsed across lines (and |quoted| names), so -V axiom-usage-info no
  longer aborts under cvc5 and coverage lists used_axioms.
- -V observers= names are checked in parse_args: unknown or empty names,
  and a test observer combined with another, are errors.
- VersionOrigin::{BranchMerge, BreakMerge}, VirObserver::on_break_merge
  and the correlator's merge records are removed: a merge takes the
  highest version a branch reached, which that branch already created,
  so they never fired.
- eval_expr keeps its single-line contract; the multi-line fallback is
  local to evaluate_bool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, detach from resident contexts

- When the --multiple-errors budget runs out, AIR's only_check_earlier
  disables every label after the first failed one, so the final Valid
  speaks only for the assertions before it. take_query cut at the last
  failure, and so reported obligations between two failures as proved
  though no round proved them. It now cuts at the first failure.
- proof-state evaluates at most MAX_EVALUATIONS (4 * MAX_CONJUNCTS)
  conjuncts in the counterexample, one solver round trip each; the rest
  are counted in conjuncts_omitted.
- A context handed to a resident session no longer carries the observers:
  its queries would run the callbacks and evaluations with nothing to
  drain them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… failures only when unambiguous, document conditional proved

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants