Merge the verification observer (verus#2965), and add coverage and proof-state observers - #52
Open
kiranandcode wants to merge 7 commits into
Open
kiranandcode wants to merge 7 commits into
kiranandcode wants to merge 7 commits into
Conversation
…wering Three observer traits each with default (no-op) methods let external tooling observe the verification pipeline without modifying Verus: - VirObserver (vir/src/vir_observer.rs): VIR->AIR lowering events — havoc, assign, branch/break merges, for-loops, quantifier and choose binders, reveal strings, function boundaries — and an optional assert-id supplier. - AirObserver (air/src/air_observer.rs): AIR passes — the lowered query (after variable versioning, before block-to-assert), version creation with its origin, lambda/choose/axiom declarations. - QueryResultObserver (air/src/query_result_observer.rs): the result of each solver call. On Invalid: the parsed counterexample model and a live boolean evaluator over the still-open solver model, plus the failing assert id. On Valid: the assert ids the query discharged and the solver's usage info (unsat core, when enabled). On Timeout: the assert id. Each trait has a single optional registration slot; when nothing is registered every hook is a no-op. The Invalid path is split so the solver model stays open through the callback; the label-disable step still runs afterwards with or without an observer. block_to_assert::lower_query is made pub so a consumer can apply the same final lowering the solver sees. Test infrastructure (test-only, behind --observers=): recording observers that emit an OBSERVER: note; a shared corpus of small programs each exercising a verification-condition (VC) shape (rust_verify_test/tests/vc_shapes); and an end-to-end suite that asserts expected values for callback payloads. In addition to coverage of VC shape, the test infrastructure varies loop-isolation settings, checks payload delivery on both valid and invalid verification results, and checks behavior neutrality by verifying every corpus program with and without observers by requiring identical outcomes. Assisted-by: kiro_cli:claude-fable-5 [Cherry-picked from verus-lang#2965 (9bccfb4) onto the BasisResearch fork. Conflicts resolved against the fork's own changes: var_to_const::lower_query keeps the fork's version recording and goal scopes and takes the observer as a third argument; the failing-assertion label is still disabled after the Invalid callback, with the fork's three-argument log_assert; config.rs, the test harness and verifier.rs keep both sides' options and bucket teardown. One change to the PR's interface: observer handles are Arc<Mutex<dyn Trait>> (the AirObserverHandle, QueryResultObserverHandle and VirObserverHandle aliases) and the traits require Send, instead of Rc<RefCell<..>>. The fork's resident sessions keep air::Context values in a Mutex served from other threads, so a Context holding an Rc does not compile here.] Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`QueryResultObserver::Invalid` hands observers `eval_bool_expr`, which sends z3's `(eval ...)` extension. cvc5 has no `eval`: it answers with an error the reader does not expect, and the run dies with "Got too many empty lines" / "Z3 reader thread failure". The PR's own `test` observer does this on any failing query here, since the fork runs cvc5 outside internal test mode; upstream's tests pass because they run z3. Under cvc5 `evaluate_bool` now asks `(get-value (e))` and reads the `((e true))` / `((e false))` answer; z3 keeps `eval`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`lower_stmt` had two paths for a new variable version: the fork's `record_versions` bookkeeping (read back by provenance, e-graph readings and speculation) and the PR's `notify_version_created`, which announces it to an `AirObserver`. They now share one: `notify_version_created` records the version when asked and then notifies the observer, for havoc, assign and merge versions alike. The declared version 0 is still recorded where it is declared, which no statement announces. A merge's version is always the highest a branch reached, so it was already in the map; recording it again changes nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two observers built on the observation interface, served by one object (rust_verify/src/observers.rs), so they may be named together. coverage: per query, every obligation Verus asserted (an assert, a postcondition, a loop invariant, a call's precondition, an overflow check) with its message, span and status: proved, failed, unknown (the solver gave up on the query) or unchecked (the query stopped before deciding it). With -V axiom-usage-info a proved query also lists its unsat-core axioms. Obligation-level coverage, beside verus-reach's function-level reachability. proof-state: at each failing obligation, while the counterexample is live, the failing assertion's conjuncts with their value in it (false first) and the counterexample value of each constant it reads, rendered as source with the names the encoders recorded. The observer sees queries but not functions, so the verifier drains it after each query, files the records under the function like the other per-query solver reports, and renders them at the end of the crate. They appear in func-details as `obligations` and `failing_asserts`, and so in --output-json and --report-json. Assertion ids are not unique (loop invariants have none; a function's postconditions share one), so a failure is placed by its error's span. A failure that reaches the verifier without the Invalid callback (no model) is still recorded from the ids the verifier saw fail, and when the --multiple-errors budget runs out the obligations after the last failure are unchecked rather than proved. Also formats the observer handle types the previous commits introduced. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s, reject unknown names
Review fixes for -V observers=coverage / proof-state and the interface:
- proof-state reads an asserted Verus temporary (`assert(a && b)` asserts
`tmp%1`, defined by `assume (= tmp%1 e)`) as its definition, and a
failing call's precondition `req%f(args)` as f's requires at those
arguments, from the `req%` axioms `on_axiom_decl` sees; the axiom
location labels are looked through. A temporary or precondition whose
definition is not available is kept whole and marked `unexpanded`.
Values are taken from what the split conjuncts read, without the
encoding's own constants (Type, Dcr, Fuel sorts: `INT`, `$`, `TYPE%..`).
- A failure without a model is placed by its id only when that id names
exactly one undecided assertion (postconditions share ids). Such
failures are passed as Option<AssertId>, so a bit-vector assert, which
has no id, is `failed` rather than `unchecked`. Failures seen through
the Invalid callback are no longer passed a second time.
- get-unsat-core: cvc5 prints the core one name per line; the reply is
parsed across lines (and |quoted| names), so -V axiom-usage-info no
longer aborts under cvc5 and coverage lists used_axioms.
- -V observers= names are checked in parse_args: unknown or empty names,
and a test observer combined with another, are errors.
- VersionOrigin::{BranchMerge, BreakMerge}, VirObserver::on_break_merge
and the correlator's merge records are removed: a merge takes the
highest version a branch reached, which that branch already created,
so they never fired.
- eval_expr keeps its single-line contract; the multi-line fallback is
local to evaluate_bool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, detach from resident contexts - When the --multiple-errors budget runs out, AIR's only_check_earlier disables every label after the first failed one, so the final Valid speaks only for the assertions before it. take_query cut at the last failure, and so reported obligations between two failures as proved though no round proved them. It now cuts at the first failure. - proof-state evaluates at most MAX_EVALUATIONS (4 * MAX_CONJUNCTS) conjuncts in the counterexample, one solver round trip each; the rest are counted in conjuncts_omitted. - A context handed to a resident session no longer carries the observers: its queries would run the callbacks and evaluations with nothing to drain them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… failures only when unambiguous, document conditional proved Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings upstream's verification-observer interface (verus-lang#2965, still open upstream) into the fork, and builds two observers on it.
1. The observer interface (cherry-pick of 9bccfb4, author kept).
VirObserver,AirObserverandQueryResultObserverare callbacks fired during VIR→AIR lowering, AIR lowering and after each query. A consumer selects one with-V observers=<name>. The conflicts with the fork are invar_to_const,smt_verify,context.rs,config.rs,verifier.rsand the test harness. Both sides were kept in each; the commit message lists how. Two changes to the PR's API:Arc<Mutex<dyn Trait>>and the traits requireSend, instead ofRc<RefCell<..>>. Resident sessions keepair::Contextin aMutexserved from other threads, so anRcinContextdoes not compile here.VersionOrigin::{BranchMerge, BreakMerge}andVirObserver::on_break_mergeare removed. They never fired. A merge takes the highest version a branch reached, and that branch already created it, so a merge makes no new version.on_break_mergehad no caller.2. A fix to the PR for cvc5. The Invalid callback's
eval_bool_exprsends z3's(eval ...). cvc5 does not accept that, and the run dies with "Z3 reader thread failure". Under cvc5 it now asks(get-value (e)).eval_expritself is unchanged; only the evaluator treats a multi-line answer as "not a boolean". Upstream's tests run z3 in internal test mode, so they never hit this. The PR's owntestobserver crashes on any failing query in the fork without this fix.3. One version hook. The fork's own version recording in
var_to_const(read by provenance, e-graph readings and speculation) now goes through the PR'snotify_version_created, rather than a second path beside it. Behaviour is unchanged. Other fork lowering hooks were not ported: the goal scopes, and source names recorded at encode time. They feed the fork's own solver-side tools, and routing them through an external observer would add a layer without removing any fork code.4.
-V observers=coverageand-V observers=proof-state. One object,rust_verify/src/observers.rs, serves both names; they can be given together. Any other name, except the test suites' own observers used alone, is an argument error. The verifier takes each query's records out after the query and files them under the function, as it does for the other per-query solver reports. They are published infunc-details, and so in--output-jsonand--report-json:obligations(coverage): per query, every obligation Verus asserted, with its message, span and one of four statuses:provedfailedunknown: the solver gave up on the queryunchecked: the query stopped first, e.g.--multiple-errors 0or the error budget ran outWith
-V axiom-usage-info, a proved query also lists its unsat-core axioms. This needed a fix to the fork: cvc5 prints the unsat core one name per line, and the one-line parser aborted the run under-V axiom-usage-infowith or without an observer. This is coverage at the obligation level, the counterpart toverus-reach(Static reachability of verified code (verus --reach, tools/verus-reach) #12/verus-reach: static reachability coverage of verified code (HTML report, explicit roots, --connected, thresholds) #39), which covers which functions are used.failing_asserts(proof-state): at each failing obligation, while cvc5's counterexample is live:For example,
(i < 40)is false withi = 40. The split looks through two things:assert(a && b)asserts a Verus temporary (tmp%1), which is read as its definition.req%f(args)is read asf's requires at the call's arguments, taken from thereq%axiom.A temporary or precondition whose definition is not available stays whole and is marked
unexpanded.Loop invariants and bit-vector asserts have no assert id, and a function's postconditions share one. So a failure is placed by its error's span, not on the first assertion with a matching id. A failure without a model has no span; the verifier passes its id, which may be none (a bit-vector assert). It is placed only when that id picks out exactly one undecided assertion.
Companion MCP PR: BasisResearch/verus-tools-mcp#60 (
obligation_coverage,failing_assert_state).Test plan
vargo build --releasevargo test --release -p rust_verify_test --test observer: the PR's 29 testsrust_verify_testsuitevargo test --release -p rust_verify_test --test obligation_observer: 10 new tests, run under cvc5--multiple-errors 0, false conjunct and valuesassert(a && b)split, callee precondition split, no encoding constants among valuesfailedused_axiomsunder cvc5cargo test --release -p rust_verify --lib observers(8),cargo test --release -p air --lib(218, including the multi-line unsat core parser and "merges announce no versions")report_json(11),difficulty(3),resident(64) suitescargo fmt -- --check,cargo clippy --all-targets -- -D warningsinsource/(as CI runs it)🤖 Generated with Claude Code