Skip to content

Reject stale-view writes before local commit - #8242

Merged
Amaury Chamayou (achamayou) merged 2 commits into
mainfrom
achamayou-reject-stale-view-writes
Sep 2, 2026
Merged

Reject stale-view writes before local commit#8242
Amaury Chamayou (achamayou) merged 2 commits into
mainfrom
achamayou-reject-stale-view-writes

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Aug 30, 2026

Copy link
Copy Markdown
Member

Why this is necessary

CommittableTx::commit() documents that "transactions that fail are rolled back, no matter the reason". That was not true for a view change.

A transaction reads state (fixing its commit view), then applies its writes and allocates a version, and only later reaches Store::commit() where its view is checked against the store's. If an election lands in that window, the transaction is refused with FAIL_NO_REPLICATE - but its write is already applied locally, with no corresponding entry ever reaching consensus. Worse, because last_replicated no longer matches the store's version, ordinary transactions committed afterwards can keep succeeding locally without replicating, until a further election restores agreement.

This was found by driving a real Store, MerkleTxHistory and aft::Aft together under pinned election interleavings. The harness that found it is not proposed for merge; the regression here reproduces the same failure with the existing stubs.

What changes

The transaction's captured view is validated atomically with the allocation of its version, under the same version_lock a rollback takes:

  • if the view moved first, no version is allocated and no map is touched - the transaction is refused before it can leave anything behind;
  • if allocation wins the race, the rollback necessarily observes the new version and truncates the writes.

FAIL_NO_REPLICATE therefore no longer implies a locally applied write.

Why this is minimal

Version allocation is the only point that is already atomic with rollback, so it is the only place the check can be made without introducing new lock ordering between the KV and consensus. The alternative - repairing the store after the fact - would have to roll back writes belonging to unrelated concurrent transactions.

Two consequential simplifications come with it:

  • next_version() returns the rollback epoch observed at allocation, which later fixes in this stack build on.
  • The caller-supplied version resolver parameter of commit() is removed. It had no callers in tree, and any future user of it would have silently bypassed this check.

Testing

kv_test gains "Stale-view writes are rejected before local application", which covers both an existing map and a dynamically created one, and asserts that replication continues normally afterwards without a healing election. It is single-threaded and deterministic.

Verified that the test fails without the fix (6 assertions, including the stale value and dynamic map surviving), and passes with it. Also exercised under ThreadSanitizer.

Labelled run-long-test.


Review stack

These five PRs come from one investigation and are stacked; review and merge in order.

PR Change
#8242 Reject stale-view writes before local commit
#8243 Order chunk metadata and snapshot scheduling with rollback
#8244 Stop a rollback moving chunk metadata forward
#8245 Guard rollback-sensitive transaction flags
#8246 Guard reserved signature side effects

All were found by an interleaving-exploration harness built over the real KV, consensus and history stack (draft #8238). The harness itself is deliberately not included here; these PRs carry only the fixes and the single-threaded regression tests that pin them.

Copilot AI lite review requested due to automatic review settings August 30, 2026 18:48
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner August 30, 2026 18:48
@achamayou Amaury Chamayou (achamayou) added the run-long-test Run Long Test job label Aug 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a KV-store consistency bug where transactions could apply writes locally and then fail to replicate after a view/term change, leaving unreplicated state behind. It moves the view validation to be atomic with version allocation (under version_lock), ensuring stale-term transactions are rejected before they can modify any maps.

Changes:

  • Make version allocation (Store::next_version) term-aware and able to refuse stale-view commits before any map commit occurs.
  • Remove the unused caller-supplied version-resolver parameter from CommittableTx::commit() and update call sites accordingly.
  • Add a regression test to ensure stale-view transactions do not leave local state behind (including for dynamically created maps), and bump version/changelog.

Custom instructions used:

  • .github/skills/testing/SKILL.md
  • .github/instructions/changelog.instructions.md

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/node/snapshotter.h Update CommittableTx::commit() call to new signature (removed version resolver arg).
src/node/rpc/frontend.h Update CommittableTx::commit() call to new signature.
src/kv/test/kv_test.cpp Add regression test covering stale-view rejection before local application (existing + dynamic maps).
src/kv/store.h Enforce term check during version allocation; simplify view-check in Store::commit().
src/kv/kv_types.h Update store interface to return optional version resolution including rollback epoch/count.
src/kv/committable_tx.h Plumb new version resolution API; map stale-term refusal to FAIL_NO_REPLICATE.
src/kv/apply_changes.h Defer map commits until version resolution succeeds; allow resolver to fail (std::optional).
python/pyproject.toml Bump project version to 7.0.14 to match changelog.
CHANGELOG.md Add 7.0.14 entries documenting the fix and API change (with PR reference).
Suppressed comments (1)

src/kv/test/kv_test.cpp:3029

  • ReadableMapHandle::get() returns std::optional<std::string>, so comparing it directly to a string literal is not a valid comparison on older language standards and will fail to compile on most toolchains. Compare against an optional (or unwrap the optional explicitly).
    auto tx = store.create_read_only_tx();
    CHECK(tx.ro(map)->get(key) == "fresh");
  }

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/kv/committable_tx.h Outdated
Comment thread src/kv/test/kv_test.cpp
Comment thread CHANGELOG.md Outdated
@achamayou

Copy link
Copy Markdown
Member Author

CI note: Long Shuffled/LTS/Snmalloc failed in the shuffled full_test_suite, not in LTS. governance_history.test_ledger_is_readable timed out waiting 5 seconds for backup node 10 to expose its local ledger chunk. This is the known backup-synchronisation race tracked by #8248, not a failure in this PR's KV change. The other jobs, including ASAN, TSAN, and both long e2e variants, passed.

@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-reject-stale-view-writes branch 2 times, most recently from 59e4923 to 10ab672 Compare September 2, 2026 12:07
A transaction whose view changed while it was committing could apply its writes to the local store and only then be refused replication, leaving state that never reaches consensus - contradicting the documented contract that a failed transaction is rolled back.

Validate the view the transaction captured atomically with the allocation of its version, under the same lock a rollback takes, so it is refused before any map is modified. If allocation wins the race instead, the rollback observes the new version and truncates the writes.

The unused caller-supplied version resolver is removed: it had no callers and would have bypassed this check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 75d99c5d-6efa-4048-8032-8c78b97208d9
Co-authored-by: cjen1-msft <chrisjensen@microsoft.com>
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-reject-stale-view-writes branch from 10ab672 to 025936e Compare September 2, 2026 13:42
@achamayou
Amaury Chamayou (achamayou) merged commit ecda4a6 into main Sep 2, 2026
13 checks passed
@achamayou
Amaury Chamayou (achamayou) deleted the achamayou-reject-stale-view-writes branch September 2, 2026 20:24
Comment thread CHANGELOG.md

- CCF and C++ applications built against it now require C++23. The supported minimum Clang version remains 18.1.2. (#8234)
- `sandbox.sh` now derives node configuration defaults and CLI descriptions from the `cchost` configuration schema, rather than using defaults selected by the end-to-end test infrastructure. This changes the sandbox defaults for signature delay (100 ms -> 1000 ms), election timeout (4000 ms -> 5000 ms), ledger chunk size (5000000 bytes -> `5MB`, or 5242880 bytes), initial node and service certificate validity (90 days -> 1 day), and tick interval (1 ms -> 10 ms). Environment variables used by the test infrastructure no longer override sandbox defaults; for example, use the existing `--election-timeout-ms` option instead of `ELECTION_TIMEOUT_MS` (#8176).
- `ccf::kv::CommittableTx::commit()` no longer takes a caller-supplied version resolver. The parameter had no callers, and bypassed the view check above. Callers passing `nullptr` for it should remove the argument (#8242).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CommittableTx is not public API, so this doesn't need to be in the CHANGELOG.

Eddy Ashton (eddyashton) added a commit that referenced this pull request Sep 4, 2026
…pstream

#8242 ("Reject stale-view writes before local commit") landed on main
during this session's rebase, independently fixing the exact bug the
NOTE_REJECTED_COMMIT_STALL-tagged tests here were built to catch - all
of them now pass reliably (deterministic and 500-sample model-checked
runs alike), so the old "expected to fail until fixed" framing and tag
were stale.

- Removed the NOTE_REJECTED_COMMIT_STALL tag and its defining comment;
  the DOCTEST_CHECKs it marked are ordinary passing assertions now.
- Reworded the two affected test cases' comments to note they are
  regression tests for #8242, and to explain what they add beyond
  kv_test.cpp's own direct, single-threaded test of the same rejection
  (driving it through a real election instead, and cross-checking
  TxHistory and raft's own replication index).
- Removed the now-defunct middle nullptr (version_resolver) argument
  from one CommittableTx::commit() call, matching #8242's own signature
  change and migration note.

Comments deliberately avoid narrating what #8242 changed or how - that's
what git history is for, and it rots fast; only "this is a regression
test for #8242" is kept, since the tests would otherwise look like
ordinary, low-value assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-long-test Run Long Test job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants