feat: add devnet 5 support

[MegaRedHand]

🗒️ Description / Motivation

Adds support for the pq-devnet-5 spec, whose headline feature is the new two-tier signature aggregation scheme from leanSpec PR #717: per-attestation Type-1 multi-signatures that get merged into a single block-level Type-2 proof. Instead of a block carrying one aggregated proof per attestation plus a proposer signature, a SignedBlock now carries a single merged proof binding every signature it depends on.

Closes #285.

What Changed

Wire format / types (crates/common/types)

• SignedBlock.signature: BlockSignatures → SignedBlock.proof: ByteList512KiB, an SSZ-encoded TypeTwoMultiSignature envelope. Helpers merged_proof_bytes() / wrap_merged_proof() handle the 4-byte SSZ offset header.
• AggregatedSignatureProof is replaced by TypeOneMultiSignature (flat container per leanSpec PR #717). BlockSignatures and AttestationSignatures are gone.

Crypto (crates/common/crypto)

• New leanVM operations: merge_type_1s_into_type_2 (block building), verify_type_2_signature (block import), and split_type_2_by_message (recovering per-attestation Type-1 proofs from a merged proof).

Blockchain

• Block builder produces the merged Type-2 proof from the pooled Type-1 proofs; block import verifies it.
• New reaggregate.rs module ports leanSpec's SyncService._deconstruct_block_into_store: after importing a block, it SNARK-splits the merged proof back into per-attestation Type-1 proofs and folds them into the local aggregated-payload pool, so block-borne votes can be republished on gossip. Splits are bounded (only when in sync, skip already-justified targets, skip participant subsets, max MAX_REAGGREGATIONS_PER_BLOCK splits per block) since each split runs a fresh SNARK.
• process_block now reports whether the block was newly imported, so reaggregation only runs once per block.

Storage

• BlockSignatures table now stores the raw proof envelope; the AggregatedPayloads table stores TypeOneMultiSignatures. Genesis blocks simply have no proof entry (the placeholder empty_block_signatures is gone).

Dependencies / CI

• leanSpec pin bumped to 30ffb6c (2026-06-03), leanVM to e2592df, leansig pinned to its devnet4 branch.
• CI fixture generation runs with -n 1: the devnet5 prover peaks at ~12 GiB per proof, so parallel provers OOM the 16 GiB runner. The fixtures cache is now only saved when generation actually succeeds, preventing a cancelled run from poisoning the cache with an empty fixture set.
• Removed the curl+tar key-download workaround (the new pin is past leanSpec PR #745).

Correctness / Behavior Guarantees

• The on-chain block root is unchanged by proof serialization: consumers always hash the inner Block, never the SignedBlock envelope.
• Block import rejects blocks whose merged Type-2 proof does not verify against the block's attestations and proposer key.
• Reaggregation never floods gossip: it is skipped while syncing, and the per-block SNARK-split budget is capped.

Tests Added / Run

• Fork choice, signature, STF, and SSZ spec tests adapted to the devnet5 fixture format (fixtures regenerated from the new leanSpec pin).
• New unit tests for the proof envelope helpers and reaggregation bounds.
• make fmt / make lint clean (CI Lint is green). The CI Test job regenerates devnet5 fixtures (~2.5 h single-prover) before running the suite.

Related Issues / PRs

• Closes Add support for pq-devnet-5 #285
• Spec reference: leanSpec PR #717 (Type-1/Type-2 aggregation wire format)

✅ Verification Checklist

• Ran make fmt — clean
• Ran make lint (clippy with -D warnings) — clean
• Ran cargo test --workspace --release — CI Test job in progress (devnet5 fixture generation takes ~2.5 h)

[github-actions]

🤖 Kimi Code Review

This PR implements the devnet5 Type-1/Type-2 multi-signature scheme (leanSpec PR #717), replacing per-attestation signature lists with a single merged SNARK proof per block. The changes are substantial but well-structured.

Security & Correctness

Block Proof Envelope Validation (crates/common/types/src/block.rs:47-64)
The merged_proof_bytes() helper correctly validates the 4-byte SSZ offset header before slicing. This prevents out-of-bounds panics on malformed input, but consider explicitly logging the UnexpectedOffset case at the call site in verify_block_signatures to aid debugging invalid blocks from the network.

Reaggregation DoS Mitigation (crates/blockchain/src/reaggregate.rs:36)
The MAX_REAGGREGATIONS_PER_BLOCK = 4 cap is appropriate given the expensive SNARK splitting operation. Item 1 in the safety checklist is satisfied.

Validator Index Bounds (crates/blockchain/src/reaggregate.rs:98-108)
Good: Participant indices are checked against num_validators before array access in the reaggregation path. The early return on out-of-range prevents panics.

Slot Overflow Handling (crates/blockchain/src/reaggregate.rs:143-147)
The silent continue on slot.try_into() overflow is safe for consensus (the attestation is simply skipped), but consider logging this at debug! level to catch fixture/generation issues.

Performance

Memory Pressure in CI (.github/workflows/ci.yml:120-128)
The change from -n auto to -n 1 for fixture generation is correct given the 12 GiB peak per prover. The accompanying comment clearly documents the OOM root cause.

Reaggregation Cloning (crates/blockchain/src/reaggregate.rs:137)
pubkeys_per_component.clone() inside the candidate loop is O(N×M) where N is attestations and M is candidates (max 4). Given the small constants, this is acceptable, but note that split_type_2_by_message could potentially take the pubkeys by reference to avoid the clone if the lean-multisig API allowed it.

Explicit Drop (crates/blockchain/src/lib.rs:534)
drop(type_one_proofs) is semantically correct but redundant; the compiler would insert this automatically after the final use. No change required.

Code Quality

SSZ Test Skipping (crates/common/types/tests/ssz_spectests.rs:66-78)
The temporary skipping of SignedBlock, BlockSignatures, etc., is acceptable given the explicit TODO and the schema migration context. Ensure an issue is filed to track re-enabling these once the leanSpec pin moves past the Type-2 schema.

Error Cleanup (crates/blockchain/src/store.rs:801-812)
Good cleanup removing AttestationSignatureMismatch and ParticipantsMismatch which are obsolete in the merged-proof world. The new SlotOutOfRange error is more precise.

Fixture Field Renames (crates/common/test-fixtures/src/common.rs:101-108)
The serde renames from attestationPubkey to attestationPublicKey match the leanSpec PR #799 schema updates. Consistent with the comment in verify_signatures.rs.

Suggestions

1. Log Level for Split Failures (crates/blockchain/src/reaggregate.rs:160-165)
   Consider upgrading the debug! log for split_type_2_by_message failures to warn! in production builds. A split failure indicates either a malformed block or a crypto library issue, both worth surfacing prominently.

2. Comment Typo (crates/blockchain/src/reaggregate.rs:2)
   "deconstruct_block_into_store" should be "deconstruct_block_into_pool" or similar to match the actual function semantics (it writes to the store's payload buffer, not the block store).

3. Skip List (crates/blockchain/tests/signature_spectests.rs:16-21)
   The SKIP_TESTS array is empty with a comment indicating Phase-3 crypto stubs. Either populate this with the tests that need real crypto, or remove the empty array to avoid confusion.

Summary

The PR correctly implements the consensus-critical Type-2 signature aggregation scheme with appropriate safety bounds. The CI fixes prevent cache poisoning and OOM kills. The temporary test skipping is documented and acceptable for a schema migration PR.

Approve with minor logging suggestions.

---

Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

[github-actions]

🤖 Codex Code Review

1. crates/storage/src/store.rs synthesizes genesis as ByteList512KiB::default(), but SignedBlock.proof is now defined as an SSZ-wrapped TypeTwoMultiSignature, and merged_proof_bytes() rejects any envelope shorter than 4 bytes. That means the node will serve a structurally invalid genesis/finalized block over BlocksByRoot and /lean/v0/blocks/finalized, which is likely to break cross-client decoding or later verification. This should synthesize SignedBlock::wrap_merged_proof(&[]) instead of an empty byte list.

2. crates/common/crypto/src/lib.rs applies the 512 KiB cap to the raw compress_without_pubkeys() Type-2 bytes, but crates/blockchain/src/lib.rs then prepends the 4-byte SSZ wrapper before putting it into SignedBlock.proof. A proof at the advertised limit will therefore pass aggregation and then fail late during wrapping, turning a valid-size merge into a proposer-side block-build failure. The size check needs to reserve 4 bytes for the envelope, or the merge helper should return raw bytes and let the caller enforce the final wrapped limit.

The rest of the proof-model migration looked internally consistent, especially the end-to-end verifier path and the fork-choice/reaggregation plumbing. Residual risk is mostly test coverage: the SSZ conformance harness now skips the newly changed wire types, so compatibility regressions there are less likely to be caught automatically.

---

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

[github-actions]

🤖 Claude Code Review

Now I have enough context to write the full review.

---

PR Review: feat: add devnet 5 support (#378)

Overview

This PR implements pq-devnet-5 support by migrating from the legacy per-attestation BlockSignatures structure (devnet-4) to a single merged Type-2 SNARK proof on SignedBlock (leanSpec PR #717). It also adds a new reaggregate.rs module that recovers per-attestation Type-1 proofs from imported blocks so block-borne votes gain fork-choice weight.

The scope is large (1520 additions, 923 deletions across 29 files), but the mechanical rename from AggregatedSignatureProof → TypeOneMultiSignature and ByteListMiB → ByteList512KiB accounts for most of the churn. The new crypto layer (merge_type_1s_into_type_2, verify_type_2_signature, split_type_2_by_message) is the core of the change.

---

Correctness

MAX_ATTESTATIONS_DATA silently halved (16 → 8)
crates/common/types/src/block.rs, line 3228. This is a protocol-level change (down from leanSpec PR #536 to PR #717's new cap). The old constant in lib.rs had an explicit citation; the new one has none. This must be confirmed against the spec and coordinated with all other clients.

SSZ spec tests disabled for all affected types
crates/common/types/tests/ssz_spectests.rs. SignedBlock, BlockSignatures, AggregatedSignatureProof, and SignedAggregatedAttestation are skipped with a TODO. The new wire format therefore has no SSZ compatibility test coverage until the leanSpec fixture pin is bumped. This is acknowledged, but it's worth tracking as a hard blocker before claiming devnet-5 readiness.

Bitfield length in the merged-case of reaggregate_from_block
crates/blockchain/src/reaggregate.rs, around line 1344:

let max_vid = union_indices.iter().copied().max().unwrap_or(0);
let mut union_bits =
    ethlambda_types::attestation::AggregationBits::with_length(max_vid as usize + 1)
        ...

When merging a block proof with local partials, the resulting bitfield is sized to max_vid + 1, not the original committee length. If the committee has 100 validators but only validators {5, 10, 15} participated, the merged bitfield would have length 16 rather than 100. Other clients may reject the gossip message if they enforce canonical committee-size bitfields. The block_t1 path (no local partials) correctly clones att.aggregation_bits and preserves the original length; the merge path should do the same. One safe fix: start from att.aggregation_bits.clone() (zeroed out) and OR in the union indices, rather than computing a minimum-length bitfield.

merged_proof_bytes() called inside the candidate loop
crates/blockchain/src/reaggregate.rs, line 1267:

for candidate in candidates {
    // ...
    let Ok(merged_bytes) = signed_block.merged_proof_bytes() else {
        debug!("Reaggregation skipped: block proof envelope unusable");
        return Vec::new();  // drops all prior progress
    };

merged_proof_bytes() only parses a 4-byte header and returns a slice—it's infallible given a valid block (and validity was already checked during import). Calling it in the loop and discarding completed candidates on failure is unnecessary. Extract it once before the loop; if it fails, return early without processing any candidates.

pubkeys_per_component.clone() inside the loop
crates/blockchain/src/reaggregate.rs, line 1273:

let split_bytes = match ethlambda_crypto::split_type_2_by_message(
    merged_bytes,
    pubkeys_per_component.clone(),  // full clone every iteration
    &data_root,
) {

pubkeys_per_component is a Vec<Vec<ValidatorPublicKey>> where each key is 52 bytes. Cloning it on every candidate (up to 4 per MAX_REAGGREGATIONS_PER_BLOCK) is expensive. split_type_2_by_message could take &[Vec<ValidatorPublicKey>] or convert internally. Even if the API requires ownership, one option is to clone once per call only if candidates.len() > 1 and avoid cloning on the last iteration.

---

Minor / Style Issues

Dead code: TypeOneMultiSignature::for_proposer
crates/common/types/src/block.rs, around line 3297. The helper is defined but not used in this PR. The block-building path in lib.rs builds the proposer component manually. Either use the helper there or remove it.

Empty skip list constant adds noise
crates/blockchain/tests/signature_spectests.rs, line 3063:

const SKIP_TESTS: &[&str] = &[];

The associated infrastructure (if SKIP_TESTS.iter().any(...)) runs on every test but always takes the false branch. Fine to keep as a documented extension point, but the comment "every fixture is exercised against the real primitive" is the key claim—it should ideally be backed by at least one passing fixture so CI actually exercises verify_type_2_signature.

leansig pinned to a branch, not a rev
Cargo.toml, line 411:

leansig = { git = "https://github.com/leanEthereum/leanSig", branch = "devnet4" }

The Cargo.lock pins to a specific SHA (15cbdd43...), so this is safe for reproducibility. But the branch reference in Cargo.toml will cause cargo update to silently change the locked hash when the branch tip advances. All other leanVM crates use rev = "e2592df". Pinning leansig the same way would be more consistent and explicit.

PR description template left blank
The description still contains the placeholder text ("What does this PR change?", "List the files or areas touched", etc.). Given the scope of the change this makes it harder to review the intent and behavioral guarantees.

---

CI / Infrastructure

The CI improvements are well-reasoned:

• Switching back to uv run python -m consensus_testing.keys --download --scheme prod (removing the manual curl+tar workaround) is correct now that the leanSpec pin has advanced past PR #745.
• -n 1 for fixture generation with a memory-budget comment is exactly the right documentation for a subtle constraint.
• The cache-poisoning fix (steps.generate-fixtures.outcome == 'success') is a real correctness improvement over the previous always() guard.

---

Summary

The architectural migration from per-attestation BlockSignatures to a single merged Type-2 proof is solid. The main issues to address before merge:

• The bitfield-length bug in the merge path of reaggregate_from_block (could cause gossip rejection by other clients).
• merged_proof_bytes() called in a loop where it should be hoisted out.
• SSZ spec coverage gap—plan to re-enable skipped tests once the fixture pin advances.
• MAX_ATTESTATIONS_DATA change from 16 → 8 needs an explicit spec citation.

---

Automated review by Claude (Anthropic) · sonnet · custom prompt

[greptile-apps]

Greptile Summary

This PR upgrades the client from pq-devnet-4 to pq-devnet-5 by replacing the per-attestation AggregatedSignatureProof model with a single merged Type-2 SNARK that binds all attestation and proposer signatures in one block-level proof (leanSpec PR #717). A new reaggregate.rs module recovers per-attestation Type-1 proofs post-import via SNARK-split, feeding them back into the aggregation pool so block-borne votes carry fork-choice weight.

• Block wire format: SignedBlock now carries a single ByteList512KiB merged-proof field; TypeOneMultiSignature (renamed from AggregatedSignatureProof) is used for gossip-level and in-memory aggregation; MAX_ATTESTATIONS_DATA drops from 16 → 8.
• Block verification: verify_block_signatures is rewritten to call the new verify_type_2_signature primitive after resolving per-component pubkeys from the block body; the previous parallel per-attestation verify path is removed.
• CI hardening: the fixture-generation step is capped at -n 1 to prevent OOM kills on the 16 GiB runner, and the fixture cache is now only saved when generation actually succeeds (eliminating the empty-cache poisoning bug).

Confidence Score: 3/5

The core devnet-5 logic (Type-2 proof assembly, verification, and reaggregation) is well-structured and carefully commented, but the Hive test driver's block-step handler diverges from the offline spec-test runner in a way that will cause incorrect fork-choice results in the Hive environment.

The offline spec-test runner inserts Type-1 proofs into the pool and re-runs update_head after every successful block import, compensating for the removal of insert_known_aggregated_payloads_batch from on_block_core. The Hive test driver's apply_step "block" branch omits both of those steps. Any Hive fork-choice fixture whose expected head depends on block-borne attestation weights will disagree with the spec, making this a live correctness gap in the test-driver path that ships with this PR.

crates/net/rpc/src/test_driver.rs — the "block" step handler needs the same post-import proof-injection and update_head logic that forkchoice_spectests.rs performs.

Important Files Changed

Filename	Overview
crates/net/rpc/src/test_driver.rs	Hive test driver apply_step "block" handler is missing the post-import proof injection and update_head call that the offline spec-test runner performs, creating a fork-choice divergence between the two test paths.
crates/blockchain/src/lib.rs	Major rework of block assembly (Type-1→Type-2 merge) and post-import reaggregation; process_block now returns bool (synced status) and reaggregation runs only when synced.
crates/blockchain/src/reaggregate.rs	New module: recovers per-attestation Type-1 proofs from a block's merged Type-2 via SNARK-split, then folds them into the aggregated payload pool; merged_proof_bytes() called per-iteration inside the candidate loop.
crates/blockchain/src/store.rs	Block verification rewritten for the Type-2 merged-proof model; on_block_core no longer stores per-attestation proofs during import; old proposer/participant mismatch error variants replaced.
crates/common/types/src/block.rs	New SignedBlock uses a single ByteList512KiB merged-proof field; TypeOneMultiSignature replaces AggregatedSignatureProof; MAX_ATTESTATIONS_DATA reduced from 16 to 8; envelope helpers are well-tested.
crates/blockchain/tests/forkchoice_spectests.rs	Correctly simulates post-import reaggregation by inserting Type-1 proofs into the known pool and calling update_head after each successful block import.
.github/workflows/ci.yml	Fixes cache-poisoning bug (no longer saves empty fixture cache on cancelled/OOM runs), simplifies key download to use the upstream helper, and limits prover parallelism to -n 1 to avoid OOM.
crates/blockchain/src/aggregation.rs	Mechanical rename from AggregatedSignatureProof / proof_data to TypeOneMultiSignature / proof; logic unchanged.

Sequence Diagram

sequenceDiagram
    participant V as Validator
    participant BC as BlockChainServer
    participant Crypto as ethlambda_crypto
    participant Store as Store

    V->>BC: propose_block(slot, validator_id, signature)
    BC->>Store: produce_block_with_signatures()
    Store-->>BC: (block, type_one_proofs[])

    BC->>Crypto: aggregate_signatures([proposer_pubkey], [sig], block_root, slot)
    Crypto-->>BC: proposer_proof_bytes (Type-1)

    loop For each type_one_proof
        BC->>Store: resolve attestation pubkeys
        Store-->>BC: pubkeys
    end

    BC->>Crypto: merge_type_1s_into_type_2(merge_inputs)
    Crypto-->>BC: merged_bytes (Type-2)

    BC->>BC: SignedBlock::wrap_merged_proof(merged_bytes)
    BC->>Store: on_block(signed_block)
    Store-->>BC: Ok(synced)

    alt "synced == true"
        BC->>BC: run_reaggregate_from_block(signed_block)
        BC->>Crypto: split_type_2_by_message(merged, pubkeys, data_root)
        Crypto-->>BC: split_bytes (Type-1 per attestation)
        BC->>Store: insert_new_aggregated_payloads_batch(entries)
    end

Loading

Comments Outside Diff (2)

1. crates/net/rpc/src/test_driver.rs, line 353-364 (link)

   Hive test driver missing post-import reaggregation step

   The offline spec-test runner (forkchoice_spectests.rs) inserts Type-1 proofs into the known payload pool and re-runs update_head after every successful block import. This compensates for the removal of insert_known_aggregated_payloads_batch from on_block_core. The Hive test driver's apply_step "block" branch calls on_block_without_verification but performs neither step. As a result, any fixture whose expected head depends on block-borne attestation weights will produce a different head in the Hive runner than in the offline runner, causing those Hive fork-choice tests to fail or silently disagree.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/net/rpc/src/test_driver.rs
   Line: 353-364
   
   Comment:
   **Hive test driver missing post-import reaggregation step**
   
   The offline spec-test runner (`forkchoice_spectests.rs`) inserts Type-1 proofs into the known payload pool and re-runs `update_head` after every successful block import. This compensates for the removal of `insert_known_aggregated_payloads_batch` from `on_block_core`. The Hive test driver's `apply_step` "block" branch calls `on_block_without_verification` but performs neither step. As a result, any fixture whose expected head depends on block-borne attestation weights will produce a different head in the Hive runner than in the offline runner, causing those Hive fork-choice tests to fail or silently disagree.
   
   How can I resolve this? If you propose a fix, please make it concise.

2. crates/blockchain/src/reaggregate.rs, line 1265-1270 (link)

   merged_proof_bytes() called redundantly inside the candidate loop

   signed_block.merged_proof_bytes() returns a shared slice into self.proof and is free to call, but it is invoked once per candidate (up to MAX_REAGGREGATIONS_PER_BLOCK times) instead of once before the loop. If the envelope is invalid, the return Vec::new() on the error arm silently discards any aggregates collected in earlier loop iterations. Calling this before the loop would make the early-return semantics clearer and avoid the partial-progress discard.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/blockchain/src/reaggregate.rs
   Line: 1265-1270
   
   Comment:
   **`merged_proof_bytes()` called redundantly inside the candidate loop**
   
   `signed_block.merged_proof_bytes()` returns a shared slice into `self.proof` and is free to call, but it is invoked once per candidate (up to `MAX_REAGGREGATIONS_PER_BLOCK` times) instead of once before the loop. If the envelope is invalid, the `return Vec::new()` on the error arm silently discards any aggregates collected in earlier loop iterations. Calling this before the loop would make the early-return semantics clearer and avoid the partial-progress discard.
   
   How can I resolve this? If you propose a fix, please make it concise.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
crates/net/rpc/src/test_driver.rs:353-364
**Hive test driver missing post-import reaggregation step**

The offline spec-test runner (`forkchoice_spectests.rs`) inserts Type-1 proofs into the known payload pool and re-runs `update_head` after every successful block import. This compensates for the removal of `insert_known_aggregated_payloads_batch` from `on_block_core`. The Hive test driver's `apply_step` "block" branch calls `on_block_without_verification` but performs neither step. As a result, any fixture whose expected head depends on block-borne attestation weights will produce a different head in the Hive runner than in the offline runner, causing those Hive fork-choice tests to fail or silently disagree.

### Issue 2 of 2
crates/blockchain/src/reaggregate.rs:1265-1270
**`merged_proof_bytes()` called redundantly inside the candidate loop**

`signed_block.merged_proof_bytes()` returns a shared slice into `self.proof` and is free to call, but it is invoked once per candidate (up to `MAX_REAGGREGATIONS_PER_BLOCK` times) instead of once before the loop. If the envelope is invalid, the `return Vec::new()` on the error arm silently discards any aggregates collected in earlier loop iterations. Calling this before the loop would make the early-return semantics clearer and avoid the partial-progress discard.

_{Reviews (1): Last reviewed commit: "docs: update README devnet support for p..." | Re-trigger Greptile}