hotfix(prod): Serialized live on benches 001, 004, 005, 008, 090 (+ contested-chain scoring the wallet-labels spec relies on) - #2422
Merged
Conversation
…nominator (#2279) Two problems on /products/[slug], both visible on Serialized. rankPerChainForBench only holds one real per-chain fact: the leader, from bestPerChain. For every other provider it reused the unfiltered aggregate order shifted by one slot, so a chip reading "#3 on Solana" was the global rank with a chain label on it, repeated identically across every chain of the bench. That reads as a measurement and is not one. Non-leader chips are gone: a chip now means "leads this chain", and its absence means "does not lead", not "ranks lower". The chips also hid how many providers were measured on the chain. "#1 on Solana" was #1 of 2 on bench 008 while sitting next to "#3 of 8" for the aggregate. They now read "#1 of 4 on Ethereum", with the denominator taken from providersPerChain and omitted when that set is unknown rather than substituting the global count. Wins accounting is unchanged: it already counted rank === 1 entries only. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit eda2da8)
…he mix (#2280) On a bench with chain dimensions the cross-chain aggregate is a mix, not a comparison. Ranking on it alone let a provider measured on one chain that nobody else reported finish above a provider that led several contested ones. Five live benches shipped that way: rpc-capabilities Binance 1st on 1 chain, PublicNode led 6 wallet-labels-coverage XRPScan 1st on 1 chain, Serialized led 4 token-quote-coverage Jupiter 1st on 1 chain, Mobula led 2 bridge-fee Squid Router 1st on 1 chain perp-liq-rate Lighter 1st on 1 chain rankedCandidates now sorts by contested-chain wins first and uses the aggregate value only to break ties. A chain counts only when at least two providers reported on it, so an uncontested chain awards nothing: you do not win a race you ran alone. Guarded by the per-chain stashes, which materialize/load.ts populates only on the unfiltered view. A chain-filtered variant has none, so ?chain=bnb keeps ranking by value as before. providers.ts reuses the same ordering. The two surfaces disagreeing is what put "#3 of 8" next to five chain-leadership chips on the same bench row. Known and accepted: a provider with one contested win now ranks above one with none and a higher aggregate figure (TonAPI over XRPScan on 008). That is what ranking on head-to-head record means; the win count is on the row. Rule documented in methodology, section II. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit b6ba6b0)
…2281) Replaces the global chain-wins sort from #2280, which fixed the ordering but broke the reading of it. Ranking on a key the reader cannot see in the column produced tables that no longer descend: 79.88% at rank 4 on 008, and a 74 ms leader at rank 3 on rpc-capabilities where lower is better. The defect was never the sort key. It is that the aggregate includes chains where a provider had no competitor, so the fix belongs on the value: with `score_scope: contested_chains` a bench is scored only on chains carrying at least two measured providers, and a provider with none of those leaves the ranked field (still visible on its own chain tab). One quantity on screen, ordering follows from it. Opt-in per bench and named for the property, not the bench, so any bench whose chain set contains uncontested chains is a candidate and the rule is readable in the public YAML. Enabled on wallet-labels-coverage only: stellar, xrp and bitcoin carry one measured provider each there. load.ts already fetched every provider's per-chain value inside its per-chain loop and discarded all but the leader; it now keeps them. Four other live benches qualify and are deliberately left untouched pending review: rpc-capabilities, token-quote-coverage, bridge-fee, perp-liq-rate. Residual limitation documented in methodology: providers are still averaged over the different subsets of contested chains they cover. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 401d0a7)
Wires serialized.xyz into metadata-coverage (004) and wallet-labels (008), the two benches where their endpoints map 1:1 to the existing scoring rule. Both providers throttle client-side at ~16 rps: Serialized enforces a hard 40 req/s burst cap and an unthrottled worker pool turns coverage into a rate-limit artifact (measured 77% -> 37%). Adds the registry entry, logo and a full onboarding audit documenting the apples-to-apples numbers and two scoring flaws the tests exposed in our own benches (logo presence vs resolution on 004, name-service strings counted as entity labels on 008). Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 5dc8d21)
The harnesses emit provider="serialized" but the leaderboards are driven by the explicit providers block in each spec, so the series never rendered. Adds the block for both benches, same query shape as the incumbents. Validated with SpecSchema.safeParse across all 218 specs; every formula stays under the 240-char cap that silently drops a spec. Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 0730b30)
Serialized publishes its chain list at the free GET /v1/meta/chains. The same list answers both benches: they run their own indexers and do not separate asset-registry coverage from DEX-pool coverage, so the count is identical on 005 and 090 by construction. Only chains the endpoint marks status=live are counted, so a future beta or deprecated status cannot inflate the number. Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 8ae50fe)
…ups (#2264) v1.0 claimed bench 001 references archive nodes and was therefore unaffected by provider clock disagreement. That repeated the spec instead of reading the harness: there is no archive-node reference in aggregator-head-lag at all, and the leaderboard gauge is computed from each provider's own timestamp. The spec says the opposite in three places. Also records the now-conclusive bench 067 result (verified 5 of 19 chains), probe-confirmed negative capabilities, and a verified 5.2x BONK mispricing traced to pool discovery missing the main market. Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit dcfe080)
…alized (#2297) Base was the only chain whose zero point was wrong, and measurably so: a block becomes queryable ~0.36 s after the timestamp it carries (median over 25 consecutive blocks) while the sequencer publishes flashblock preconfirmations every 200 ms inside the 2 s interval. Measuring against the block timestamp therefore described the chain's stamping convention, not the provider's pipeline, and charged anything reading preconfirmations with a negative lag: 99% of Serialized's Base emissions, 140 of 147. base_flashblock_ref.go feeds the existing reference clock from that stream. The clock already keeps the first observation, so the preconfirmation wins over the sealed-block logs subscription without further change. The endpoint is public and anycast: 1.1 ms RTT from both the Paris and Singapore boxes, the same distance as the provider endpoints, so no region is handicapped against the feeds it measures. headlineLag applies the substitution in one place, so it lands on all four providers at once. Re-basing one provider and leaving the others on the old ruler is the asymmetry this change exists to remove. A Base emission with no flashblock match is dropped, not measured against a different ruler: a dead reference must show up as missing data, never as quietly different numbers. Verified from two vantage points before shipping. Serialized on Base moves from -1.016 to +0.239 (Paris) and -1.081 to +0.244 (laptop), stable, with 4 negatives out of 125 left as network jitter. Mobula's old-reference figure from Paris (+1.070) matches production (0.951), which is what makes the vantage trustworthy. Only Base changes. Solana, BNB and Robinhood carry no preconfirmation layer a provider consumes and their chain-supplied timestamps sit within ~60 ms of the observable moment, so they keep the on-chain timestamp. Serialized is re-added with the pinned Robinhood token address corrected: the old value acked and delivered nothing, which is what produced the "they publish nothing on Robinhood" report. With the address /v1/pool returns, they deliver 98 events in 5 minutes at p50 +0.596 s. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 9a354bc)
…mark centred in its box (#2400) Claude-Session: https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit eb3591f)
Flotapponnier
added a commit
that referenced
this pull request
Sep 17, 2026
…the prod sitemap smoke passes (#2423) The Serialized deploy (#2422) rolled back: the fresh all-benchmarks cache listed bench 267 from the dev-built blob and its prod URL 404s. Claude-Session: https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks from dev, in order: #2279, #2280, #2281, #2282 (contested-chain ranking +
score_scope, required by the wallet-labels spec), #2261, #2262, #2263, #2264 (Serialized on 004/008/005/090 + audit doc), #2297 spec part (Serialized on aggregator-head-lag; the harness half was already on main, main's harness kept byte-for-byte), #2301, #2400 (brand blue, centred logo).bench-unfiltered-v65,all-benchmarks-v60.pnpm validate221 green,bun test263 pass, typecheck clean.After merge: rebuild the materialize worker so the blobs pick up the five specs.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw