Skip to content

Add autosplit committed-window reader#1152

Open
bootjp wants to merge 1 commit into
mainfrom
feature/hotspot-m3-observe-reader
Open

Add autosplit committed-window reader#1152
bootjp wants to merge 1 commit into
mainfrom
feature/hotspot-m3-observe-reader

Conversation

@bootjp

@bootjp bootjp commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • add committed window boundaries to keyviz matrix columns
  • add an autosplit snapshot reader that converts committed keyviz columns into detector windows without calling SplitRange
  • document the M3-PR2b-a slice and leave Top-K, leadership watermarks, and scheduler wiring as remaining M3 work

Evidence

  • go test ./keyviz ./distribution/autosplit
  • go test ./adapter ./internal/admin -run 'KeyViz|Subrange|TestCommittedWindows|TestObserveSnapshot|TestFlushStampsWindowStart'\n- git diff --check\n\nAuthor: bootjp

Summary by CodeRabbit

  • 新機能
    • スナップショットから確定済みウィンドウを読み取り、観測専用の自動分割判定を実行できるようになりました。
    • ウィンドウ開始時刻を優先して処理し、不正なデータをスキップできるようになりました。
  • 改善
    • サンプラーがウィンドウの開始・終了時刻を正確に保持するようになりました。
    • スナップショットのコピー時にもウィンドウ境界情報が維持されます。
  • ドキュメント
    • 自動化マイルストーンの進捗と関連作業の整理を更新しました。
  • テスト
    • ウィンドウ境界、読み取り範囲、データ欠落時のリセット動作を検証するテストを追加しました。

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (63ea6a7):

  • distribution/autosplit/detector.go
  • distribution/autosplit/sampler_reader.go
  • distribution/autosplit/sampler_reader_test.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

MatrixColumn にコミット区間の開始時刻を追加し、deep copy で保持するよう変更しました。スナップショットを ColumnWindow に変換して autosplit detector で評価する observe-only 読み取り経路と、その境界処理・リセット処理のテストを追加しました。

Changes

コミット済みウィンドウと observe-only 統合

Layer / File(s) Summary
MatrixColumn の境界保持と deep copy
keyviz/sampler.go, keyviz/ring_buffer.go, keyviz/sampler_test.go
MemSampler.FlushWindowStart を設定し、Snapshot の deep copy が境界情報を保持するよう変更しました。
スナップショットから detector への変換
distribution/autosplit/detector.go, distribution/autosplit/sampler_reader.go
スナップショット読み取り設定と結果型を追加し、WindowStart を優先した ColumnWindow 変換結果を Evaluate へ渡す経路を追加しました。
読み取り経路の検証と設計記録
distribution/autosplit/sampler_reader_test.go, docs/design/...
境界フォールバック、取得範囲、無効列のスキップ、サイレントギャップのリセット処理をテストし、Milestone 3 の PR 分割記録を更新しました。

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ObserveSnapshot
  participant ReadCommittedWindows
  participant SnapshotSource
  participant CommittedWindowsFromColumns
  participant Evaluate
  ObserveSnapshot->>ReadCommittedWindows: SnapshotReadConfig を渡す
  ReadCommittedWindows->>SnapshotSource: Snapshot(from, now)
  SnapshotSource-->>ReadCommittedWindows: MatrixColumn スナップショット
  ReadCommittedWindows->>CommittedWindowsFromColumns: 列を ColumnWindow へ変換
  CommittedWindowsFromColumns-->>ReadCommittedWindows: Windows と newest を返す
  ObserveSnapshot->>Evaluate: Windows と Now を渡す
  Evaluate-->>ObserveSnapshot: Result を返す
Loading

Possibly related PRs

  • bootjp/elastickv#639: cloneColumnMatrixColumn.WindowStart を保持する変更に関連します。
  • bootjp/elastickv#1097: committed MatrixColumnColumnWindow に変換して detector の Evaluate へ渡す経路に関連します。
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed タイトルは主要変更であるautosplitのcommitted-window reader追加を簡潔に表しています。
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md`:
- Line 12: Update the “Updated” date in the milestone 3 automation design
document to the actual modification date, replacing the future/review-following
date while preserving the document’s date format.

In `@keyviz/sampler.go`:
- Around line 1239-1243: Flush 全体を専用 mutex で直列化し、At の取得から history への push、および
lastFlushAt の更新までを同じロックで保護してください。対象は Flush の処理フローと、WindowStart を補正して col
を追加する箇所であり、並行実行時も列順序と実際の window 境界が一致する状態を維持してください。
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: fe18b72f-4498-4d5e-ace9-832fcc18f14d

📥 Commits

Reviewing files that changed from the base of the PR and between 5d4c5ca and 63ea6a7.

📒 Files selected for processing (7)
  • distribution/autosplit/detector.go
  • distribution/autosplit/sampler_reader.go
  • distribution/autosplit/sampler_reader_test.go
  • docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md
  • keyviz/ring_buffer.go
  • keyviz/sampler.go
  • keyviz/sampler_test.go

Author: bootjp
Date: 2026-06-11
Updated: 2026-07-18
Updated: 2026-07-23

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

更新日を実際の更新日に修正してください。

2026-07-23 はレビュー日である 2026-07-22 の翌日です。未来日付の設計記録にならないよう、実際の更新日へ訂正してください。

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md` at
line 12, Update the “Updated” date in the milestone 3 automation design document
to the actual modification date, replacing the future/review-following date
while preserving the document’s date format.

Comment thread keyviz/sampler.go
Comment on lines +1239 to +1243
col.WindowStart = s.lastFlushAt
if !col.WindowStart.Before(col.At) {
col.WindowStart = col.At.Add(-s.opts.Step)
}
s.lastFlushAt = col.At

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Flush 全体を直列化して、列順序と境界を一致させてください。

At は Line 1215 で history lock の前に取得されます。並行した Flush で新しい At の列が先に push されると、古い列は lastFlushAt より前になり補正され、ring buffer の時系列順序と実際の window 境界が壊れます。Range の二分探索や detector の duration 判定が誤ります。専用 mutex で時刻取得から history push までを直列化してください。

修正案
 type MemSampler struct {
+	flushMu sync.Mutex
 	// historyMu guards the ring buffer.
 	historyMu sync.Mutex
 }

 func (s *MemSampler) Flush() {
 	if s == nil {
 		return
 	}
+	s.flushMu.Lock()
+	defer s.flushMu.Unlock()
+
 	col := MatrixColumn{At: s.now()}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
col.WindowStart = s.lastFlushAt
if !col.WindowStart.Before(col.At) {
col.WindowStart = col.At.Add(-s.opts.Step)
}
s.lastFlushAt = col.At
type MemSampler struct {
flushMu sync.Mutex
// historyMu guards the ring buffer.
historyMu sync.Mutex
}
func (s *MemSampler) Flush() {
if s == nil {
return
}
s.flushMu.Lock()
defer s.flushMu.Unlock()
col := MatrixColumn{At: s.now()}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@keyviz/sampler.go` around lines 1239 - 1243, Flush 全体を専用 mutex で直列化し、At の取得から
history への push、および lastFlushAt の更新までを同じロックで保護してください。対象は Flush
の処理フローと、WindowStart を補正して col を追加する箇所であり、並行実行時も列順序と実際の window
境界が一致する状態を維持してください。

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63ea6a7f62

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +105 to +106
if !col.At.After(lastProcessedAt) {
continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reset stale confidence when history no longer reaches the watermark

When an autosplit evaluation is delayed past the retained keyviz history (or HistoryColumns is smaller than the confidence window), Snapshot can return only columns whose first At is already well after LastProcessedAt. This loop treats those columns as normal new evidence and Evaluate preserves the route's existing ConsecutiveOver, so a route with e.g. 2/3 hot columns before the gap can promote on the first retained hot column instead of failing closed and re-earning the full window count. Detect that the fetched history no longer bridges LastProcessedAt and reset/advance the detector state (or surface the gap to the caller) before appending windows.

Useful? React with 👍 / 👎.

Comment on lines +112 to +114
if start.IsZero() || !start.Before(col.At) {
skipped++
continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reset confidence instead of dropping invalid windows

When a fetched committed column has no proven lower boundary (for example a zero/malformed WindowStart), this branch only increments SkippedInvalid and drops the column before Evaluate sees it. That preserves any existing ConsecutiveOver streak across an unscored interval, so a route that was hot before the invalid column can promote on the next hot column even though the unknown window should fail closed. Clear route confidence through col.At when skipping here, or pass an invalid ColumnWindow to the detector so its reset path runs.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant