Skip to content

replayer: add bounded Rust dry-run foundation - #1388

Open
bb7133 wants to merge 4 commits into
rustfrom
migration/bb7133-pr-210
Open

bb7133 wants to merge 4 commits into
rustfrom
migration/bb7133-pr-210

Conversation

@bb7133

@bb7133 bb7133 commented Oct 16, 2026

Copy link
Copy Markdown
Member

Recreated from bb7133/tiproxy#210. Original author: bb7133; created: 2026-09-03T11:19:13Z; source body SHA-256: 72df8c9adf732cc77ae9e88128fdf8b8dc8d2d74355e97fdd2bf3bfc0c83ce7e. Historical reviews/checks remain on the source PR.

What is changed

This draft advances the standalone Rust owner for CP-REPLAYER (#1320) without changing the SQL-serving binary or claiming production cutover.

  • freeze the Rust offline-replayer ownership, compatibility, restart, and safety contract
  • add a standalone tiproxy-replayer composition root which currently accepts dry-run only
  • add bounded OpenDAL input ownership for fs, s3, gcs, azblob, oss, and cos
  • decode TiProxy native captures plus audit plugin v1/v2 records with deterministic ordering
  • preserve Go-compatible prepared-statement and read-only-filter behavior, including binary KindBytes
  • support gzip + AES-256-CTR native inputs with bounded stored, expanded, metadata, and record sizes
  • add content/config-bound, full-frontier schema-v2 atomic local checkpoints
  • add an immutable Go/Rust semantic corpus, differential comparator, and mutation kill to CI

Review hardening

The second exact head addresses the independent semantic review:

  • make bare checkpoint filenames sync the current directory correctly
  • preserve binary plugin parameters as bytes instead of rejecting non-UTF-8 payloads
  • add command_ordinal to ordering, resume frontiers, checkpoint schema, and input identity
  • reject unknown, duplicate, root, bucket, and container object-store query overrides
  • match the Go audit-extension command-start/end frontier mapping
  • fail closed on start-time grace-period overflow
  • narrow the evidence, retry, zeroization, and reorder-buffer claims to what this foundation proves
  • shorten all temporary OpenDAL RustSec exception expiries to 2026-12-01

Honest scope boundary

This is a foundation-only draft and does not close #1320. The following remain intentionally fail-closed or deferred:

  • backend MySQL/TLS/authentication, response draining, pacing, reconnects, and error policy
  • replay-only HTTP service mode and lifecycle
  • dynamic input / wait-on-EOF ownership and remote checkpoint publication
  • SQL/report output, signals, supervised restart, and release packaging
  • bounded storage retry plus MinIO/provider, fault, and restart evidence
  • bounded merge/scheduling that honors the configured reorder window
  • final bridge/Go-owner removal and production cutover

tiproxy-replayer rejects service mode, backend execution, dynamic input, wait-on-EOF, and SQL output in this slice.

Supply-chain exception disclosure

The workspace is pinned to Rust 1.89 and Tokio 1.47.1. OpenDAL 0.53.3 is therefore used for the required provider set; its signer graph currently carries RUSTSEC-2023-0071, RUSTSEC-2026-0194, and RUSTSEC-2026-0195. OpenDAL 0.58 resolves the quick-xml advisories but requires Rust 1.91.

The exceptions are explicitly synchronized across cargo-audit, cargo-deny, and the repository-owned exception ledger with owner @bb7133, rationale, and expiry 2026-12-01. Exposure in this draft is bounded to an operator-run dry-run binary with trusted storage endpoints; backend and service modes fail closed. The OpenDAL/toolchain upgrade is required before enabling either deferred mode.

Verification

  • make rust-test
  • make rust-lint
  • cargo test -p replayer-core --locked
  • tests/replayer/differential/run.sh (Go/Rust equality + mutation detected)
  • python3 rust/ci/check_vulnerability_exceptions.py
  • hosted Audit dependencies and policy (cargo-audit + cargo-deny + exception ledger)
  • git diff --check

Issue: #1320

@ti-chi-bot

ti-chi-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 16, 2026
@ti-chi-bot
ti-chi-bot Bot requested a review from YangKeao October 16, 2026 22:26
@ti-chi-bot

ti-chi-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign bb7133 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1
  • release-*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5aaa8658-1222-4e4b-893a-6d4909c34337

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 16, 2026
@pingcap-cla-assistant

pingcap-cla-assistant Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@bb7133
bb7133 marked this pull request as ready for review October 19, 2026 03:09
@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 19, 2026
@ti-chi-bot
ti-chi-bot Bot requested a review from djshow832 October 19, 2026 03:09
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T03:12:46.465057Z 445d678 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 445d678432

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +279 to +287
CommandCode::STMT_EXECUTE
| CommandCode::STMT_SEND_LONG_DATA
| CommandCode::STMT_RESET
| CommandCode::STMT_FETCH => {
Ok(command.captured_statement_id.is_some_and(|statement_id| {
prepared
.get(&command.connection_id)
.is_some_and(|statements| statements.contains(&statement_id))
}))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear stale prepared IDs after connection reset

When --read-only processes a captured connection that prepares a SELECT with ID N, receives COM_RESET_CONNECTION, and then reuses N for a write prepare/execute, this branch still finds N because the map is only cleared on QUIT. The capture side explicitly clears its prepared-statement state on reset (pkg/sqlreplay/capture/capture.go:404-405), so the later write STMT_EXECUTE is incorrectly counted as replayable under read-only mode. Clear the per-connection prepared state on reset (and other session-reset commands) and remove an existing ID when its replacement prepare is not read-only.

Useful? React with 👍 / 👎.

Comment on lines +873 to +875
if bytes[index] != b'"' {
return Err("expected quoted audit parameter".to_owned());
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Support single-quoted audit-plugin parameters

A valid audit-plugin EXECUTE_PARAMS value can decode to a list such as ['KindInt64 1']: the existing Go decoder explicitly accepts both quote types before unquoting each parameter (pkg/sqlreplay/cmd/audit_log_plugin.go:400-412). This parser requires every list element to begin with ", so such captures abort the dry run instead of producing the same prepared execute command as the Go replayer.

Useful? React with 👍 / 👎.

@bb7133
bb7133 force-pushed the migration/bb7133-pr-210 branch from 445d678 to c431d5a Compare October 19, 2026 03:17
@ti-chi-bot

ti-chi-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown

@bb7133: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-unit-test c431d5a link true /test pull-unit-test
pull-check c431d5a link true /test pull-check

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant