SLM-418: convert replay preference rows to PreferencePair (DSH5-10) by Tyler-R-Kendrick · Pull Request #1131 · Tyler-R-Kendrick/slm-training

Tyler-R-Kendrick · 2026-07-27T11:47:08Z

Summary

Sixth slice of the DSH5-10 replay-preference scope (SLM-418). Extraction of all 7 named patterns finished in the prior slice (#1119); this takes the smallest next step of the issue's still-unattempted training scope: converting an already-extracted OperatorReplayPreferenceRowV1 into this repo's existing PreferencePair shape (slm_training.harnesses.preference) — the same schema slm preference build-pairs/train already reads/writes — so a later slice can feed rows into that harness without inventing a second corpus shape.

preference_pair_from_replay_row / preference_pairs_from_trace (new, in src/slm_training/dsl/operators/replay_preference.py), re-exported from operators/__init__.py.
prompt is the row's real, pack-authorized input-state OpenUI source — a ConversationTraceV1's turns are AST operations, never user utterances, so there's no natural-language instruction to recover instead. chosen/rejected are the row's own legal-set action tokens verbatim (e.g. "undo", "checkout:<state>"), not replayed alternate full-program renderings — that replay is explicitly out of scope for this slice.
composite_reward is deliberately never called on those tokens (scoring "undo" as OpenUI source would manufacture a meaningless number); chosen_score/rejected_score stay at the 0.0 default, with the distinction recorded in meta.
A row whose input_state_id is foreign to its trace (the MERGE_SUCCESS case, grounded on a branch-edit tip rather than any single trace) is skipped with a recorded reason, never fabricated.
8 new regression tests: one conversion test per each of the 7 extracted relation types, plus one proving the blank-prompt refusal.

Claim class: wiring, same tier as every prior slice in this doc. Not attempted (unchanged from before this slice): SFT/preference training against the DSH3-selected policy/control heads, the four-baseline comparison, held-out benefit measurement, turn-depth/context-view ablations, or replaying a rejected action to an alternate program.

Verification (real, this session)

NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py \
  tests/test_dsl/test_operator_conversation.py tests/test_evals/test_advanced_operator_disposition.py \
  tests/test_scripts/test_validate_advanced_operator_disposition.py

→ 69 passed (61 baseline + 8 new)

ruff check on every changed Python file → clean
python -m scripts.verify_version_stamps --check --base origin/main → ok (dsl.operators.replay_preference v6→v7; dsl.operators.contracts no-bump for the new re-exports)
python -m scripts.repo_policy → ok
python -m scripts.verify_decode_invariants → clean

Full details, design rationale, and the doc's own reproducibility log: docs/design/dsh5-10-replay-preference-rows.md — "Sixth slice (v7)".

Test plan

pytest -q tests/test_dsl/test_replay_preference.py … (see above) — 69 passed
ruff check — clean
verify_version_stamps --check — ok
repo_policy — ok
verify_decode_invariants — clean

Generated by Claude Code

Summary by CodeRabbit

New Features
- Added conversion of replay preference rows into the standard preference-pair format.
- Added support for preserving chosen and rejected actions with related metadata.
- Added safeguards to skip rows unavailable in the trace and reject blank prompts.
Documentation
- Updated slice status, supported conversions, safeguards, and verification details.
Tests
- Added coverage for multiple replay scenarios, successful conversions, skipped rows, and invalid prompts.

Sixth slice of the DSH5-10 replay-preference scope: extraction of all 7 named patterns finished in the prior slice, so this takes the smallest next step of the still-unattempted training scope -- converting an already extracted OperatorReplayPreferenceRowV1 into this repo's existing PreferencePair shape (the same schema slm preference build-pairs/train already reads/writes), so a later slice can feed rows into that harness without inventing a second corpus shape. prompt is the row's real, pack-authorized input-state OpenUI source (traces carry no user utterances to recover instead); chosen/rejected are the row's own legal-set action tokens verbatim, not replayed alternate programs; composite_reward is never called on those tokens since scoring "undo" as OpenUI source would manufacture a meaningless number. A row whose input state is foreign to its trace (e.g. MERGE_SUCCESS, grounded on a branch-edit tip) is skipped with a reason, never fabricated. Claim class: wiring, same tier as every prior slice. No training, checkpointing, or composite-reward scoring is attempted; the DSH3 policy heads, four-baseline comparison, and held-out measurement remain fully unattempted. Verified (fresh Python 3.12 venv, NODE_OPTIONS= npm ci in src/apps/openui_bridge): 69 passed (61 baseline + 8 new) across tests/test_dsl/test_replay_preference.py plus the sibling merge/ conversation/disposition suites; ruff check clean; verify_version_stamps --check ok (dsl.operators.replay_preference v6->v7); repo_policy ok; verify_decode_invariants clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUwBZ5X9HJzzvVjXcd2RZ9

vercel · 2026-07-27T11:47:14Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
slm-training	Ready	Preview, Comment	Jul 27, 2026 11:47am

coderabbitai · 2026-07-27T11:47:30Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 15d79158-dbc0-46f7-9a52-614ec53dd585

📥 Commits

Reviewing files that changed from the base of the PR and between 5f94b92 and f3adde1.

📒 Files selected for processing (5)

docs/design/dsh5-10-replay-preference-rows.md
src/slm_training/dsl/operators/__init__.py
src/slm_training/dsl/operators/replay_preference.py
src/slm_training/resources/versions.json
tests/test_dsl/test_replay_preference.py

📝 Walkthrough

Walkthrough

Changes

This change adds conversion from extracted replay preference rows into PreferencePair objects, exports and versions the helpers, and adds tests and design documentation for mapping, trace validation, skipped rows, and blank prompt rejection.

Replay preference conversion

Layer / File(s)	Summary
Replay row conversion `src/slm_training/dsl/operators/replay_preference.py`	Adds helpers that create `PreferencePair` objects from replay rows, preserve grounded metadata, reject blank prompt sources, and skip rows missing from the trace.
Public exports and version registry `src/slm_training/dsl/operators/__init__.py`, `src/slm_training/resources/versions.json`	Exports the conversion helpers and records the replay-preference v7 component history.
Conversion validation and slice documentation `tests/test_dsl/test_replay_preference.py`, `docs/design/dsh5-10-replay-preference-rows.md`	Tests conversion and guardrails, while documenting the sixth-slice scope and updated verification results.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant preference_pairs_from_trace
  participant ConversationTraceV1
  participant preference_pair_from_replay_row
  participant PreferencePair
  Caller->>preference_pairs_from_trace: provide trace and replay rows
  preference_pairs_from_trace->>ConversationTraceV1: locate input_state_id
  ConversationTraceV1-->>preference_pairs_from_trace: return input state source
  preference_pairs_from_trace->>preference_pair_from_replay_row: convert grounded row
  preference_pair_from_replay_row->>PreferencePair: create mapped preference pair

Possibly related PRs

Tyler-R-Kendrick/slm-training#1075: Adds the replay preference row extraction that these conversion helpers consume.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: converting replay preference rows into PreferencePair and references the related slice.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/great-dirac-ubqjrd

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…1132) * SLM-418: convert replay preference rows to PreferencePair (DSH5-10) Sixth slice of the DSH5-10 replay-preference scope: extraction of all 7 named patterns finished in the prior slice, so this takes the smallest next step of the still-unattempted training scope -- converting an already extracted OperatorReplayPreferenceRowV1 into this repo's existing PreferencePair shape (the same schema slm preference build-pairs/train already reads/writes), so a later slice can feed rows into that harness without inventing a second corpus shape. prompt is the row's real, pack-authorized input-state OpenUI source (traces carry no user utterances to recover instead); chosen/rejected are the row's own legal-set action tokens verbatim, not replayed alternate programs; composite_reward is never called on those tokens since scoring "undo" as OpenUI source would manufacture a meaningless number. A row whose input state is foreign to its trace (e.g. MERGE_SUCCESS, grounded on a branch-edit tip) is skipped with a reason, never fabricated. Claim class: wiring, same tier as every prior slice. No training, checkpointing, or composite-reward scoring is attempted; the DSH3 policy heads, four-baseline comparison, and held-out measurement remain fully unattempted. Verified (fresh Python 3.12 venv, NODE_OPTIONS= npm ci in src/apps/openui_bridge): 69 passed (61 baseline + 8 new) across tests/test_dsl/test_replay_preference.py plus the sibling merge/ conversation/disposition suites; ruff check clean; verify_version_stamps --check ok (dsl.operators.replay_preference v6->v7); repo_policy ok; verify_decode_invariants clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUwBZ5X9HJzzvVjXcd2RZ9 * docs(autotrain): joint seed x steps smoke sweep, real new evidence Follow-up to the ledger's diminishing-returns note after batch #4: instead of another single-variable check, this batch varies --seed (1, 2, 3) and --steps (4, 16) jointly against main HEAD f3adde1 (PR #1131, already merged), same wf_smoke_v2 fixture/model/recipe as every prior batch. 6 more independently-run rows, combined with the prior single-variable batches into a full seed x steps grid: seed-to-seed spread is real but bounded and shrinks as steps increase, and seed introduces variance without a directional bias (seed 0 is neither systematically highest nor lowest). Genuinely new joint evidence, not a repeat of either single-variable batch. Still fixture_or_scratch: n=1 per (seed, steps) cell, 101-record fixture, context-backend scratch. No convergence/generalization/ship claim. Per the ledger's own next-steps note, the smoke-loop's role as a harness liveness check is now thoroughly covered at this fixture size; the next scheduled iteration should move off this fixture entirely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUwBZ5X9HJzzvVjXcd2RZ9 --------- Co-authored-by: Claude <noreply@anthropic.com>

…er adapter gap Investigation-only design note, no code or training changes. Confirms and extends PR #1131's reward-shape mismatch finding with a deeper structural gap: OperatorActionViewV1 (operator_policy_view.py) requires operator_id/ verdict/coverage fields that history-control actions (undo/redo/checkout/ merge -- 6 of the 7 confirmed ReplayPreferenceRelation members) simply don't have, since those actions never route through the operator-legality compiler. Only PRONOUN_FOCUS_FOLLOWUP rows draw both chosen/rejected from real OperatorActionViewV1-compatible legal_actions. Lays out two candidate designs (extend the frozen OperatorPolicyInputV1 schema vs. a separate history-control scorer) with tradeoffs and a recommendation, so a future session with real time budget (MAX_RUN_MINUTES is 3, too tight for this) doesn't have to re-derive the mismatch before writing code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VDeTk7RxBAbWX5fKjBqb4r

…er adapter gap (#1144) Investigation-only design note, no code or training changes. Confirms and extends PR #1131's reward-shape mismatch finding with a deeper structural gap: OperatorActionViewV1 (operator_policy_view.py) requires operator_id/ verdict/coverage fields that history-control actions (undo/redo/checkout/ merge -- 6 of the 7 confirmed ReplayPreferenceRelation members) simply don't have, since those actions never route through the operator-legality compiler. Only PRONOUN_FOCUS_FOLLOWUP rows draw both chosen/rejected from real OperatorActionViewV1-compatible legal_actions. Lays out two candidate designs (extend the frozen OperatorPolicyInputV1 schema vs. a separate history-control scorer) with tradeoffs and a recommendation, so a future session with real time budget (MAX_RUN_MINUTES is 3, too tight for this) doesn't have to re-derive the mismatch before writing code. Claude-Session: https://claude.ai/code/session_01VDeTk7RxBAbWX5fKjBqb4r Co-authored-by: Claude <noreply@anthropic.com>

vercel Bot deployed to Preview July 27, 2026 11:47 View deployment

Tyler-R-Kendrick mentioned this pull request Jul 27, 2026

docs(autotrain): joint seed x steps smoke sweep, real new evidence #1132

Merged

3 tasks

Tyler-R-Kendrick merged commit b349623 into main Jul 27, 2026
6 checks passed

Tyler-R-Kendrick deleted the claude/great-dirac-ubqjrd branch July 27, 2026 14:45

Tyler-R-Kendrick mentioned this pull request Jul 27, 2026

SLM-418: render replay preference rows into trainable pairs (DSH5-10) #1124

Closed

6 tasks

Tyler-R-Kendrick mentioned this pull request Jul 27, 2026

docs(DSH5-10): scope the replay-preference -> TypedOperatorPolicyScorer adapter gap #1144

Merged

4 tasks

Navigation Menu

SLM-418: convert replay preference rows to PreferencePair (DSH5-10) - #1131

SLM-418: convert replay preference rows to PreferencePair (DSH5-10)#1131
Tyler-R-Kendrick merged 1 commit into
mainfrom
claude/great-dirac-ubqjrd

Tyler-R-Kendrick commented Jul 27, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Tyler-R-Kendrick commented Jul 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification (real, this session)

Test plan

Summary by CodeRabbit

Uh oh!

vercel Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Tyler-R-Kendrick commented Jul 27, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Jul 27, 2026 •

edited

Loading

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading