LongProc RL driver: long procedural generation with verifiable rewards - #6
Open
mmjerge wants to merge 7 commits into
Open
LongProc RL driver: long procedural generation with verifiable rewards#6mmjerge wants to merge 7 commits into
mmjerge wants to merge 7 commits into
Conversation
… rewards) LongProc outputs are 0.5k-8k tokens and rule-checkable (row-F1 for html_to_tsv, constraint satisfaction for travel_planning, ...), so the programmatic evaluators serve directly as RLVR rewards -- and the long decode exercises cache eviction during *generation*, a regime the HELMET QA campaigns (long prefill, 32-token decode) never touch. Includes --eval-only for base-model capability probes; memory recipe defaults match the validated 7B/48GB configuration.
…ature 0.7) At temperature 1.0 no sampled rollout produces the strict fenced output format the checkers require, so the primary metric is 0.0 for every group member and RLOO has no gradient (observed: 30+ steps of all-zero rewards while greedy eval scores 0.157). Shape the training reward with a small bonus on the evaluator's extraction_rate (format adherence) and default rollout temperature to 0.7. Eval stays unshaped greedy.
…c key Evaluators follow HELMET's convention of eval_fn(prediction, full_record); passing record['item'] crashed travel_planning's checker, and the crash guard scored it 0.0 -- the earlier capability-gate exclusion of travel_planning was OUR bug, not model incapability (reference output scores accuracy 1.0 through the fixed path). countdown/travel_planning return 'accuracy', not 'success'. html_to_tsv accepts both record forms, so the in-flight html runs are unaffected.
… ahead Port of upstream PR awslabs#149 onto the RL branch: - _unpack_from_annotation walks the scatter/cat annotation chain across multi-chunk gaps, parking early-applied intermediate states - grpo_step passes may_match_twice=may_match_twice_flex_attention_sdpa (kept alongside the KV_DEBUG_ANNOTATIONS tracing hook) - regression test for chunk gaps 1/2/4 Unblocks long-generation runs (lm32k, lp*_html).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GRPO/RLOO on LongProc
Driver for the candidate genuinely hard flagship task (per docs/RL_STATUS_REPORT.md): LongProc long procedural generation, integrated as an RL environment.
Why LongProc:
Contents:
examples/grpo_longproc.py— RLOO/GRPO training +--eval-onlyprobe mode; deterministic train/eval partition; reward = each task's primary metric with crash-safe scoring; memory defaults match the validated 7B/48GB recipe (paged 8-bit AdamW, group 4 x 2 accumulated prompts). Verified locally: reference output scores 1.0, garbage scores 0.Setup:
git clone --depth 1 https://github.com/princeton-pli/LongProc.git repos/longproc(data ships in-repo, ~100MB;repos/is gitignored).