Repository navigation
fix(release-autopilot-onboarding): address #229 review follow-ups - #237
Merged
Merged
Conversation
… and mapping, clear opt-out overrides Follow-ups from review on #229: - Bring your own key, auto-cleanup, and auto-archival change settings in the browser, so they now happen only after 'Looks good'; Preview lists them and never asks for them. - GitHub runs only for auto-flagging, auto-releasing, or auto-cleanup, and repo mapping only for auto-flagging or auto-releasing; auto-archival alone picks projects directly. - A picked capability that a repo override keeps off is turned back on, and verification checks the effective value. - Evals: mappings must name the exact project, the policy case must take the 403-then-policies path, and plan mode must write the repo mapping. New cases for archival-only, preview with cleanup, and the repo override. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
review Step 8 now sends enabled:true for a picked capability a repo override kept off (unless it cannot work yet), and Step 7 describes that write. Preview with GitHub disconnected lists mapping as a later step instead of contradicting the no-mapping-without-GitHub rule. Resume re-lists pending browser steps after the confirmation. Evals detect browser-step and repo-picker questions by content, not only by link, require the override case to read the repo back, and add a bring-your-own-key case (24 total). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
vroske-ld
reviewed
Oct 30, 2026
- Browser steps count as approved only after a 'Looks good' answer, not after the confirmation question is merely shown. - Release setup lookups must name the requested project and one of its environments; the check accepts either path, release policies (403 then a policy lookup for the project) or environment release plans, so it keeps working at GA. - Preview with GitHub disconnected and auto-releasing picked also lists the deferred release setup (environments and release policies or plans) after repo mapping. New eval case (25 total). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vroske-ld
reviewed
Oct 31, 2026
approvedConfirmBefore walked back past a 'Change something' answer to an older 'Looks good', so a withdrawn approval still passed. It now inspects only the latest confirmation before the browser step or write, and confirmsBeforeWriting uses the same helper. Addresses review on #237. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the release-plans path, every lookup must target an environment the case selected, and each selected environment needs a successful lookup (the plan-mode case selects Production; a Test lookup no longer passes). The single account-wide call stays only on the 403 release-policies path. Addresses review on #237. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vroske-ld
approved these changes
Nov 2, 2026
fsaresh
marked this pull request as ready for review
November 2, 2026 19:10
The hold-intent cases held production until 2026-09-01 and 2026-10-01. Both dates have passed, so the agent correctly treats the hold as over and releases production, and the 'Single-target date hold' case fails on every run (main's nightly eval has failed since 2026-10-30). Use 2099 dates so the hold is always in the future. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fsaresh
commented
Nov 2, 2026
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3cd2b63. Configure here.
…ck; Bugbot follow-ups flag-release: its cases name dev, staging, and production, but the shared get-environments mock has only test and production, so an agent that checked environments mapped staging to test or stopped to ask. The suite now loads its own environments mock via tool_mocks; 15/15 local runs pass. Onboarding (Bugbot on #237): the no-writes edge case now keeps browser-only plans going through confirmation and Step 8, and Preview with GitHub disconnected lists repo mapping only for auto-flagging or auto-releasing (auto-cleanup alone picks projects). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups to the review on #229.
Skill (
launchdarkly-release-autopilot-onboarding0.3.0-experimental)enabled: true), unless it cannot work yet, and Step 9 checks the effective value before calling the setup complete.Evals (
evals/release-autopilot-onboarding, 25 cases)projectKey.acme/demo→defaultmapping after the release plan.Unrelated fixes included to unblock CI (
evals/flag-release)dev,staging, andproduction, but the sharedget-environmentsmock has onlytestandproduction, so an agent that checked environments first mappedstagingtotestor stopped to ask. The suite now loads its own environments mock viatool_mocks; other suites are unchanged.Testing
python3 scripts/validate_skills.pyandtests/test_*.pypasscd evals && npm test: 125 passnpm run eval:release-autopilot-onboarding: 25/25 pass; the new and changed cases passed three runs each🤖 Generated with Claude Code
via LD Research 🤖
Note
Overview
Bumps
launchdarkly-release-autopilot-onboardingto 0.3.0-experimental with wizard flow fixes from #229 review: settings-changing browser work (bring-your-own-key, auto-cleanup, auto-archival) moves to Step 8 after "Looks good"; Preview lists those steps (and deferred GitHub mapping / release setup) instead of prompting in the browser. GitHub and repo mapping run only when the picked capabilities need them; auto-archival-only skips GitHub, mapping, and the AI model step. Repo overrides that block a picked capability are cleared withenabled: trueand re-verified in Step 9.The release-autopilot-onboarding eval suite grows to 25 cases with stricter assertions: latest "Looks good" before writes or browser steps, release setup lookups scoped to the right project and environments (policy vs plan paths), plan-before-repo for
acme/demo, and new scenarios (archival-only, preview cleanup, repo override, BYOK, preview without GitHub).CI unblock for
flag-releaseevals: production hold dates move from 2026 to 2099 so holds stay in the future, and the suite gets a dedicatedget-environmentsmock with dev, staging, and production instead of the shared mock that lacked staging.Reviewed by Cursor Bugbot for commit 315f72a. Bugbot is set up for automated code reviews on this repo. Configure here.