docs(triage): CumulusCI v5 issue triage evidence pack + xfail regression tests#3979
Draft
jstvz wants to merge 4 commits into
Draft
docs(triage): CumulusCI v5 issue triage evidence pack + xfail regression tests#3979jstvz wants to merge 4 commits into
jstvz wants to merge 4 commits into
Conversation
Pushable evidence from a triage of 142 open issues in
SFDO-Tooling/CumulusCI:
- README.md: methodology, results table, AI-assistance disclosure,
spec basis, Tranche 1 candidate list
- proposals.md: per-issue classification + proposed pass-1 action
for all 142 issues
- repro-results.{md,csv}: narrative + machine-readable evidence for
the 115 issues that went through the reproducibility pass
(80 REPRODUCED, 23 NOT-REPRODUCED, 11 INCONCLUSIVE, 1 duplicate)
- themes.md: 12 theme clusters, duplicate detection, Tranche 1 list
Phase 1 GitHub state mutations (gh issue close / label) are NOT
executed by this PR. Maintainer approval gate applies.
Adds cumulusci/tests/triage/ with 73 xfail tests, one per REPRODUCED-on-dev or REPRODUCED-on-v4.10.0 issue with a code-level assertion target: - 10 from Round 3 (themes covered: keychain, robotframework, docs, python-modernization, auth, plus stragglers) - 63 from Wave A retroactive recovery (all R1+R2 REPRODUCED issues with code-level repros, split across recover-1..4 subagents) Run results: 79 xfailed, 0 xpassed (3 non-xfail "control" tests emitted by some subagents were removed during harvest to keep the directory xfail-only per cumulusci/tests/triage/README.md). Test convention: - @pytest.mark.xfail(strict=False, reason="repro for #NNNN — ...") - Each captures a code-level assertion that the fix-PR will flip green - strict=False means an independent fix surfaces as XPASS, not a CI failure - 10 R3 test files had stale path refs (docs/triage/2026-05-cci-backlog/) normalized to docs/triage/v5/ README at cumulusci/tests/triage/README.md documents the convention, lifecycle, and cross-refs to docs/triage/v5/. Plan reference: docs/superpowers/plans/2026-05-14-cci-triage-upstream-plan.md Task 7 (harvest xfail tests).
For each REPRODUCED issue not in Tranche 1 (75 of 80), add a per-issue fix-sketch at docs/triage/v5/fix-sketches/issue_NNNN.md. Each sketch links to: - The full narrative in docs/triage/v5/repro-results.md - The xfail regression test in cumulusci/tests/triage/test_issue_NNNN.py - The GitHub issue Sketches extract from the underlying narrative: - Bug summary (from Evidence / Classification / Reproduction path) - Target file:line (from fenced code-block locator or Target field) - Recommended approach (from Recommended action / Proposed fix sketch) The Size/risk table is left as TBD-by-fix-PR-author. Tranche 1 (#3852, #3854, #3886, #3938, #3939) does NOT get a fix-sketch because those receive immediate fix PRs (Task 9). #3610 and #3910 are fix-already-on-dev and get a closed:pr-resolved-NNNN pass-1 verdict rather than a fix-sketch. Plan reference: docs/superpowers/plans/2026-05-14-cci-triage-upstream-plan.md Task 6 (build fix-sketches/).
Cleans the evidence pack and harvested xfail tests for external review: - Strip /tmp/repro/ session paths; replace with cumulusci/tests/triage/ test paths when the harvested test exists, else neutral placeholder. - Strip internal tracking IDs (Salesforce GUS work-item numbers) from narratives and notes. - Strip per-pass coordination scaffolding (round/subagent/wave headers and per-subagent summary chunks reduced to neutral prose). - Strip scratch-org alias names and DevHub alias from narratives. - Replace all em-dashes / en-dashes with ASCII colons or hyphens. - Cross-link Tranche 1 references between proposals.md, themes.md, and README.md. - Tighten AI assistance disclosure to omit specific model name and references to local-only intermediate artifacts. No semantic changes to verdicts, recommendations, or xfail markers. Test suite remains 79 xfailed / 0 xpassed.
This was referenced May 14, 2026
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Snapshot of an AI-assisted triage of the 142 open issues in this repository, preparing the backlog for CumulusCI v5. This is the umbrella PR; five small fix PRs target individual recent regressions.
docs/triage/v5/: classification matrix (proposals.md), reproducibility-pass narrative + CSV (repro-results.md/.csv), theme clustering and duplicate detection (themes.md), and one fix-sketch perREPRODUCEDissue not slated for an immediate fix (fix-sketches/issue_NNNN.md).cumulusci/tests/triage/: 79@pytest.mark.xfail(strict=False)regression tests, one perREPRODUCEDissue with a code-level assertion target. Non-strict so a bug that resolves independently surfaces asXPASSrather than failing CI. Fix-PRs remove the marker.No GitHub state mutations against the 142 open issues are executed by this PR; pass-1 close/label proposals are gated on explicit maintainer approval and recorded in
proposals.md.Reproducibility verdicts (115 of 142 issues)
REPRODUCED-on-v4.10.0REPRODUCED-on-devNOT-REPRODUCED-on-v4.10.0NOT-REPRODUCED-on-devINCONCLUSIVE-needs-*(kept-open)closed:duplicate-of-#3544The 27
closed:pre-v4.0.0issues were excluded from the repro pass (they predate v4.0 and are tagged separately for close).Companion fix PRs
The five
REPRODUCEDrecent regressions slated for immediate fix-PRs againstdev:triage/fix-tranche1-3852triage/fix-tranche1-3854triage/fix-tranche1-3886triage/fix-tranche1-3938triage/fix-tranche1-3939Each fix PR removes the xfail marker on its corresponding
cumulusci/tests/triage/test_issue_NNNN.pyand lands the fix as a separate commit.Test plan
uv run pytest cumulusci/tests/triage/ -qreports 79 xfailed / 0 xpassed (no surprise resolutions).proposals.mdpass-1 recommendations class-by-class; no labels or closes are auto-applied.Provenance
Methodology, AI-assistance disclosure, and pass-1 vocabulary are documented in
docs/triage/v5/README.md.