Repository navigation
Prepare Intuitive Software Design 1.3.1 - #1
Merged
Merged
Conversation
…n contract Evals: - Replace prompts that restated the rubric with raw product evidence, so the no-plugin baseline is not handed the graded constraints. - Split each single conjunctive rubric into independently scored criteria. Recovery and validation graders mark examples as illustrative, and a validation result must come from a check with intended users; post-launch aggregates such as support volume only supplement it. - Test eval files: documented grader keys and types, at least three scored criteria per case, explicit PASS/FAIL conditions, and resolvable anchors. Plugin: - Standard 1.3: a product dimension without representative coverage is NE; Behavior Confidence from one observed success path is never representative. - Add the standard's recovery and validation contract (section 41) and point the audit, design, review, and quality checks at it. - Main and purpose-first skills summarize and link the contract (the standard controls); the scoring reference and audit template apply the NE rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…evals Plugin (standard 1.4): - Do not state business rules the evidence does not establish, even in proposed interface copy; use a placeholder and list them for confirmation. - Report every product dimension as NE without representative coverage; no provisional or partial dimension percentages. - Replace the blanket "read the standard for every mode" requirement with mode-specific links to the standard sections each mode requires. Evals: - targeted-improvement-heldout-v2 adds evidence that a rental can change while the extension dialog is open; the failure-state criterion is unchanged. - redesign-routing-heldout-v2 clarifies the analytics grader: a measurable hypothesis passes; a causal claim or guaranteed result fails. - Every case gains an unscored standard-read indicator. Versioned cases must state that their scores are not comparable with earlier versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five new cases with 24 scored criteria, written by a fresh-context author that saw no plugin instructions, earlier answers, or graders. Includes the pre-registered analysis plan, analysis tooling with regression checks, the blind-reviewer prompt, and a SHA-256 freeze manifest. Nothing has been run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The evals/ cases shaped the plugin's development, so the README no longer calls them held-out. It now identifies evals-holdout/ as the frozen validation suite for candidate 43f042d and records its v1 result: no improvement was demonstrated on those cases, with same-family grading and no human audit as limitations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical and moderate issues remain in CI validation, frozen-suite metadata, and analysis integrity.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
Open (4)
What changed in this PR
Prepares Intuitive Software Design 1.3.1 with stronger evidence boundaries, recovery guidance, Behavior Confidence scoring, evaluation tooling, and package validation.
Changes:
- Refines skills, standards, scoring, and audit requirements.
- Adds regression and frozen holdout evaluation cases and tooling.
- Updates manifests, marketplace metadata, licensing, documentation, and CI validation.
| File | Description |
|---|---|
tests/test_plugin.py |
Expands plugin and evaluation contract checks. |
skills/purpose-first-redesign/SKILL.md |
Adds evidence, recovery, and validation guidance. |
skills/purpose-first-redesign/agents/openai.yaml |
Updates the default redesign prompt. |
skills/intuitive-software-design/SKILL.md |
Refines mode routing and recovery guidance. |
skills/intuitive-software-design/references/scoring-reference.md |
Adds Behavior Confidence scoring. |
skills/intuitive-software-design/references/intuitive-software-design-standard.md |
Updates normative standards and validation rules. |
skills/intuitive-software-design/references/audit-template.md |
Adds interaction-level audit evidence. |
skills/behavioral-evaluation/references/case-index.md |
Documents evaluation case status and limitations. |
README.md |
Documents evaluation, validation, licensing, and marketplace use. |
NOTICE |
Documents artwork licensing. |
evals/targeted-improvement-heldout-v2/prompt.md |
Adds targeted-improvement evaluation input. |
evals/targeted-improvement-heldout-v2/graders/validation.md |
Grades validation quality. |
evals/targeted-improvement-heldout-v2/graders/standard-read.md |
Tracks standard reads. |
evals/targeted-improvement-heldout-v2/graders/skill-read.md |
Tracks skill reads. |
evals/targeted-improvement-heldout-v2/graders/skill-invoked.md |
Tracks skill invocation. |
evals/targeted-improvement-heldout-v2/graders/focused-correction.md |
Grades proportional correction. |
evals/targeted-improvement-heldout-v2/graders/failure-states.md |
Grades failure-state handling. |
evals/targeted-improvement-heldout-v2/graders/evidence-boundaries.md |
Grades evidence limits. |
evals/targeted-improvement-heldout-v2/graders/diagnosis.md |
Grades evidence-grounded diagnosis. |
evals/redesign-routing-heldout-v2/prompt.md |
Adds redesign-routing evaluation input. |
evals/redesign-routing-heldout-v2/graders/validation.md |
Grades redesign validation. |
evals/redesign-routing-heldout-v2/graders/standard-read.md |
Tracks standard reads. |
evals/redesign-routing-heldout-v2/graders/skill-read.md |
Tracks skill reads. |
evals/redesign-routing-heldout-v2/graders/skill-invoked.md |
Tracks skill invocation. |
evals/redesign-routing-heldout-v2/graders/recovery-states.md |
Grades recovery-state coverage. |
evals/redesign-routing-heldout-v2/graders/no-implementation.md |
Prevents implementation output. |
evals/redesign-routing-heldout-v2/graders/goal-first.md |
Grades goal-first routing. |
evals/redesign-routing-heldout-v2/graders/business-rules.md |
Grades business-rule accuracy. |
evals/redesign-routing-heldout-v2/graders/analytics-evidence.md |
Grades analytics evidence boundaries. |
evals/redesign-routing-heldout-v2/graders/accessibility.md |
Grades accessibility requirements. |
evals/behavior-confidence-heldout/prompt.md |
Adds Behavior Confidence evaluation input. |
evals/behavior-confidence-heldout/graders/vocabulary.md |
Grades Behavior Confidence vocabulary. |
evals/behavior-confidence-heldout/graders/unobserved-states.md |
Grades unobserved-state identification. |
evals/behavior-confidence-heldout/graders/standard-read.md |
Tracks standard reads. |
evals/behavior-confidence-heldout/graders/skill-read.md |
Tracks skill reads. |
evals/behavior-confidence-heldout/graders/skill-invoked.md |
Tracks skill invocation. |
evals/behavior-confidence-heldout/graders/score-integrity.md |
Grades score integrity. |
evals/behavior-confidence-heldout/graders/observed-assessment.md |
Grades observed-behavior assessment. |
evals/behavior-confidence-heldout/graders/evidence-plan.md |
Grades evidence planning. |
evals/behavior-confidence-heldout/graders/claims-vs-evidence.md |
Grades claims and evidence separation. |
evals-holdout/storage-downgrade/prompt.md |
Adds storage-downgrade holdout input. |
evals-holdout/storage-downgrade/graders/standard-read.md |
Tracks standard reads. |
evals-holdout/storage-downgrade/graders/skill-invoked.md |
Tracks skill invocation. |
evals-holdout/storage-downgrade/graders/no-invented-backup-restoration-policy.md |
Grades unsupported recovery claims. |
evals-holdout/storage-downgrade/graders/no-dark-patterns.md |
Grades downgrade friction. |
evals-holdout/storage-downgrade/graders/mitigation-export-option.md |
Grades preservation options. |
evals-holdout/storage-downgrade/graders/irreversible-deletion-disclosure.md |
Grades deletion disclosure. |
evals-holdout/storage-downgrade/graders/impact-scope-visibility.md |
Grades impact visibility. |
evals-holdout/school-trip-consent/prompt.md |
Adds consent-flow holdout input. |
evals-holdout/school-trip-consent/graders/trip-type-consent-threshold.md |
Grades consent thresholds. |
evals-holdout/school-trip-consent/graders/standard-read.md |
Tracks standard reads. |
evals-holdout/school-trip-consent/graders/split-household-routing.md |
Grades split-household routing. |
evals-holdout/school-trip-consent/graders/skill-invoked.md |
Tracks skill invocation. |
evals-holdout/school-trip-consent/graders/no-invented-revocation-rule.md |
Grades policy uncertainty handling. |
evals-holdout/school-trip-consent/graders/medical-note-tiered-access.md |
Grades medical-note privacy. |
evals-holdout/school-trip-consent/graders/consent-vs-visibility.md |
Grades consent and visibility separation. |
evals-holdout/meal-planner-sync/prompt.md |
Adds synchronization holdout input. |
evals-holdout/meal-planner-sync/graders/standard-read.md |
Tracks standard reads. |
evals-holdout/meal-planner-sync/graders/skill-invoked.md |
Tracks skill invocation. |
evals-holdout/meal-planner-sync/graders/proportionate-scope.md |
Grades proportionality. |
evals-holdout/meal-planner-sync/graders/preserve-private-notes.md |
Grades privacy preservation. |
evals-holdout/meal-planner-sync/graders/no-silent-conflict-loss.md |
Grades conflict handling. |
evals-holdout/meal-planner-sync/graders/no-network-blame-only.md |
Grades root-cause diagnosis. |
evals-holdout/meal-planner-sync/graders/honest-save-vs-sync-status.md |
Grades save and sync status clarity. |
evals-holdout/FREEZE.sha256 |
Freezes holdout-suite hashes. |
evals-holdout/expense-form-wizard/prompt.md |
Adds expense-form holdout input. |
evals-holdout/expense-form-wizard/graders/targeted-receipt-visibility-fix.md |
Grades receipt-visibility fixes. |
evals-holdout/expense-form-wizard/graders/standard-read.md |
Tracks standard reads. |
evals-holdout/expense-form-wizard/graders/skill-invoked.md |
Tracks skill invocation. |
evals-holdout/expense-form-wizard/graders/proportionate-change-scope.md |
Grades proportionality. |
evals-holdout/expense-form-wizard/graders/preserve-finance-policy-constraints.md |
Grades policy preservation. |
evals-holdout/expense-form-wizard/graders/data-grounded-wizard-recommendation.md |
Grades data-grounded recommendations. |
evals-holdout/expense-form-wizard/graders/correctly-scope-reimbursement-timing-issue.md |
Grades issue scoping. |
evals-holdout/AUTHORING.md |
Documents holdout authoring. |
evals-holdout/AUTHOR-NOTES.md |
Records case rationale and revisions. |
evals-holdout/analysis/test_analyze.py |
Adds analysis regression checks. |
evals-holdout/analysis/REVIEWER-PROMPT.md |
Defines blind-review requirements. |
evals-holdout/analysis/analyze.py |
Implements holdout analysis and review tooling. |
evals-holdout/ANALYSIS-PLAN.md |
Defines frozen-suite methodology. |
evals-holdout/address-change-review/prompt.md |
Adds address-review holdout input. |
evals-holdout/address-change-review/graders/standard-read.md |
Tracks standard reads. |
evals-holdout/address-change-review/graders/skill-invoked.md |
Tracks skill invocation. |
evals-holdout/address-change-review/graders/flag-dispatched-order-scoping.md |
Grades business-rule scoping. |
evals-holdout/address-change-review/graders/flag-contradictory-states.md |
Grades contradictory-state detection. |
evals-holdout/address-change-review/graders/distinguish-screenshot-from-runtime.md |
Grades runtime evidence boundaries. |
evals-holdout/address-change-review/graders/accessibility-grounded-in-visible.md |
Grades screenshot-grounded accessibility observations. |
assets/LICENSE |
Adds artwork license terms. |
.gitignore |
Ignores local evaluation results. |
.github/workflows/validate.yml |
Adds CI and package validation. |
.codex-plugin/plugin.json |
Updates Codex metadata. |
.claude-plugin/plugin.json |
Updates Claude plugin metadata. |
.claude-plugin/marketplace.json |
Adds marketplace metadata. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+305
to
+309
| source, target = run.get("tracePath"), transcript_path(results_dir, case["name"], arm, n) | ||
| if source and os.path.exists(source) and os.path.abspath(source) != os.path.abspath(target): | ||
| os.makedirs(os.path.dirname(target), exist_ok=True) | ||
| shutil.copyfile(source, target) | ||
| index.append(f"{case['name']}\t{arm}\t{n}\t{source}\t{sha256(target) if os.path.exists(target) else 'MISSING'}") |
Comment on lines
+480
to
+482
| sealed = read_json(os.path.join(review_dir, "key", "SEALED-KEY.json")) | ||
| if sealed.get("eligible") != [list(k) for k in eligible]: | ||
| raise SystemExit("The eligible runs differ from those exported for review; stop and investigate before reviewing.") |
Comment on lines
+648
to
+652
| def review(results_dir, review_dir): | ||
| """Returns False, and writes no conclusions, when any reviewer output fails validation.""" | ||
| agg = load(results_dir) | ||
| sealed = load_key(results_dir, review_dir, agg) | ||
| verdicts, errors, problems = validate_review(agg, sealed, review_dir) |
| - name: Validate Claude plugin and marketplace | ||
| run: | | ||
| claude plugin validate --strict . | ||
| claude plugin validate --strict .claude-plugin/plugin.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The plugin's review and audit guidance needed clearer evidence limits, distinct Behavior Confidence scoring, and complete recovery and validation requirements. Version 1.3.1 adds those rules, directs agents to the standard sections for their chosen mode, and improves broad-redesign routing while keeping focused improvements proportional to the diagnosed problem.
This update also repairs evaluation routing checks, adds regression and frozen validation tooling, configures package validation in CI, and aligns both plugin manifests, artwork licensing, and the self-hosted Claude marketplace entry. The README distinguishes development cases from the frozen validation suite and records its result: no improvement was demonstrated on these cases. Raw evaluation reports and transcripts remain local and gitignored.
New blind reviews use
scripts/review_integrity.py, which seals aggregate, transcript, packet, and key hashes at export and rejects drift before sampling or reporting. It also checks saved reviewer transcripts for tool use and matching successful outputs. The original v1 snapshot remains unchanged for historical provenance; current documentation directs new reviews to the guarded entry point. Hashes detect accidental drift, not deliberate replacement of evidence and hashes together.Validation at commit
a39efa1:The evaluation's same-family grading, non-discriminating cases, and lack of a human audit limit its conclusions. This PR does not submit or publish a marketplace update.