feat(test): add manual Qwen 3.6 production qualification - #636
Conversation
Graffioh
changed the title
264b8b9 to
243397a
Compare
Graffioh
changed the title
243397a to
79ac7ab
Compare
Graffioh
changed the title
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 43 files
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
Re-trigger cubic
79ac7ab to
869f1c9
Compare
There was a problem hiding this comment.
1 existing issue remains and no new issues found across 32 files
Requires human review: Auto-approval blocked because this review re-detected 6 unresolved issues already reported by Cubic.
Re-trigger cubic
a9970d3 to
419e72e
Compare
Graffioh
changed the title
419e72e to
96f5c87
Compare
Graffioh
changed the title
|
@cubic-dev-ai review this PR |
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
96f5c87 to
9f67122
Compare
There was a problem hiding this comment.
Review completed against the latest diff
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
9f67122 to
9794f0e
Compare
Graffioh
changed the title
9794f0e to
925e204
Compare
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 18 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
ad1e25f to
e67ea59
Compare
Graffioh
changed the title
e67ea59 to
d9fdcd8
Compare
Graffioh
changed the title
There was a problem hiding this comment.
All reported issues were addressed across 21 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
d9fdcd8 to
7e8baac
Compare
|
@cubic-dev-ai review this PR |
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 21 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 21 files
Requires human review: Auto-approval blocked by 11 unresolved issues from a previous review of this commit.
Re-trigger cubic
7e8baac to
7a61ea0
Compare
|
@cubic-dev-ai review this PR |
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 21 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
7a61ea0 to
e2af84d
Compare
|
@cubic-dev-ai review this PR |
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 21 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
e2af84d to
af2cfc6
Compare
|
@cubic-dev-ai review this PR |
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai review this PR |
There was a problem hiding this comment.
No issues found across 21 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 21 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
af2cfc6 to
b79419d
Compare
|
@cubic-dev-ai review this PR |
@Graffioh I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
4 issues found across 21 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="harness/qualification/qwen36/qualify.py">
<violation number="1" location="harness/qualification/qwen36/qualify.py:102">
P2: When a stale PID file points to a reused PID, cleanup can kill an unrelated process or process group. Store and verify the launched process identity before sending signals.</violation>
<violation number="2" location="harness/qualification/qwen36/qualify.py:173">
P3: With a monitor interval below six seconds, a slow telemetry sample can outlive the 10-second join timeout and turn cleanup into a qualification failure. Use a join timeout longer than the sample's bounded 11-second execution time.</violation>
<violation number="3" location="harness/qualification/qwen36/qualify.py:323">
P1: When a retained generation report omits prompt cases, this validator accepts the subset because it trusts the report’s own case count. Require the submitted case IDs and count to match the canonical corpus before allowing generation evidence to pass.</violation>
<violation number="4" location="harness/qualification/qwen36/qualify.py:348">
P2: When `gold_accuracy` is `NaN` or negative infinity, the threshold comparison evaluates as passing. Reject non-finite and boolean accuracy values before applying the quality threshold.</violation>
</file>
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
| if not isinstance(summary, dict) or summary.get("status") != "pass": | ||
| raise ValueError(f"{path.name}: generation verdict did not pass") | ||
| cases = report.get("cases") | ||
| if not isinstance(cases, list) or not cases: |
There was a problem hiding this comment.
P1: When a retained generation report omits prompt cases, this validator accepts the subset because it trusts the report’s own case count. Require the submitted case IDs and count to match the canonical corpus before allowing generation evidence to pass.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/qualification/qwen36/qualify.py, line 323:
<comment>When a retained generation report omits prompt cases, this validator accepts the subset because it trusts the report’s own case count. Require the submitted case IDs and count to match the canonical corpus before allowing generation evidence to pass.</comment>
<file context>
@@ -0,0 +1,907 @@
+ if not isinstance(summary, dict) or summary.get("status") != "pass":
+ raise ValueError(f"{path.name}: generation verdict did not pass")
+ cases = report.get("cases")
+ if not isinstance(cases, list) or not cases:
+ raise ValueError(f"{path.name}: generation cases are missing")
+ if summary.get("cases") != len(cases):
</file context>
| if pid_file is None or not pid_file.is_file(): | ||
| return {} | ||
| try: | ||
| pid = int(pid_file.read_text().strip()) |
There was a problem hiding this comment.
P2: When a stale PID file points to a reused PID, cleanup can kill an unrelated process or process group. Store and verify the launched process identity before sending signals.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/qualification/qwen36/qualify.py, line 102:
<comment>When a stale PID file points to a reused PID, cleanup can kill an unrelated process or process group. Store and verify the launched process identity before sending signals.</comment>
<file context>
@@ -0,0 +1,907 @@
+ if pid_file is None or not pid_file.is_file():
+ return {}
+ try:
+ pid = int(pid_file.read_text().strip())
+ values: dict[str, Any] = {"pid": pid}
+ for line in Path(f"/proc/{pid}/status").read_text().splitlines():
</file context>
| if summary.get("min_gold_accuracy") != minimum_gold_accuracy: | ||
| raise ValueError(f"{path.name}: quality threshold differs from the profile") | ||
| accuracy = summary.get("gold_accuracy") | ||
| if not isinstance(accuracy, int | float) or accuracy < minimum_gold_accuracy: |
There was a problem hiding this comment.
P2: When gold_accuracy is NaN or negative infinity, the threshold comparison evaluates as passing. Reject non-finite and boolean accuracy values before applying the quality threshold.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/qualification/qwen36/qualify.py, line 348:
<comment>When `gold_accuracy` is `NaN` or negative infinity, the threshold comparison evaluates as passing. Reject non-finite and boolean accuracy values before applying the quality threshold.</comment>
<file context>
@@ -0,0 +1,907 @@
+ if summary.get("min_gold_accuracy") != minimum_gold_accuracy:
+ raise ValueError(f"{path.name}: quality threshold differs from the profile")
+ accuracy = summary.get("gold_accuracy")
+ if not isinstance(accuracy, int | float) or accuracy < minimum_gold_accuracy:
+ raise ValueError(f"{path.name}: quality score did not pass")
+ if summary.get("gold_scored") != len(cases):
</file context>
|
|
||
| def stop(self) -> None: | ||
| self.stop_event.set() | ||
| self.thread.join(timeout=max(10.0, self.interval + 5.0)) |
There was a problem hiding this comment.
P3: With a monitor interval below six seconds, a slow telemetry sample can outlive the 10-second join timeout and turn cleanup into a qualification failure. Use a join timeout longer than the sample's bounded 11-second execution time.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/qualification/qwen36/qualify.py, line 173:
<comment>With a monitor interval below six seconds, a slow telemetry sample can outlive the 10-second join timeout and turn cleanup into a qualification failure. Use a join timeout longer than the sample's bounded 11-second execution time.</comment>
<file context>
@@ -0,0 +1,907 @@
+
+ def stop(self) -> None:
+ self.stop_event.set()
+ self.thread.join(timeout=max(10.0, self.interval + 5.0))
+
+ if self.thread.is_alive():
</file context>
| self.thread.join(timeout=max(10.0, self.interval + 5.0)) | |
| self.thread.join(timeout=max(15.0, self.interval + 5.0)) |
2 participants
Summary
harness/qualification/qwen36Scope
The runner checks the model artifact and hardware identity, builds and starts one long-lived server, then runs correctness, bounded quality, determinism, concurrency, performance capture, and resource-drift stages.
R5 records repeated c1 and c4 performance samples. It does not apply a baseline pass or fail threshold yet. R7 cross-checks the candidate samples against the retained raw reports and validates
/propsagainst the selected model profile.This PR does not add the dashboard, performance-history system, checked-in baselines, or an automatic model-backed GPU job. PR #687 owns kernel-level correctness and route-witness coverage.
Verification
Two process-group signal tests cannot run on the current macOS host because the sandbox blocks the required signals. They remain in the hosted Linux CI suite.
Left before production sign-off
The PR remains a draft until the model-backed runs complete.