Skip to content

fix(metrics): speed up top workflows by runs count query - #3771

Merged
migmartri merged 3 commits into
chainloop-dev:mainfrom
migmartri:miguel/pfm-8118-bug-intermittent-error-code-13-in-sdlc-insights
Nov 3, 2026
Merged

migmartri merged 3 commits into
chainloop-dev:mainfrom
migmartri:miguel/pfm-8118-bug-intermittent-error-code-13-in-sdlc-insights

Conversation

@migmartri

@migmartri migmartri commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Summary

The SDLC Insights "top workflows" metric (OrgMetricsService/TopWorkflowsByRunsCount) had two inefficiencies, visible in production traces for a large organization:

  • The aggregate query reached workflow_runs through the organizations and workflows tables, so it could not use the (organization_id, created_at) index. Its latency varied widely between calls (6ms to 300ms in sampled traces).
  • After aggregating, it loaded every workflow with runs in the time window one by one, plus its project (about 2 queries per workflow, ~160 sequential queries in the sampled traces), and only then kept the top N (at most 20).

This change:

  • Aggregates directly on workflow_runs.organization_id so the existing index drives the scan.
  • Selects the top N workflows before loading them, and loads them with their projects in a single query.
  • No longer counts runs of soft-deleted workflows, which previously could appear in the top list.
  • Breaks ties by workflow ID so the order is deterministic.

Refs PFM-8118. This reduces the latency and variance of one of the SDLC Insights requests. It does not on its own confirm the root cause of the intermittent code 13 errors reported in the ticket.

This change was produced with the assistance of Claude Code.

View guided diff

The top workflows metric reached workflow runs through the organization
and workflows tables, so the aggregate could not use the
(organization_id, created_at) index on workflow_runs. It then loaded
every workflow with runs in the window one by one, plus its project,
before keeping only the top N.

Aggregate on workflow_runs.organization_id directly, pick the top N
workflows first, and load them with their projects in a single query.
Runs of soft-deleted workflows are no longer counted, and ties are
broken by workflow ID so the order is deterministic.

Refs PFM-8118

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: 15511d4e-2aab-40e1-b9b1-9b196874fc95
@chainloop-platform

chainloop-platform Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ✅ 3 passing

Status Policy Material Messages
✅ Passed pr-min-approvals pr-info -
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — 🟡 81% · ⚠️ 2 failing

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟡 81% 2 ⚠️ 2 89% AI / 11% Human 2 +244 / -56 29m23.643s

🟡 78% — 89% AI — ⚠️ 1 policies failing

Nov 2, 2026 22:38 UTC · 28m19s · $16.33 · 288 in / 106.0k out · claude-code 2.1.316 (claude-opus-5-5)

View session details ↗

Change Summary

  • Optimizes OrgMetricsRepo.TopWorkflowsByRunsCount to query by workflow_runs.organization_id and batch workflow loading.
  • Excludes deleted workflows from ranked results and makes top-N ordering deterministic.
  • Adds integration coverage for ranking, project filtering, empty visibility, time-window, and deleted-workflow behavior.

AI Session Overall Score

🟡 78% — Focused fix, but ticket alignment and page-level verification stayed only partial.

AI Session Analysis Breakdown

🟢 95% · scope-discipline

No notes.

🟢 93% · solution-quality

🟢 The fix targets the hot query directly with indexed filtering and batched loading. · High Impact

🟢 84% · user-trust-signal

🟡 The user asked for RBAC and parity rechecks after the PR opened, signaling a brief confidence dip. · Low Severity

🟡 76% · verification

🟢 AI reran old and new implementations instead of stopping at the first passing suite. · High Impact

🟡 67% · context-and-planning

🟢 AI investigated traces first and asked before pivoting to the query-fix path. · High Impact

🟠 The bug hunt became code work without a written plan or explicit ticket-level success criteria. · Medium Severity

💡 When a bug hunt narrows to a likely cause, write a short plan before coding so the success bar stays explicit.

🟡 58% · alignment

🟠 The query fix landed, but neither the diff nor the session confirms the original SDLC Insights code-13 ticket is resolved. · Medium Severity

💡 When you narrow a bug ticket to a suspected cause, keep the ticket open until the user-visible symptom is verified.

🟠 AI summarized before-and-after test parity before it had been verified, then corrected the claim after the user asked. · Medium Severity

💡 Run the explicit before-and-after comparison before summarizing parity; silent subtests are easy to overread.


File Attribution

█████████████████░░░ 89% AI / 11% Human

Status Attribution File Lines
modified ai app/controlplane/pkg/biz/orgmetrics_integration_test.go +162 / -2
modified ai app/controlplane/pkg/data/orgmetrics.go +70 / -49

Policies (4, 1 failing)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-15511d -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-15511d -
⚠️ Failed ai-config-no-secrets ai-coding-session-15511d
  • Secret (generic-password) detected in session content [turn=634, source=assistant-tool_use:Edit, line=1]: {"file_path":"/home/migmartri/work/chainloop/chainloop/.claude/worktrees/temporal-hatching-reef/app/controlplane/pkg/biz/orgmetrics_integration_test.go","new_string":"// OCI credentials used to create...
  • Secret (generic-password) detected in session content [turn=651, source=tool_result, line=4]: 42:var ociTestKeypair = &credentials.OCIKeypair{Repo: ociTestRepo, Username: ociTestUsername, Password: [CHAINLOOP_TRACE_REDACTED:generic-password]
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-15511d -

🟢 84% — 100% AI — ⚠️ 1 policies failing

Nov 2, 2026 22:54 UTC · 14m7s · $1.31 · 81 in / 16.1k out · claude-code 2.1.316 (claude-sonnet-5-5)

View session details ↗

Change Summary

  • Refactors TopWorkflowsByRunsCount to aggregate from workflow_runs, keep deterministic ordering, and exclude soft-deleted workflows in both query stages.
  • Adds orgmetrics integration coverage for deleted workflows, project filters, top-N limits, and empty-result windows.
  • Reuses shared OCI test constants in the integration suite to satisfy lint.

AI Session Overall Score

🟢 84% — Clean session overall; only the new biz integration coverage went unrun.

AI Session Analysis Breakdown

🟢 92% · alignment

No notes.

🟢 91% · solution-quality

🟢 The query filters deleted workflows during aggregation and the later batch load. · High Impact

🟢 90% · scope-discipline

No notes.

🟢 88% · context-and-planning

🟢 The opening task specified scope, workflow, testing, and trailer constraints. · High Impact

🟢 86% · user-trust-signal

No notes.

🟡 65% · verification

🟢 It ran SKIP_INTEGRATION=true go test ./pkg/data/ and observed a pass. · High Impact

🟠 The session ran go test ./pkg/data/, but not the changed biz integration tests. · Medium Severity

💡 When you touch a new test file, run that package tests once before declaring the follow-up done.


File Attribution

████████████████████ 100% AI / 0% Human

Status Attribution File Lines
modified ai app/controlplane/pkg/biz/orgmetrics_integration_test.go +11 / -4
modified ai app/controlplane/pkg/data/orgmetrics.go +1 / -1

Policies (4, 1 failing)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-a527cc -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-a527cc -
⚠️ Failed ai-config-no-secrets ai-coding-session-a527cc Secret (generic-password) detected in session content [turn=177, source=assistant-tool_use:Bash, line=1]: f=pkg/biz/orgmetrics_integration_test.go && sed -i 's/Password: "pass"}/Password: [CHAINLOOP_TRACE_REDACTED:generic-password]; s/"username", "pass", backendType/"username", ociPassword, backendType/' ...
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-a527cc -

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed cwe-top25 -
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@migmartri
migmartri requested a review from a team November 2, 2026 22:49

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread app/controlplane/pkg/data/orgmetrics.go Resolved
…rkflows

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: a527cce1-a4d8-4e3f-9265-6598c30714f5
Fixes the goconst lint failure on the repeated "pass" literal.

Refs PFM-8118

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: 15511d4e-2aab-40e1-b9b1-9b196874fc95
@migmartri
migmartri merged commit c623da0 into chainloop-dev:main Nov 3, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants