Skip to content

fix(orb): index orb_relay_pending.created_at to stop full table scans#7510

Merged
JSONbored merged 1 commit into
mainfrom
fix/orb-relay-pending-created-at-index
Jul 20, 2026
Merged

fix(orb): index orb_relay_pending.created_at to stop full table scans#7510
JSONbored merged 1 commit into
mainfrom
fix/orb-relay-pending-created-at-index

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

orb-relay-drain's periodic pull-mode drain has been timing out since 2026-07-15 and is still recurring (Sentry orb_relay_drain/TimeoutError, 13+ occurrences, most recent within hours). Its handler unconditionally runs pruneRelayPending first on every 30s pull. That function issues a SELECT and a DELETE against orb_relay_pending filtered only by created_at — but both existing indexes on that table lead with installation_id, so neither query could use an index. Both fell back to a full table scan, fleet-wide, every 30 seconds, occasionally exceeding the pull request's own 30s timeout budget.

This adds the missing created_at index (migration 0167, mirrored in src/db/schema.ts) so both queries become index range scans instead.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Closes #7430

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally — new regression test (test/unit/orb-relay-pending-created-at-index.test.ts) asserts EXPLAIN QUERY PLAN uses the new index (SEARCH, not SCAN) for both queries pruneRelayPending runs; npm run db:migrations:check and npm run db:schema-drift:check both clean (0167 confirmed as the next free migration number)
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • apps/loopover-ui's ui:test (vitest/jsdom) fails locally with window.localStorage undefined, on this sandbox's Node v26.5.0 (the repo pins Node 22 via .nvmrc, unavailable via nvm in this environment). Verified this reproduces identically on a clean, unmodified origin/main checkout with a fresh npm ci — it is a pre-existing local Node-version artifact unrelated to this change (which touches no apps/loopover-ui files), not a regression. Confirmed real GitHub Actions CI (validate-tests shards 1–6 + merge) is green on the exact main commit this branches from.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS surface touched.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no API/OpenAPI/MCP surface touched.)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — no UI changes.)
  • Visible UI changes include a UI Evidence section below with screenshots. (N/A — no visible UI change; this is a DB index + migration only.)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (N/A — internal query-plan fix, no user-facing behavior or doc change.)

Notes

  • Root-caused via Sentry (orb_relay_drain) triage: pruneRelayPending's SELECT was added by a same-day commit on 2026-07-15, doubling the pre-existing unindexed DELETE's scan work — matching the exact day this Sentry issue started firing.

…#7430)

pruneRelayPending's SELECT and DELETE filter only by created_at, but both
existing indexes on orb_relay_pending lead with installation_id -- neither
could serve that predicate, so both fell back to a full table scan, run
fleet-wide on every 30s orb-relay-drain pull, occasionally exceeding the
pull request's own 30s timeout budget (Sentry: orb_relay_drain TimeoutError,
recurring since 2026-07-15).

Adds the missing created_at index (migration 0167, mirrored in the Drizzle
schema) so both queries become index range scans.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 20, 2026
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.35%. Comparing base (25decd9) to head (b0cc806).
⚠️ Report is 36 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #7510   +/-   ##
=======================================
  Coverage   91.35%   91.35%           
=======================================
  Files         716      716           
  Lines       72990    72990           
  Branches    21628    21627    -1     
=======================================
  Hits        66678    66678           
  Misses       5272     5272           
  Partials     1040     1040           
Flag Coverage Δ
shard-1 33.69% <ø> (+1.01%) ⬆️
shard-2 40.33% <ø> (+4.21%) ⬆️
shard-3 28.76% <ø> (-5.77%) ⬇️
shard-4 39.57% <ø> (-3.32%) ⬇️
shard-5 35.72% <ø> (-1.25%) ⬇️
shard-6 34.83% <ø> (+1.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/db/schema.ts 72.97% <ø> (ø)

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 20, 2026
@JSONbored
JSONbored merged commit 88a2f61 into main Jul 20, 2026
18 checks passed
@JSONbored
JSONbored deleted the fix/orb-relay-pending-created-at-index branch July 20, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(orb): orb-relay-drain pull times out — pruneRelayPending full-scans orb_relay_pending

1 participant