Org backfill + discovery scripts (seed and fill the org leaderboard)#104
Merged
Conversation
The request path won't build orgs over 25 members on demand (it records them and moves on). This standalone script fills those recorded-but-empty orgs — and any org named explicitly (google, microsoft, github) — with the same lifetime per-member totals a small org's page produces, writing only the existing org_members + entities tables. No migration, no monthly resolution. Politely paced under GitHub's hourly limit with a reserved floor for live traffic; idempotent, resumable, interruptible. bun scripts/backfill-orgs.ts # every recorded org not yet filled bun scripts/backfill-orgs.ts google microsoft # specific orgs (records them if new)
- No-arg mode now orders by memberCount ascending (was lastFetched), so small orgs resolve fast and mega-orgs like microsoft fall to the back instead of blocking every other org behind one 1,000-member fill. - Fix resume: a member is skipped when its lastFetched is set (from this run OR a previous aborted one), not just when fetched since this run started. So aborting mid-org and re-running now continues from where it stopped rather than re-fetching the whole org. Add --force to re-fetch everyone (refresh).
LOGIN_RE required a login to start AND end alphanumeric, rejecting real legacy
grandfathered accounts that end in a hyphen or contain double hyphens (e.g.
"Link-", "gil--", "p-" — all real accounts created 2010-2013). The regex is only
an injection guard for interpolating the login into GraphQL, so relax it to the
actual GitHub charset+length ([A-Za-z0-9-]{1,39}); quotes/spaces/etc still blocked.
This fixes org backfill (and the request-path org build) aborting on such members.
Also harden backfill-orgs: a per-member 400/404 now records zeros + continues
instead of sinking the whole org; rate limits and 5xx still abort as before.
# Conflicts: # src/lib/github.ts
Prefills the DB with more organizations to look up. GitHub has no endpoint that ranks orgs by public-member count, so this searches type:org by followers (the best prominence proxy — repos-sort returns education/spam orgs), enriches each candidate with fetchOrgProfile to get its real public member count, sorts/filters by that, and records the survivors as entities rows (kind='org', builtAt null). backfill-orgs.ts then fills them. Separate from backfill-orgs.ts by design. --limit / --min-members / --query / --dry-run flags; skips already-tracked orgs; idempotent upsert; paced under the search (30/min) and GraphQL (5k/hr, 500 floor) limits.
peetzweg
changed the title
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two companion operational scripts for populating the organization leaderboard, side by side so they can be tested and merged together. Neither is wired into the app — both are manual, politely rate-limited
bunscripts. No schema change.scripts/discover-orgs.ts— find orgs to trackGitHub has no endpoint that ranks orgs by public-member count, so this searches
type:orgby followers (best prominence proxy), enriches each candidate withfetchOrgProfilefor its real member count, sorts/filters by that, and records the survivors asentitiesrows (kind='org',builtAtnull). Flags:--limit(max 1000),--min-members,--query,--dry-run. Skips already-tracked orgs.scripts/backfill-orgs.ts— fill the ones we trackFills recorded-but-unbuilt orgs (the request path refuses to build >25-member orgs live) with each public member's lifetime org-scoped totals, rolled onto the
entitiesrow. Orders smallest-first so small orgs resolve fast and mega-orgs fall to the back; true cross-run resume (skips members already fetched);--forceto re-refresh.Workflow
Both need the
read:orgtoken from.env(henceenv -u GITHUB_TOKEN), pace under GitHub's limits with a 500-request floor for live traffic, and are idempotent/interruptible. tsc + tests green.(Folds in the discovery script from #117.)