Description
while comparing the wakatime cron route with the newer /api/cron/sync one i noticed the wakatime one still does an unbounded fetch of the entire users table.
The Problem
src/app/api/wakatime/sync/route.ts:13-17:
const { data: users, error } = await supabaseAdmin
.from("users")
.select("id, wakatime_api_key_encrypted, wakatime_api_key_iv")
.not("wakatime_api_key_encrypted", "is", null)
.not("wakatime_api_key_iv", "is", null);
no .range(), no .limit(). every wakatime user in the DB (plus their encrypted keys) is pulled into serverless memory in one shot. line 29 then loops through users.length in chunks of 5.
failure scenario: on a real deployment with a few thousand wakatime users this either hits the vercel 4.5mb response limit from supabase, blows the 10s/60s function timeout, or gets killed by the runtime memory cap and the entire nightly sync silently fails. same class of bug that was already fixed in src/app/api/cron/sync/route.ts:38-128 which pages with PAGE_SIZE = 50 and a while (hasMore) loop.
Proposed Fix
- adopt the same pagination pattern as cron/sync: page through 50 users at a time using
.range(page * PAGE_SIZE, (page + 1) * PAGE_SIZE - 1)
- break out of the while loop when the returned page is smaller than
PAGE_SIZE
- keep the existing
Promise.allSettled chunk-of-5 inside each page
- also add
.order("id") so pagination is deterministic
I would like to work on this under GSSoC 26. please assign!
for labels i think gssoc + level2 + type:performance fits here.
Description
while comparing the wakatime cron route with the newer /api/cron/sync one i noticed the wakatime one still does an unbounded fetch of the entire users table.
The Problem
src/app/api/wakatime/sync/route.ts:13-17:
no
.range(), no.limit(). every wakatime user in the DB (plus their encrypted keys) is pulled into serverless memory in one shot. line 29 then loops throughusers.lengthin chunks of 5.failure scenario: on a real deployment with a few thousand wakatime users this either hits the vercel 4.5mb response limit from supabase, blows the 10s/60s function timeout, or gets killed by the runtime memory cap and the entire nightly sync silently fails. same class of bug that was already fixed in src/app/api/cron/sync/route.ts:38-128 which pages with
PAGE_SIZE = 50and awhile (hasMore)loop.Proposed Fix
.range(page * PAGE_SIZE, (page + 1) * PAGE_SIZE - 1)PAGE_SIZEPromise.allSettledchunk-of-5 inside each page.order("id")so pagination is deterministicI would like to work on this under GSSoC 26. please assign!
for labels i think gssoc + level2 + type:performance fits here.