Skip to content

[Performance] /api/wakatime/sync loads every user with a wakatime key in one unbounded query #3161

Description

@vedant7007

Description

while comparing the wakatime cron route with the newer /api/cron/sync one i noticed the wakatime one still does an unbounded fetch of the entire users table.

The Problem

src/app/api/wakatime/sync/route.ts:13-17:

const { data: users, error } = await supabaseAdmin
  .from("users")
  .select("id, wakatime_api_key_encrypted, wakatime_api_key_iv")
  .not("wakatime_api_key_encrypted", "is", null)
  .not("wakatime_api_key_iv", "is", null);

no .range(), no .limit(). every wakatime user in the DB (plus their encrypted keys) is pulled into serverless memory in one shot. line 29 then loops through users.length in chunks of 5.

failure scenario: on a real deployment with a few thousand wakatime users this either hits the vercel 4.5mb response limit from supabase, blows the 10s/60s function timeout, or gets killed by the runtime memory cap and the entire nightly sync silently fails. same class of bug that was already fixed in src/app/api/cron/sync/route.ts:38-128 which pages with PAGE_SIZE = 50 and a while (hasMore) loop.

Proposed Fix

  • adopt the same pagination pattern as cron/sync: page through 50 users at a time using .range(page * PAGE_SIZE, (page + 1) * PAGE_SIZE - 1)
  • break out of the while loop when the returned page is smaller than PAGE_SIZE
  • keep the existing Promise.allSettled chunk-of-5 inside each page
  • also add .order("id") so pagination is deterministic

I would like to work on this under GSSoC 26. please assign!

for labels i think gssoc + level2 + type:performance fits here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions