Skip to content

Speed up user social graph fields#893

Open
raymondjacobson wants to merge 1 commit into
mainfrom
codex/user-social-set-bitmaps
Open

Speed up user social graph fields#893
raymondjacobson wants to merge 1 commit into
mainfrom
codex/user-social-set-bitmaps

Conversation

@raymondjacobson
Copy link
Copy Markdown
Member

Summary

  • Store per-user followee/follower Roaring bitmaps in Postgres (user_social_sets) with a dirty queue maintained by a lightweight follows trigger.
  • Add an indexer parity job that rebuilds dirty/missing snapshots asynchronously.
  • Move does_current_user_follow, does_follow_current_user, does_current_user_subscribe, and current_user_followee_follow_count hydration out of GetUsers SQL into Go using snapshots, with raw follows fallback for missing rows or pre-migration environments.
  • Add tests proving the snapshot path is used and that rebuilt snapshots match the raw follows result for the moved fields.

Performance context

  • Previous production read-only probes on the same 30-user stress sample found the original correlated query at roughly 45.4s, and the batched SQL version at roughly 13.1s.
  • With the social graph intersection removed from GetUsers, the remaining base query on that sample was roughly 315ms cold I/O in production.
  • This PR keeps the field populated by doing bitmap intersection in the API process instead of repeatedly walking the follows fanout in the main SQL query.

Indexing impact

  • Synchronous write-path work is limited to two small upserts into user_social_set_dirty per follow insert/update/delete.
  • Bitmap rebuilds are async in the indexer job, so large follower/followee set serialization is kept off the follow indexing transaction path.

Validation

  • go test ./api -run 'TestUserQuery|TestGetUsers|TestUserQueryUsesSocialSetSnapshots|TestUserQuerySocialSetSnapshotsMatchFollows'
  • go test ./jobs ./api/dbv1
  • go test ./api ./api/dbv1 ./jobs
  • Migration syntax was smoke-tested earlier against a disposable local DB cloned from test_api; a rerun in this final session was blocked because local Postgres was not listening on /tmp/.s.PGSQL.5432.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant