Skip to content

Improves lock contention when multiple parallel read-queries are being executed#177

Merged
staticlibs merged 7 commits intoduckdb:mainfrom
jprafael:codex/sqlite-scanner-contention-pr
Mar 16, 2026
Merged

Improves lock contention when multiple parallel read-queries are being executed#177
staticlibs merged 7 commits intoduckdb:mainfrom
jprafael:codex/sqlite-scanner-contention-pr

Conversation

@jprafael
Copy link
Copy Markdown
Contributor

@jprafael jprafael commented Mar 14, 2026

Context
Using duckdb-sqllite as the metadata catalog for ducklake.
When running multiple read-only queries in parallel (using .cursor()) the code runs into lock contention and has worse performance than single threaded.

Hotspot 1

  • In SqliteInitInternal invokes sqlite Prepare for each query, even in the common case where the sql is exactly the same as before (next iteration with different rowid filters).
  • This version caches the prepared statement and reuses it if its the same as before.
  • Q for maintainers: can it be assumed that the statement that is prepared is always the same (e.g. same bind_data/column_ids)? if so we can skip the equality check after the prepared statement is cached.

Hotspot 2

  • In SqliteParallelStateNext the code holds the gstate.lock while running SqliteInitInternal.
  • This version mutates the global state while holding the lock, but releases it before running SqliteInitInternal

This leads to a ~10x speedup in my environment (16x threads)

@jprafael jprafael changed the title WIP: Improves lock contention when multiple parallel read-queries are being executed Improves lock contention when multiple parallel read-queries are being executed Mar 14, 2026
Comment thread src/sqlite_scanner.cpp
@staticlibs
Copy link
Copy Markdown
Collaborator

Thanks!

@staticlibs staticlibs merged commit 3c325ab into duckdb:main Mar 16, 2026
6 checks passed
@jprafael jprafael deleted the codex/sqlite-scanner-contention-pr branch March 16, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants