ADR 036: add benchmarking workloads#1307
Merged
Merged
Conversation
7c9e8ae to
6de67bf
Compare
6de67bf to
de33476
Compare
robsdedude
commented
May 8, 2026
Contributor
AndyHeap-NeoTech
left a comment
There was a problem hiding this comment.
Looks good, a possible bug to sanity check and one comment.
There was a problem hiding this comment.
Pull request overview
Adds an internal benchmarking runner and a suite of driver workloads under benchmarks/ (ADR-036 / DRIVERS-354), plus wiring for running/type-checking the benchmarks in development and Docker.
Changes:
- Introduce a
benchbenchmark runner (CLI, registry, timing, context) and workload modules for common serialization/deserialization + API scenarios. - Add a dedicated
benchmarks/project (pyproject, lockfile, Dockerfile, README) to run benchmarks locally or in containers. - Add
tyto typing dependencies and a pre-commit hook to type-check the benchmarks project.
Reviewed changes
Copilot reviewed 50 out of 52 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Adds ty to the main repo typing dependency group. |
| benchmarks/uv.lock | Adds a frozen lockfile for the benchmarks environment. |
| benchmarks/src/bench/workloads/util/const.py | Defines integer boundary constants used by workloads. |
| benchmarks/src/bench/workloads/util/_timer.py | Implements timing collection + CSV flushing for benchmark results. |
| benchmarks/src/bench/workloads/util/_deserialization_template.py | Shared template for deserialization workloads (setup + timed work). |
| benchmarks/src/bench/workloads/util/_benchmark_registry.py | Registry + decorator for workload discovery and retry wrapper. |
| benchmarks/src/bench/workloads/util/_benchmark_context.py | Connection/auth context and driver factory for workloads. |
| benchmarks/src/bench/workloads/util/_bencher.py | Core benchmarking loop logic (warmup/timed iterations + retries). |
| benchmarks/src/bench/workloads/util/init.py | Exposes benchmark utility APIs for workloads/runner. |
| benchmarks/src/bench/workloads/serialize_vector.py | Workload: serialize vectors via execute_query. |
| benchmarks/src/bench/workloads/serialize_string.py | Workload: serialize large string payloads. |
| benchmarks/src/bench/workloads/serialize_null.py | Workload: serialize large null lists. |
| benchmarks/src/bench/workloads/serialize_list.py | Workload: serialize deeply nested lists. |
| benchmarks/src/bench/workloads/serialize_integer.py | Workload: serialize mixed-size integer payloads. |
| benchmarks/src/bench/workloads/serialize_float.py | Workload: serialize float payloads. |
| benchmarks/src/bench/workloads/serialize_dictionary.py | Workload: serialize nested dictionaries. |
| benchmarks/src/bench/workloads/serialize_bytes.py | Workload: serialize bytes payloads. |
| benchmarks/src/bench/workloads/serialize_boolean.py | Workload: serialize large boolean lists. |
| benchmarks/src/bench/workloads/record_streaming.py | Workload: stream a large number of records. |
| benchmarks/src/bench/workloads/home_db_resolution.py | Workload: measure home DB resolution behavior. |
| benchmarks/src/bench/workloads/deserialize_vector.py | Workload: deserialize vectors from properties. |
| benchmarks/src/bench/workloads/deserialize_string.py | Workload: deserialize large string property. |
| benchmarks/src/bench/workloads/deserialize_relationship.py | Workload: deserialize relationships + collections. |
| benchmarks/src/bench/workloads/deserialize_path.py | Workload: deserialize paths. |
| benchmarks/src/bench/workloads/deserialize_null.py | Workload: deserialize large list of nulls. |
| benchmarks/src/bench/workloads/deserialize_node.py | Workload: deserialize nodes from match results. |
| benchmarks/src/bench/workloads/deserialize_list.py | Workload: deserialize deeply nested list structure. |
| benchmarks/src/bench/workloads/deserialize_integer.py | Workload: deserialize mixed-size integer payloads. |
| benchmarks/src/bench/workloads/deserialize_float.py | Workload: deserialize float list property. |
| benchmarks/src/bench/workloads/deserialize_dictionary.py | Workload: deserialize nested map structures. |
| benchmarks/src/bench/workloads/deserialize_bytes.py | Workload: deserialize bytes property. |
| benchmarks/src/bench/workloads/deserialize_boolean.py | Workload: deserialize large boolean lists. |
| benchmarks/src/bench/workloads/cold_driver.py | Workload: measure cold driver creation overhead. |
| benchmarks/src/bench/workloads/chunking.py | Workload: measure large payload chunking behavior. |
| benchmarks/src/bench/workloads/api_unmanaged_transaction.py | Workload: unmanaged tx API usage. |
| benchmarks/src/bench/workloads/api_session_run.py | Workload: session.run API usage. |
| benchmarks/src/bench/workloads/api_managed_transaction.py | Workload: execute_write (managed tx) API usage. |
| benchmarks/src/bench/workloads/api_execute_query.py | Workload: execute_query API usage. |
| benchmarks/src/bench/workloads/_data_vector.py | Shared vector test data definitions. |
| benchmarks/src/bench/workloads/_data_float.py | Shared float test data definitions. |
| benchmarks/src/bench/workloads/_data_dictionary.py | Shared dictionary test data definitions. |
| benchmarks/src/bench/workloads/_data_bytes.py | Shared bytes test data definitions. |
| benchmarks/src/bench/workloads/init.py | Auto-imports workload modules so registry picks them up. |
| benchmarks/src/bench/main.py | Benchmark runner CLI entrypoint. |
| benchmarks/src/bench/init.py | Declares the bench package. |
| benchmarks/README.md | Documents local and Docker benchmark execution + lockfile maintenance. |
| benchmarks/pyproject.toml | Adds a dedicated benchmarks project configuration. |
| benchmarks/Dockerfile | Adds a Docker image to run benchmarks reproducibly. |
| benchmarks/.gitignore | Ignores benchmarks venv/results artifacts. |
| .pre-commit-config.yaml | Adds a ty check hook for the benchmarks project. |
| .gitignore | Ignores Ruff/Mypy cache directories. |
| .dockerignore | Updates build context exclusions (incl. benchmark results). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
AndyHeap-NeoTech
approved these changes
May 12, 2026
Contributor
AndyHeap-NeoTech
left a comment
There was a problem hiding this comment.
Looks good. Thanks for the clarifications.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: DRIVERS-354