Skip to content

ADR 036: add benchmarking workloads#1307

Merged
robsdedude merged 12 commits into
neo4j:6.xfrom
robsdedude:036-benchmarking
May 12, 2026
Merged

ADR 036: add benchmarking workloads#1307
robsdedude merged 12 commits into
neo4j:6.xfrom
robsdedude:036-benchmarking

Conversation

@robsdedude
Copy link
Copy Markdown
Member

Closes: DRIVERS-354

@robsdedude robsdedude marked this pull request as ready for review May 7, 2026 08:34
Comment thread benchmarks/src/bench/workloads/util/_timer.py Outdated
Comment thread benchmarks/src/bench/workloads/util/_timer.py Outdated
Comment thread benchmarks/src/bench/workloads/util/_benchmark_registry.py
Comment thread benchmarks/src/bench/workloads/api_unmanaged_transaction.py
Copy link
Copy Markdown
Contributor

@AndyHeap-NeoTech AndyHeap-NeoTech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, a possible bug to sanity check and one comment.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an internal benchmarking runner and a suite of driver workloads under benchmarks/ (ADR-036 / DRIVERS-354), plus wiring for running/type-checking the benchmarks in development and Docker.

Changes:

  • Introduce a bench benchmark runner (CLI, registry, timing, context) and workload modules for common serialization/deserialization + API scenarios.
  • Add a dedicated benchmarks/ project (pyproject, lockfile, Dockerfile, README) to run benchmarks locally or in containers.
  • Add ty to typing dependencies and a pre-commit hook to type-check the benchmarks project.

Reviewed changes

Copilot reviewed 50 out of 52 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
pyproject.toml Adds ty to the main repo typing dependency group.
benchmarks/uv.lock Adds a frozen lockfile for the benchmarks environment.
benchmarks/src/bench/workloads/util/const.py Defines integer boundary constants used by workloads.
benchmarks/src/bench/workloads/util/_timer.py Implements timing collection + CSV flushing for benchmark results.
benchmarks/src/bench/workloads/util/_deserialization_template.py Shared template for deserialization workloads (setup + timed work).
benchmarks/src/bench/workloads/util/_benchmark_registry.py Registry + decorator for workload discovery and retry wrapper.
benchmarks/src/bench/workloads/util/_benchmark_context.py Connection/auth context and driver factory for workloads.
benchmarks/src/bench/workloads/util/_bencher.py Core benchmarking loop logic (warmup/timed iterations + retries).
benchmarks/src/bench/workloads/util/init.py Exposes benchmark utility APIs for workloads/runner.
benchmarks/src/bench/workloads/serialize_vector.py Workload: serialize vectors via execute_query.
benchmarks/src/bench/workloads/serialize_string.py Workload: serialize large string payloads.
benchmarks/src/bench/workloads/serialize_null.py Workload: serialize large null lists.
benchmarks/src/bench/workloads/serialize_list.py Workload: serialize deeply nested lists.
benchmarks/src/bench/workloads/serialize_integer.py Workload: serialize mixed-size integer payloads.
benchmarks/src/bench/workloads/serialize_float.py Workload: serialize float payloads.
benchmarks/src/bench/workloads/serialize_dictionary.py Workload: serialize nested dictionaries.
benchmarks/src/bench/workloads/serialize_bytes.py Workload: serialize bytes payloads.
benchmarks/src/bench/workloads/serialize_boolean.py Workload: serialize large boolean lists.
benchmarks/src/bench/workloads/record_streaming.py Workload: stream a large number of records.
benchmarks/src/bench/workloads/home_db_resolution.py Workload: measure home DB resolution behavior.
benchmarks/src/bench/workloads/deserialize_vector.py Workload: deserialize vectors from properties.
benchmarks/src/bench/workloads/deserialize_string.py Workload: deserialize large string property.
benchmarks/src/bench/workloads/deserialize_relationship.py Workload: deserialize relationships + collections.
benchmarks/src/bench/workloads/deserialize_path.py Workload: deserialize paths.
benchmarks/src/bench/workloads/deserialize_null.py Workload: deserialize large list of nulls.
benchmarks/src/bench/workloads/deserialize_node.py Workload: deserialize nodes from match results.
benchmarks/src/bench/workloads/deserialize_list.py Workload: deserialize deeply nested list structure.
benchmarks/src/bench/workloads/deserialize_integer.py Workload: deserialize mixed-size integer payloads.
benchmarks/src/bench/workloads/deserialize_float.py Workload: deserialize float list property.
benchmarks/src/bench/workloads/deserialize_dictionary.py Workload: deserialize nested map structures.
benchmarks/src/bench/workloads/deserialize_bytes.py Workload: deserialize bytes property.
benchmarks/src/bench/workloads/deserialize_boolean.py Workload: deserialize large boolean lists.
benchmarks/src/bench/workloads/cold_driver.py Workload: measure cold driver creation overhead.
benchmarks/src/bench/workloads/chunking.py Workload: measure large payload chunking behavior.
benchmarks/src/bench/workloads/api_unmanaged_transaction.py Workload: unmanaged tx API usage.
benchmarks/src/bench/workloads/api_session_run.py Workload: session.run API usage.
benchmarks/src/bench/workloads/api_managed_transaction.py Workload: execute_write (managed tx) API usage.
benchmarks/src/bench/workloads/api_execute_query.py Workload: execute_query API usage.
benchmarks/src/bench/workloads/_data_vector.py Shared vector test data definitions.
benchmarks/src/bench/workloads/_data_float.py Shared float test data definitions.
benchmarks/src/bench/workloads/_data_dictionary.py Shared dictionary test data definitions.
benchmarks/src/bench/workloads/_data_bytes.py Shared bytes test data definitions.
benchmarks/src/bench/workloads/init.py Auto-imports workload modules so registry picks them up.
benchmarks/src/bench/main.py Benchmark runner CLI entrypoint.
benchmarks/src/bench/init.py Declares the bench package.
benchmarks/README.md Documents local and Docker benchmark execution + lockfile maintenance.
benchmarks/pyproject.toml Adds a dedicated benchmarks project configuration.
benchmarks/Dockerfile Adds a Docker image to run benchmarks reproducibly.
benchmarks/.gitignore Ignores benchmarks venv/results artifacts.
.pre-commit-config.yaml Adds a ty check hook for the benchmarks project.
.gitignore Ignores Ruff/Mypy cache directories.
.dockerignore Updates build context exclusions (incl. benchmark results).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread benchmarks/src/bench/workloads/util/_timer.py Outdated
Comment thread benchmarks/src/bench/workloads/util/_bencher.py Outdated
Comment thread benchmarks/src/bench/workloads/chunking.py Outdated
Comment thread benchmarks/Dockerfile Outdated
Comment thread benchmarks/pyproject.toml
Comment thread benchmarks/src/bench/workloads/serialize_float.py Outdated
Comment thread benchmarks/src/bench/workloads/serialize_integer.py Outdated
Comment thread benchmarks/src/bench/workloads/api_unmanaged_transaction.py Outdated
Comment thread benchmarks/src/bench/workloads/api_managed_transaction.py Outdated
Comment thread benchmarks/src/bench/workloads/api_execute_query.py Outdated
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 50 out of 52 changed files in this pull request and generated 1 comment.

Comment thread benchmarks/src/bench/workloads/util/_timer.py Outdated
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 51 out of 53 changed files in this pull request and generated no new comments.

Copy link
Copy Markdown
Contributor

@AndyHeap-NeoTech AndyHeap-NeoTech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks for the clarifications.

@robsdedude robsdedude merged commit 9e23c90 into neo4j:6.x May 12, 2026
33 checks passed
@robsdedude robsdedude deleted the 036-benchmarking branch May 12, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants