CHORE: Improve PR performance sensitivity and measurement reliability

[Gaurav Sharma (bewithgaurav)]

Work Item / Issue Reference

ADO Work Item: Fixed AB#44819

---

Summary

Make PR Performance Reports more sensitive to consistent changes while reducing first-use and short-measurement noise.

• Add tentative small impact above 10% and large impact above 20%, retaining the per-workload 1 ms floor and 80% pair agreement at each tier.
• Keep inconsistent movement out of headlines, badges, and signal fingerprints. Preserve raw percentages and agreement in the expanded timing table.
• Warm steady-state tasks inside every fresh measurement process. Keep connection opening cold.
• Measure short read/query tasks in identical fixed repetition blocks on both revisions, retaining raw inner timings and normalizing phase diagnostics per original workload.
• Validate repetition and warmup identity while preserving retained single-pass artifacts. Label clean partial coverage incomplete.

Validation

164 profiler CI contract tests passed. Pinned formatting and changed-file lint checks passed.

Five fresh-process same-build pairs against local SQL Server 2022 covered all 22 tasks with zero signals, 0.63% median absolute paired movement, and 3.40% maximum movement. This is a local macOS control, not a claim that hosted Linux variance has been eliminated.

No runtime driver changes, additional dependencies, dedicated-runner configuration, or CI matrix expansion.

[github-actions]

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 SMALL IMPROVEMENTS 0 SMALL SLOWDOWNS 0 LARGE IMPROVEMENTS 0 LARGE SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task	Before	After	Paired change	Agreement	Result
Connection opening	11.144 ms	11.414 ms	-0.2%	0/5 beyond 10%	no signal
SELECT queries	0.841 ms	0.827 ms	-3.6%	0/5 beyond 10%	no signal
Row insertion	34.396 ms	34.622 ms	+1.3%	0/5 beyond 10%	no signal
Executemany inserts	165.630 ms	165.508 ms	-1.7%	0/5 beyond 10%	no signal
Fetch-all queries	125.223 ms	126.507 ms	+3.2%	0/5 beyond 10%	no signal
Row-by-row fetching	14.521 ms	14.320 ms	-3.5%	0/5 beyond 10%	no signal
Batched row fetching	129.240 ms	130.503 ms	+1.0%	0/5 beyond 10%	no signal
Transaction commit and rollback	115.715 ms	117.316 ms	+1.2%	0/5 beyond 10%	no signal
Arrow row fetching	77.480 ms	76.780 ms	+2.6%	1/5 beyond 10%	no signal
100,000-row insertion	493.299 ms	469.442 ms	-4.0%	0/5 beyond 10%	no signal
Row fetching in batches of 100	122.561 ms	123.706 ms	-0.4%	0/5 beyond 10%	no signal
Row fetching in batches of 10,000	134.977 ms	126.277 ms	-7.4%	1/5 beyond 10%	no signal
Repeated positional queries	33.782 ms	34.071 ms	+1.1%	1/5 beyond 10%	no signal
Repeated named-parameter queries	36.649 ms	36.584 ms	-0.5%	0/5 beyond 10%	no signal
Legacy 100,000-row insertion	372.364 ms	388.405 ms	+4.8%	0/5 beyond 10%	no signal
Insertion with explicit input sizes	528.067 ms	521.625 ms	+0.5%	0/5 beyond 10%	no signal
Joined aggregation queries	178.439 ms	176.642 ms	-1.8%	0/5 beyond 10%	no signal
Large joined-result fetching	194.422 ms	188.684 ms	+0.6%	0/5 beyond 10%	no signal
1.2-million-row fetching	3818.981 ms	3656.437 ms	-3.6%	0/5 beyond 10%	no signal
Common table expression queries	5.119 ms	5.045 ms	-1.4%	1/5 beyond 10%	no signal
256 KiB VARCHAR(MAX) / fetchall()	1.400 ms	1.337 ms	+0.5%	0/5 beyond 10%	no signal
10,000 scalar values / fetchval() (debug disabled)	108.292 ms	107.206 ms	-0.1%	0/5 beyond 10%	no signal

Unix / SQL Server 2025

Database task	Before	After	Paired change	Agreement	Result
Connection opening	95.843 ms	96.114 ms	-0.0%	0/5 beyond 10%	no signal
SELECT queries	0.745 ms	0.729 ms	-2.0%	0/5 beyond 10%	no signal
Row insertion	31.216 ms	31.306 ms	-0.3%	0/5 beyond 10%	no signal
Executemany inserts	133.375 ms	144.392 ms	+10.3%	3/5 beyond 10%	no signal
Fetch-all queries	116.168 ms	114.845 ms	-1.1%	1/5 beyond 10%	no signal
Row-by-row fetching	13.357 ms	12.545 ms	-6.6%	2/5 beyond 10%	no signal
Batched row fetching	111.571 ms	111.345 ms	-0.2%	0/5 beyond 10%	no signal
Transaction commit and rollback	102.790 ms	101.744 ms	-1.0%	0/5 beyond 10%	no signal
Arrow row fetching	70.847 ms	70.929 ms	+3.2%	0/5 beyond 10%	no signal
100,000-row insertion	395.762 ms	394.306 ms	-0.6%	0/5 beyond 10%	no signal
Row fetching in batches of 100	106.496 ms	106.974 ms	+0.3%	1/5 beyond 10%	no signal
Row fetching in batches of 10,000	117.301 ms	125.474 ms	+0.8%	1/5 beyond 10%	no signal
Repeated positional queries	30.149 ms	29.493 ms	-1.6%	0/5 beyond 10%	no signal
Repeated named-parameter queries	32.526 ms	32.352 ms	+0.3%	0/5 beyond 10%	no signal
Legacy 100,000-row insertion	306.321 ms	307.432 ms	+0.3%	0/5 beyond 10%	no signal
Insertion with explicit input sizes	434.865 ms	435.584 ms	+0.2%	1/5 beyond 10%	no signal
Joined aggregation queries	167.180 ms	164.924 ms	-0.6%	0/5 beyond 10%	no signal
Large joined-result fetching	174.657 ms	173.687 ms	-1.2%	0/5 beyond 10%	no signal
1.2-million-row fetching	3754.490 ms	3697.833 ms	-0.1%	0/5 beyond 10%	no signal
Common table expression queries	4.393 ms	4.414 ms	+0.5%	0/5 beyond 10%	no signal
256 KiB VARCHAR(MAX) / fetchall()	1.341 ms	1.340 ms	+0.1%	0/5 beyond 10%	no signal
10,000 scalar values / fetchval() (debug disabled)	101.315 ms	100.430 ms	-1.2%	0/5 beyond 10%	no signal

Build and measurement details

ADO build 183372

PR head: 540e463c9963fdeeae759d1407e0a2adedfd62f8
Base: 9424705251baf8ded41a8a4968216b1fd9b00929
Measured merge: 85e9fa087a5289226ddc3cdccfd95b17269f8856

• Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4305.5; 5 paired comparisons and 1 discarded warmup pair.
  Steady-state tasks warm up once inside each worker; connection opening remains cold. Short-task blocks: SELECT queries: 100 repetitions; Row-by-row fetching: 10 repetitions; Repeated positional queries: 3 repetitions; Repeated named-parameter queries: 3 repetitions; Common table expression queries: 20 repetitions; 256 KiB VARCHAR(MAX) / fetchall(): 100 repetitions.
• Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5015.4; 5 paired comparisons and 1 discarded warmup pair.
  Steady-state tasks warm up once inside each worker; connection opening remains cold. Short-task blocks: SELECT queries: 100 repetitions; Row-by-row fetching: 10 repetitions; Repeated positional queries: 3 repetitions; Repeated named-parameter queries: 3 repetitions; Common table expression queries: 20 repetitions; 256 KiB VARCHAR(MAX) / fetchall(): 100 repetitions.

Tentative small impact requires more than 10% median paired movement; large impact requires more than 20%. Both require at least 1 ms between the median per-workload runtimes and at least 80% of pairs exceeding that tier's threshold in the same direction. Large impact is evaluated first. Unconfirmed movement remains no signal.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes. Repeated blocks use the mean time per original workload; phase timings and call counts are normalized per repetition. Inner repetitions are not additional independent pairs.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

[Copilot]

🔵 Needs a closer look

Hosted Linux behavior and timing reliability remain dependent on CI evidence beyond the supplied local macOS control.

0 open findings

What changed in this PR

Improves profiler sensitivity and reliability through tiered signals, warmups, repeated short workloads, and stricter artifact validation.

Changes:

• Adds 10% tentative-small and 20% large impact tiers.
• Warms steady-state scenarios and repeats short workloads.
• Normalizes diagnostics and expands tests/documentation.

File	Description
tests/​test_036_profiler_ci.py	Covers tiers, repetition validation, warmups, and reporting.
eng/​profiler_benchmarks/​workloads.py	Defines fixed short-workload repetition counts.
eng/​profiler_benchmarks/​report.py	Validates, classifies, normalizes, and renders results.
eng/​profiler_benchmarks/​README.md	Documents updated measurement semantics.
eng/​profiler_benchmarks/​controller.py	Implements per-process warmup and repeated measurement.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 85%

📈 Total Lines Covered: 9443 out of 11092 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 62.6%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection_pool.cpp: 82.3%
mssql_python.pybind.connection.connection.cpp: 83.1%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.fetch_temporal.hpp: 92.1%
mssql_python.cursor.py: 92.5%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report