feat: n_jobs for threaded slice aggregation

[FBumann]

Second of the three perf drafts — the parallel-slice experiment, with honest numbers. Supersedes the closed #46 (same design, remeasured on the current stack). Closes #12 if merged.

What

aggregate(..., n_jobs=) threads the per-slice loop: None/1 sequential (default, unchanged), -1 all CPUs, -2 all but one, positive N exact (joblib convention), capped at the slice count. Slices share no state; results are identical to sequential (asserted in test/test_parallel.py).

Measured (8 CPUs, min of 3)

shape	sequential	2 threads	4 threads	8 threads
8 slices × 8 cols, default	0.30 s	0.26 s	0.25 s	0.26 s
8 slices × 8 cols, full config	0.59 s	0.54 s	0.54 s	0.55 s
8 slices × 64 cols, full config	3.90 s	3.47 s	3.43 s	3.46 s

~10–13%, flat beyond 2 workers — the hot paths (duration representation, accuracy metrics) are GIL-holding pandas/sklearn loops, so threads mostly wait. A ProcessPoolExecutor variant was measured too and is net negative (4.4 s vs 3.9 s sequential — worker spawn/imports + pickling exceed the gains), which also reproduces the conclusion from #46's description and is why this PR does not offer a process backend.

The old #46 claimed 2×; that does not reproduce on the current stack/config.

Verdict / recommendation

Parallelism is not the lever today — the same GIL-holding loops filed as FBumann/tsam#49 and FBumann/tsam#50 are the reason. Once those vectorize (numpy releases the GIL), this exact thread pool should scale near-linearly, and the API is already shaped for it. Options: merge now for the modest win + future-proofing, or keep this draft parked until the upstream fixes land and remeasure. Draft until that call is made.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3884d18e-1f95-438f-a921-91de95dfef0b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/parallel-slices

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[FBumann]

Not worth it

[github-actions]

Benchmarks

Δ% vs base, one table sorted by biggest change first
(min walltime and memray peak). Peak memory is
deterministic — any delta there is real; walltime on GitHub
runners is noisy, so only large time deltas are meaningful.
The interactive plot is attached as the benchmark-plot artifact.

Full table

benchmarks/test_bench_aggregate.py

name	metric	base	head
test_wrapper_to_dataframe	time	0.0003594s	+1.3%
	peak	8.55 MiB	0.0%
test_user_disaggregate	time	0.01312s	+0.8%
	peak	26.7 MiB	0.0%
test_config_representation[dist_cluster]	time	0.05408s	+0.5%
	peak	1.78 MiB	0.0%
test_wrapper_concat_results	time	0.02216s	+0.4%
	peak	8.72 MiB	0.0%
test_e2e_default	time	0.08754s	+0.3%
	peak	7.11 MiB	0.0%
test_e2e_full	time	0.1637s	+0.3%
	peak	6.6 MiB	0.0%
test_config_representation[medoid]	time	0.05546s	0.0%
	peak	1.9 MiB	0.0%
test_e2e_slices	time	1.335s	0.0%
	peak	14.3 MiB	0.0%
test_config_representation[dist_global]	time	0.1425s	0.0%
	peak	1.78 MiB	0.0%
test_config_representation[dist_global_minmax]	time	0.1449s	-0.1%
	peak	1.78 MiB	0.0%
test_config_extremes[append]	time	0.05717s	-0.3%
	peak	1.9 MiB	0.0%
test_config_representation[mean]	time	0.05274s	-0.3%
	peak	1.78 MiB	0.0%
test_config_extremes[replace]	time	0.05718s	-0.4%
	peak	1.9 MiB	0.0%
test_e2e_wide[dist_global]	time	2.063s	-0.5%
	peak	104 MiB	0.0%
test_user_apply	time	0.4524s	-0.7%
	peak	104 MiB	0.0%
test_e2e_wide[medoid]	time	0.5637s	-1.0%
	peak	113 MiB	0.0%
test_e2e_multidim	time	0.2537s	-1.7%
	peak	26.7 MiB	0.0%
test_wrapper_result_conversion	time	0.04033s	-9.3%
	peak	60.3 MiB	0.0%