[RFC] [QDP] [Feature] Benchmark roadmap

[ryankert01]

This roadmap wants to create a new benchmark that values:

1. Fairness: Implementing proper warmup, cache clearing, and CUDA event synchronization
2. Statistical rigor: Collecting multiple runs with comprehensive statistics (mean, std, percentiles, min, max)
   • MAHOUT-816: [Benchmark] Create Statistical Measurement Framework #817
3. tested with real/respected datasets

Category	Dataset / Example	Notes
Image	MNIST (0 vs 1, 4×4 downsample)	Common QML toy dataset, used in PennyLane & Qiskit
Image	Synthetic images	Generated low-res grayscale images
Image	Image (generic)	Placeholder for custom image pipelines
Tabular	Iris (binary, 2 features)	Standard classical→quantum baseline
Tabular	Synthetic blobs	Linearly / non-linearly separable data
Tabular	Tabular (generic)	Arbitrary CSV / NumPy-style input
Scale	Synthetic blobs (100k samples)	Stress-test data loading & batching
Scale	Large tabular data	Focus on I/O, preprocessing overhead
Scale	Large image dataset	Measures memory + pipeline throughput

4. test out different hyper-params (qubits, samples, etc.)
5. create nice graphs for blogposts, presentations, and academic papers
   • compare different hyper-params with different frameworks(mahout qdp, pennylane, quiskit, etc)

cc @guan404ming @rich7420 @400Ping

[ryankert01]

@400Ping do you want to help with planning? It requires reading some papers and understanding current patterns on frontier performance engineering article spaces.

[400Ping]

Yeah, sounds good to me. Will start researching later.

[400Ping]

I was caught up with my last final project this Friday, will start researching this weekend.