diff --git a/README.md b/README.md index 73bced4..5d1f1f3 100644 --- a/README.md +++ b/README.md @@ -14,24 +14,44 @@ These benchmarks quantify the trade-offs: where OME-Arrow is faster, where it is The real-image benchmarks use four datasets from the [OME-IRIS](https://github.com/WayScience/ome-iris) catalog. All are small representative subsets drawn from [`cytomining/CytoDataFrame`](https://github.com/cytomining/CytoDataFrame) test data; they are not full plates or full studies. -| OME-IRIS dataset ID | Images | Channels | Dimensionality | Description | Upstream source | -| --------------------------- | ------ | ------------------ | -------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -| `nf1-cellpainting-shrunken` | 15 | 3 (DAPI, GFP, RFP) | 2D | 5 sites from 1 well of an NF1 fibroblast Cell Painting assay; includes feature profiles | [CytoDataFrame NF1](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/cytotable/NF1_cellpainting_data_shrunken) | -| `jump-plate-example` | 3 | 3 (ch2, ch3, ch5) | 2D | 1 field of view from JUMP Cell Painting plate BR00117006 | [CytoDataFrame JUMP BR00117006](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/cytotable/JUMP_plate_BR00117006) | -| `pediatric-cancer-atlas` | 6 | 3 (ch1, ch2, ch3) | 2D | 2 fields from a pediatric cancer atlas Cell Painting plate; includes feature profiles | [CytoDataFrame pediatric atlas](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/cytotable/pediatric_cancer_atlas_profiling) | -| `nuclei-3d` | 4 | 1 | 3D | CellProfiler tutorial 3D noise nuclei segmentation volumes (~13 MB each) | [CytoDataFrame 3D nuclei](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/CP_tutorial_3D_noise_nuclei_segmentation) | +| Figure label | OME-IRIS dataset ID | Purpose/design | Image tensor shape | TIFF ref (MB) | Upstream source | +| ------------ | --------------------------- | ------------------------------------------------------------ | --------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | +| NF1 | `nf1-cellpainting-shrunken` | NF1 fibroblast Cell Painting subset with morphology profiles | TCZYX=1x1x1x904x1224 | 10.06 | [CytoDataFrame NF1](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/cytotable/NF1_cellpainting_data_shrunken) | +| JUMP | `jump-plate-example` | Single field from JUMP Cell Painting plate BR00117006 | TCZYX=1x1x1x1080x1080 | 7.65 | [CytoDataFrame JUMP BR00117006](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/cytotable/JUMP_plate_BR00117006) | +| Ped. atlas | `pediatric-cancer-atlas` | Pediatric cancer atlas Cell Painting subset with profiles | TCZYX=1x1x1x1080x1080 | 11.65 | [CytoDataFrame pediatric atlas](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/cytotable/pediatric_cancer_atlas_profiling) | +| 3D nuclei | `nuclei-3d` | CellProfiler tutorial 3D nuclei segmentation volumes | TCZYX=1x1x100x258x258 | 50.88 | [CytoDataFrame 3D nuclei](https://github.com/cytomining/CytoDataFrame/tree/main/tests/data/CP_tutorial_3D_noise_nuclei_segmentation) | + +Size and experimental-design considerations: + +- These are small representative OME-IRIS subsets, not full plates or population-scale screens. They are useful for isolating format and access-pattern behavior under controlled, reproducible conditions. +- The 2D Cell Painting datasets test common high-content screening access patterns: multi-channel field-of-view reads, image metadata scans, and profile-to-image metadata joins. NF1 and Pediatric Cancer Atlas include morphology-profile tables, so they exercise cross-modal image/profile access rather than image-only I/O. +- The 3D nuclei dataset tests volumetric image access with larger per-image payloads and a different tensor shape, which is why its artifact sizes are larger despite fewer source images. +- Artifact sizes should be compared within each dataset row. Differences across rows reflect image count, image dimensions, dimensionality, bit depth, channel count, and compression behavior, not only the storage format. ## Key findings > Read this section for a quick summary. The individual benchmark sections below provide the full picture. -- **OME-Arrow reads and writes faster than OME-Zarr, which matters at profiling scale.** OA NT (OME-Arrow nested table) Parquet writes 2.6–3.4x faster than OME-Zarr and reads 20–50% faster when reading all images in a dataset. OA NT Vortex is 6–14x faster than OME-Zarr for multi-image reads. In image-based profiling pipelines - where a single Cell Painting experiment may involve thousands of images - faster multi-image reads directly reduce the wall time of feature extraction, QC, and model training. For random single-image access, OA NT Parquet is slower than OME-Zarr, but OA NT Vortex and Lance are both faster. OA DS (OMEArrowDataset) Parquet is the slowest OME-Arrow variant but remains comparable to OME-Zarr. +- **OME-Arrow reads and writes faster than OME-Zarr** OA NT (OME-Arrow nested table) Parquet writes 2.6–3.4x faster than OME-Zarr and reads 20–50% faster when reading all images in a dataset. OA NT Vortex is 6–14x faster than OME-Zarr for multi-image reads. In image-based profiling pipelines - where a single Cell Painting experiment may involve thousands of images - faster multi-image reads directly reduce the wall time of feature extraction, QC, and model training. +- For random single-image access, OA NT Parquet is slower than OME-Zarr, but OA NT Vortex and Lance are both faster. OA DS (OMEArrowDataset) Parquet is the slowest OME-Arrow variant but remains comparable to OME-Zarr. - **OME-Arrow enables queries that OME-Zarr cannot, and fits existing data science tooling.** Because images and feature profiles are stored together in a single Arrow table, downstream tools can filter on metadata, join profile rows to pixel data, and run SQL or Arrow compute expressions without opening individual files. At plate scale this removes a whole class of file-management overhead that per-directory formats like OME-Zarr require. Arrow and Parquet are standard formats in data science tooling - [DuckDB](https://duckdb.org), [pandas](https://pandas.pydata.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) all work with OME-Arrow tables natively, without special adapters. This also means profiles and images share one type system and one file, replacing the two-format world (feature Parquet + image directories) typical in Cell Painting pipelines today. -- **Profiling joins are fast.** Joining feature rows to image metadata via an OME-Arrow nested table takes ~4 ms at observation scale (~15 k rows) - comparable to a plain Parquet path join (~3 ms). The small overhead buys structured, queryable image metadata in place of opaque file paths. +- **Profile-to-image joins are fast.** Joining feature rows (i.e., morphology profiles) to image metadata via an OME-Arrow nested table takes ~4 ms for ~15 k feature rows - comparable to a plain Parquet path join (~3 ms). The small overhead provides structured, queryable image metadata in place of brittle and often opaque file paths. - **Pre-converting removes the join at query time, at the cost of upfront conversion.** When profile and image data are co-located in a single OME-Arrow table, metadata reads drop to ~1 ms because no join is performed at query time. Pre-conversion is worth considering for read-heavy workloads where the upfront cost amortises quickly. - **Chunk-level Zstd-1 is the practical compression default.** Across all tested OME-IRIS datasets, chunk-level Zstd-1 achieves strong size reduction with minimal read overhead. Doubling compression to Zstd-6 shrinks files further but increases write time without meaningfully cutting read time. - **Reuse the `OMEArrowDataset` handle.** Keeping the handle open across reads is marginally faster than reopening per call, with the clearest benefit on random single-image reads. The difference is small in the tested datasets but the pattern is consistent. +![OME-Arrow benchmark advantage summary](figures/ome_arrow_advantage_summary.png) + +**OME-Arrow accelerates image analysis I/O while adding relational access.** + +- Panels A-B compare OME-Zarr timings against OME-Arrow and show 6.8-14.7x faster bulk reads with the OME-Arrow nested-table Vortex backend and 1.7-3.4x faster writes with the OME-Arrow nested-table Parquet backend. +- Panel C shows that OME-Arrow Parquet storage remains comparable to OME-Zarr across representative OME-IRIS datasets. +- Panels D-E focus on the NF1 Cell Painting join benchmark: We perform a 15,000 feature rows join to images. OME-Arrow performs this join in ~4 ms, while OME-Zarr does not provide this functionality; converted OME-Arrow metadata scans are 8.6x faster than converted OME-Zarr metadata scans. + +These speedups arise because OME-Arrow stores image payloads, metadata, and profile-linked identifiers in Arrow-compatible columnar tables rather than separate directory-based image stores plus external path indexes. +This reduces file traversal, object traversal, and format-conversion overhead during bulk reads, writes, and metadata scans. +Apache Arrow's contiguous columnar buffers also support zero-copy data access and SIMD-friendly execution, creating a path toward CPU/GPU acceleration because downstream tools can operate on memory layouts already suited for vectorized and accelerator-backed computation. + ## Benchmarks ### Parquet, Vortex, and LanceDB @@ -76,11 +96,13 @@ File size varies by dataset because OME-IRIS datasets differ in image count, res Breaks the NF1 experiment into individual operations across four panels: query time, in-memory result size, rows returned per query, and benchmark artifact storage. -**Query time (top-left):** path joins and OA metadata joins are both in the low-millisecond range at profile scale (~15 k rows). The pre-converted OA metadata scan is fastest. Payload operations - materializing full pixel arrays - are 10–40x slower than metadata operations and dominate the right side of the bar chart. +**Query time (top-left):** path joins and OA metadata joins are both in the low-millisecond range for ~15 k morphology feature rows. +The pre-converted OA metadata scan is fastest. +Payload operations - materializing full pixel arrays - are 10–40x slower than metadata operations and dominate the right side of the bar chart. **Result size (top-right):** metadata operations return small result tables (kilobytes). Payload operations materialize the full pixel arrays in memory; the OME-Arrow path allocates more memory than TIFF or Zarr because of Arrow's nested struct representation. -**Rows returned (bottom-left):** the row counts differ because two access patterns are present. Operations labeled "profile rows" join at observation scale (~15 k repeated feature rows); "image rows" operations return one row per real image (15). The log scale is needed to show both on the same axis. +**Rows returned (bottom-left):** the row counts differ because two access patterns are present. Operations labeled "profile rows" join ~15 k repeated feature rows; "image rows" operations return one row per real image (15). The log scale is needed to show both on the same axis. **Artifact storage (bottom-right):** shows how much disk space each converted artifact requires. The OME-Arrow image index, converted profile-image table, OME-TIFF files, and OME-Zarr groups are all within a factor of 3 of each other for this dataset. @@ -166,6 +188,12 @@ An OME-Arrow variant lives at `notebooks/compare_parquet_vortex_lance_ome.py` wh An OME-Arrow-only + OME-Zarr benchmark lives at `src/benchmarks/compare_ome_arrow_only.py`, focusing on table formats that store one OME image column plus directory-per-image TIFF and OME-Zarr comparisons. +The benchmark advantage summary can be regenerated from saved parquet outputs without rerunning benchmarks: + +```bash +uv run python src/benchmarks/plot_ome_arrow_advantage_summary.py +``` + ## Additional validation benchmarks These benchmark scripts provide broader coverage over real imaging datasets and are useful for validating results beyond the smaller synthetic and single-image benchmarks. diff --git a/figures/ome_arrow_advantage_summary.png b/figures/ome_arrow_advantage_summary.png new file mode 100644 index 0000000..23e6ce0 Binary files /dev/null and b/figures/ome_arrow_advantage_summary.png differ diff --git a/src/benchmarks/plot_ome_arrow_advantage_summary.py b/src/benchmarks/plot_ome_arrow_advantage_summary.py new file mode 100644 index 0000000..8bb8f41 --- /dev/null +++ b/src/benchmarks/plot_ome_arrow_advantage_summary.py @@ -0,0 +1,287 @@ +"""OME-Arrow benchmark advantage summary figure. + +This script derives a concise summary figure from existing benchmark summary +tables. It does not rerun benchmarks; it normalizes the saved results so the +OME-Arrow advantages are visible at summary scale. +""" + +from pathlib import Path + +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd + + +DATA_DIR = Path("data") +FIGURES_DIR = Path("figures") +PLOT_PNG = FIGURES_DIR / "ome_arrow_advantage_summary.png" + +ALL_IMAGES_SUMMARY = DATA_DIR / "compare_ome_iris_all_images_summary.parquet" +JOINS_SUMMARY = DATA_DIR / "compare_ome_iris_joins_summary.parquet" + +DATASET_LABELS = { + "CP_tutorial_3D_noise_nuclei_segmentation": "3D nuclei", + "JUMP_plate_BR00117006": "JUMP", + "NF1_cellpainting_data_shrunken": "NF1", + "pediatric_cancer_atlas_profiling": "Ped. atlas", +} +DATASET_ORDER = tuple(DATASET_LABELS) + +GREEN = "#1F7A4D" +BLUE = "#2F6FDB" +GOLD = "#B7791F" +BROWN = "#6F4E37" +GRAY = "#6B7280" +INK = "#111827" +GRID = "#D1D5DB" +SHADE = "#E5E7EB" + + +def benchmark_value( + summary: pd.DataFrame, dataset: str, method: str, column: str +) -> float: + """Return one scalar benchmark value from the all-image summary table.""" + rows = summary[(summary["dataset"] == dataset) & (summary["method"] == method)] + if rows.empty: + raise KeyError(f"Missing {method!r} for {dataset!r} in {column!r}") + return float(rows.iloc[0][column]) + + +def operation_row(summary: pd.DataFrame, operation: str) -> pd.Series: + """Return one operation row from the join benchmark summary table.""" + rows = summary[summary["operation"] == operation] + if rows.empty: + raise KeyError(f"Missing operation {operation!r}") + return rows.iloc[0] + + +def style_axis(ax: plt.Axes) -> None: + """Apply common visual styling for summary figure panels.""" + ax.spines[["top", "right"]].set_visible(False) + ax.grid(axis="y", color=GRID, lw=0.8, alpha=0.75) + ax.set_axisbelow(True) + + +def panel_label(ax: plt.Axes, label: str, x: float = -0.18) -> None: + """Place panel labels outside the plotting area to avoid overlap.""" + ax.text( + x, + 1.08, + label, + transform=ax.transAxes, + fontsize=16, + fontweight="bold", + color=INK, + va="top", + clip_on=False, + ) + + +def labeled_bars( + ax: plt.Axes, + values: list[float], + color: str, + fmt: str, + y_offset: float, +) -> None: + """Draw vertical bars with bold value labels.""" + x = np.arange(len(values)) + bars = ax.bar(x, values, color=color, width=0.62) + for bar, value in zip(bars, values, strict=True): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + y_offset, + fmt.format(value), + ha="center", + va="bottom", + fontweight="bold", + color=INK, + ) + + +def plot_advantage(summary: pd.DataFrame, joins: pd.DataFrame) -> None: + """Build and save the benchmark advantage summary figure.""" + labels = [DATASET_LABELS[dataset] for dataset in DATASET_ORDER] + + read_speedup = [ + benchmark_value(summary, dataset, "Zarr", "read_all_avg_s") + / benchmark_value(summary, dataset, "OA NT Vortex", "read_all_avg_s") + for dataset in DATASET_ORDER + ] + write_speedup = [ + benchmark_value(summary, dataset, "Zarr", "write_avg_s") + / benchmark_value(summary, dataset, "OA NT Parquet", "write_avg_s") + for dataset in DATASET_ORDER + ] + storage_ratio = [ + benchmark_value(summary, dataset, "OA NT Parquet", "size_mb") + / benchmark_value(summary, dataset, "Zarr", "size_mb") + for dataset in DATASET_ORDER + ] + + path_join = operation_row(joins, "Path Parquet join, observation scale") + oa_join = operation_row(joins, "OA NT Parquet metadata join, observation scale") + converted_oa_meta = operation_row(joins, "Converted OA NT Parquet metadata scan") + tiff_meta = operation_row(joins, "OME-TIFF metadata scan") + zarr_meta = operation_row(joins, "OME-Zarr metadata scan") + + join_ms = [path_join["avg_seconds"] * 1000, oa_join["avg_seconds"] * 1000] + scan_ms = [ + converted_oa_meta["avg_seconds"] * 1000, + tiff_meta["avg_seconds"] * 1000, + zarr_meta["avg_seconds"] * 1000, + ] + plt.rcParams.update( + { + "font.family": "DejaVu Sans", + "font.size": 11, + "axes.titlesize": 14, + "axes.labelsize": 11, + "xtick.labelsize": 10, + "ytick.labelsize": 10, + } + ) + fig = plt.figure(figsize=(14, 8.6), facecolor="white") + grid = fig.add_gridspec(2, 3, hspace=0.36, wspace=0.48) + + ax_read = fig.add_subplot(grid[0, 0]) + ax_write = fig.add_subplot(grid[0, 1]) + ax_storage = fig.add_subplot(grid[0, 2]) + ax_join = fig.add_subplot(grid[1, 0:2]) + ax_scan = fig.add_subplot(grid[1, 2]) + + x = np.arange(len(labels)) + + labeled_bars( + ax_read, + read_speedup, + GREEN, + "{:.1f}x", + max(read_speedup) * 0.035, + ) + ax_read.axhline(1, color=GRAY, lw=1.2, ls="--") + ax_read.set_xticks(x, labels) + ax_read.set_ylabel("OME-Arrow speedup vs. OME-Zarr") + ax_read.set_title("Bulk image reads") + ax_read.set_ylim(0, max(read_speedup) * 1.23) + panel_label(ax_read, "A") + style_axis(ax_read) + + labeled_bars( + ax_write, + write_speedup, + BLUE, + "{:.1f}x", + max(write_speedup) * 0.04, + ) + ax_write.axhline(1, color=GRAY, lw=1.2, ls="--") + ax_write.set_xticks(x, labels) + ax_write.set_ylabel("OME-Arrow speedup vs. OME-Zarr") + ax_write.set_title("Image writes") + ax_write.set_ylim(0, max(write_speedup) * 1.30) + panel_label(ax_write, "B") + style_axis(ax_write) + + labeled_bars( + ax_storage, + storage_ratio, + GOLD, + "{:.2f}x", + max(storage_ratio) * 0.04, + ) + ax_storage.axhline(1, color=GRAY, lw=1.2, ls="--") + ax_storage.fill_between( + [-0.5, len(labels) - 0.5], + 0.8, + 1.3, + color=SHADE, + alpha=0.45, + zorder=0, + ) + ax_storage.set_xticks(x, labels) + ax_storage.set_ylabel("storage / OME-Zarr") + ax_storage.set_title("Storage footprint") + ax_storage.set_ylim(0, max(storage_ratio) * 1.35) + panel_label(ax_storage, "C") + style_axis(ax_storage) + + join_labels = [ + "Parquet path\njoin", + "OME-Arrow\nmetadata join", + "OME-Zarr native\nprofile join", + ] + join_colors = [GRAY, BROWN] + join_x = np.arange(len(join_labels)) + bars = ax_join.bar(join_x[:2], join_ms, color=join_colors, width=0.48) + for bar, value in zip(bars, join_ms, strict=True): + ax_join.text( + bar.get_x() + bar.get_width() / 2, + value + max(join_ms) * 0.07, + f"{value:.1f} ms", + ha="center", + va="bottom", + fontweight="bold", + color=INK, + ) + ax_join.text( + join_x[2], + max(join_ms) * 0.28, + "not native", + ha="center", + va="center", + color=GRAY, + fontweight="bold", + fontsize=13, + ) + ax_join.set_xticks(join_x, join_labels) + ax_join.set_xlim(-0.45, len(join_labels) - 0.55) + ax_join.set_ylabel("milliseconds") + ax_join.set_title("Profile-to-image metadata joins") + ax_join.set_ylim(0, max(join_ms) * 1.38) + panel_label(ax_join, "D", x=-0.07) + style_axis(ax_join) + + scan_labels = ["OME-\nArrow", "OME-\nTIFF", "OME-\nZarr"] + scan_colors = [BROWN, GOLD, GRAY] + scan_x = np.arange(len(scan_labels)) + bars = ax_scan.bar(scan_x, scan_ms, color=scan_colors, width=0.56) + for bar, value in zip(bars, scan_ms, strict=True): + ax_scan.text( + bar.get_x() + bar.get_width() / 2, + value + max(scan_ms) * 0.05, + f"{value:.1f}", + ha="center", + va="bottom", + fontweight="bold", + color=INK, + ) + ax_scan.set_xticks(scan_x, scan_labels) + ax_scan.set_ylabel("milliseconds") + ax_scan.set_title("Metadata scans") + ax_scan.set_ylim(0, max(scan_ms) * 1.35) + panel_label(ax_scan, "E") + style_axis(ax_scan) + + fig.suptitle( + "OME-Arrow benchmark advantage summary", + fontsize=19, + fontweight="bold", + y=0.97, + ) + + FIGURES_DIR.mkdir(exist_ok=True) + fig.subplots_adjust(top=0.88, bottom=0.10, left=0.07, right=0.98) + fig.savefig(PLOT_PNG, dpi=300, bbox_inches="tight") + plt.close(fig) + + +def main() -> None: + """Load saved benchmark summaries and regenerate the summary figure.""" + all_images = pd.read_parquet(ALL_IMAGES_SUMMARY) + joins = pd.read_parquet(JOINS_SUMMARY) + plot_advantage(all_images, joins) + print(f"Wrote {PLOT_PNG}") + + +if __name__ == "__main__": + main()