iceberg-bioimage

iceberg-bioimage reads the metadata of OME-TIFF and OME-Zarr image files (shape, dtype, axes, chunking — not pixel data) into Arrow tables that can be queried directly or cataloged in Apache Iceberg.

Why this exists

Image-based profiling workflows often keep microscopy images, image metadata, single-cell profiles, and quality-control outputs in separate files or systems. That makes it easy for tables to drift apart from the images they describe and hard to ask simple questions like "which image produced this profile row?"

iceberg-bioimage gives those workflows a lightweight metadata layer:

• scan image stores without loading pixel data;
• represent experiments made from one file, many files, or a mix of supported formats such as OME-TIFF and OME-Zarr;
• join image metadata to Cytomining profile tables using stable keys;
• catalog many image stores in Apache Iceberg for data-lake scale, versioned snapshots, schema evolution, and access from Iceberg-compatible query engines and views;
• keep metadata in Arrow-friendly tables that query engines can process with vectorized execution and hardware-aware optimizations such as SIMD;
• export portable Parquet warehouse layouts for tools that do not use Iceberg.

This README follows that metadata-first path. Pixel-data workflows are covered later in the OME-Arrow section because they require an optional integration and pay a separate conversion cost.

Two terms used throughout this README

• scan — read an image's metadata (shape, dtype, axes, chunk layout) without loading pixel data. Fast and constant-time regardless of image size.
• store — a single image dataset this package can scan: one Zarr directory, or one OME-TIFF file, on local disk or remote (S3) storage.

Note on "registration": in this README, "register"/"registration" means recording an image's metadata as a row in an Iceberg table, not the bioimage-analysis sense of spatially aligning two images.

Install

pip install iceberg-bioimage

Optional extras:

pip install 'iceberg-bioimage[duckdb]'     # DuckDB query helpers
pip install 'iceberg-bioimage[ome-arrow]'  # OME-Arrow image pixel-data access (≥0.0.10)
pip install 'iceberg-bioimage[s3]'         # S3 / remote URI support for OME-TIFF

---

Usage

1 — Virtual datasets (no catalog required)

Scan any image directly into an Arrow table. No Iceberg catalog, no setup. This is the fastest path to start exploring a dataset: only the file's metadata is read, so it works the same whether the image is 1 MB or 100 GB.

from iceberg_bioimage import scan_as_arrow_table

table = scan_as_arrow_table("data/experiment.ome.tiff")
# or: scan_as_arrow_table("data/experiment.zarr")
# or: scan_as_arrow_table("s3://bucket/experiment.ome.tiff")

print(table.schema)
# dataset_id: string
# image_id: string
# format_family: string
# uri: string
# array_path: null
# shape_json: string
# dtype: string
# chunk_shape_json: null
# metadata_json: string

The result is a standard pyarrow.Table — inspect it with Pandas, query it with DuckDB, or pass it directly to downstream tools.

df = table.to_pandas()
print(df[["dataset_id", "shape_json", "dtype"]])

import duckdb

duckdb.query("SELECT dataset_id, shape_json FROM table WHERE dtype = 'uint16'")

2 — Cataloging images in Iceberg

Once you have many datasets and want to query across them, write their metadata into a persistent Apache Iceberg catalog instead of re-scanning files each time. See docs/src/catalog-setup.md for catalog configuration.

from iceberg_bioimage import register_store

register_store("data/experiment.zarr", "default", "myproject.images")

Register a whole directory when an experiment writes one image store per well, field, plate, or time point. This creates one cataloged metadata row per store, so you can query the full experiment later without manually registering or rescanning each image file.

from iceberg_bioimage import register_directory

register_directory(
    "data/plates/",
    "default",
    "myproject.images",
    glob="**/*.ome.tiff",   # default is **/*.ome.zarr
)

Use replace=True when an image has changed and you want to refresh its catalog metadata. The old rows are removed first, so later queries use the latest shape, dtype, chunking, and file location without duplicate entries.

register_store("data/experiment.zarr", "default", "myproject.images", replace=True)

Remove a dataset's rows from the catalog entirely:

from iceberg_bioimage import deregister_store

deregister_store("data/experiment.zarr", "default", "myproject.images")

Pass chunk_index_table=None to skip writing the optional chunk-index table. The chunk-index table records Zarr array chunk coordinates that can be queried later for region-level access. TIFF scans do not expose a Zarr-style chunk grid, so TIFF-only registrations have no chunk rows to write; skipping the table avoids an unused table and unnecessary writes:

register_store("data/experiment.ome.tiff", "default", "myproject.images", chunk_index_table=None)

→ details: docs/src/catalog-setup.md

3 — Profile tables

A "profile table" here means a CellProfiler / Pycytominer measurements file (cell counts, intensities, shape features, etc.), keyed by image and/or well. Publish one into the catalog under a profiles namespace:

from iceberg_bioimage import register_profile_table

register_profile_table(
    "data/profiles.parquet",
    "default",
    "myproject.profiles",   # conventional: <experiment>.profiles
)

Tools in the Cytomining ecosystem do not all use the same metadata column names (Image_Metadata_Well, Metadata_Plate, Metadata_Site, ...). The join helpers need stable keys such as well_id and plate_id to match profile rows to image metadata. "Alias resolution" maps common variants onto those canonical names so joins work without you renaming columns by hand.

Before registering, you can check whether a profile table has everything needed to join against image metadata. This package's microscopy join contract is the set of canonical columns a join needs: dataset_id and image_id are required, plate_id/well_id/site_id are recommended but optional.

from iceberg_bioimage import validate_microscopy_profile_table

result = validate_microscopy_profile_table("data/profiles.parquet")

print(result.is_valid)
# False if dataset_id/image_id can't be found or aliased

print(result.missing_required_columns)
# e.g. ['dataset_id', 'image_id'] — columns the join contract needs but
# this table doesn't have, even after alias matching

print(result.warnings)
# e.g. ["well_id resolved from Image_Metadata_Well"] — non-standard column
# names that were matched to canonical join keys

4 — Joining images to profiles

from iceberg_bioimage import join_profiles_with_store

joined = join_profiles_with_store("data/experiment.zarr", "data/profiles.parquet")
print(joined.num_rows)

Alias resolution means matching known profile-table column variants, such as Metadata_ImageID, to the canonical join keys used by this package, such as image_id.

When dataset_id is absent from the profile table but all rows belong to one dataset, supply it directly instead of relying on alias resolution:

joined = join_profiles_with_store(
    "data/experiment.zarr",
    "data/profiles.parquet",
    profile_dataset_id="experiment",
)

5 — Cytomining warehouse export

Export images and profiles into a Parquet warehouse layout that Pycytominer and CytoTable can consume directly, without going through Iceberg at all. Choose this path when downstream tools or collaborators expect plain Parquet files on disk, or when you want a portable analysis bundle without requiring an Iceberg catalog service. See docs/src/cytomining.md for full warehouse export workflows.

from iceberg_bioimage import export_store_to_cytomining_warehouse

export_store_to_cytomining_warehouse(
    "data/experiment.zarr",
    "warehouse-root",
    profiles="data/profiles.parquet",
    profile_dataset_id="experiment",
)

This writes:

warehouse-root/
  images/image_assets/
  profiles/joined_profiles/

→ details: docs/src/cytomining.md

6 — CLI

A command-line wrapper around the same functions, for use in shell scripts and CI pipelines where writing a Python file isn't worth it. Each subcommand mirrors a Python function from the sections above:

iceberg-bioimage scan data/experiment.zarr
iceberg-bioimage summarize data/experiment.zarr
iceberg-bioimage register --catalog default --namespace myproject.images data/experiment.zarr
iceberg-bioimage ingest --catalog default --namespace myproject.images data/a.zarr data/b.zarr
iceberg-bioimage export-cytomining --warehouse-root warehouse-root data/experiment.zarr
iceberg-bioimage validate-contract data/profiles.parquet
iceberg-bioimage join-profiles data/experiment.zarr data/profiles.parquet --output joined.parquet

---

S3 and remote URIs

Zarr stores on S3 work out of the box — no extra install needed — because zarr already pulls in fsspec. OME-TIFF on S3 requires an extra install because tifffile does not:

pip install 'iceberg-bioimage[s3]'

The reason for the difference: zarr already depends on fsspec, so s3:// paths work immediately. tifffile (used for OME-TIFF) does not understand fsspec URLs on its own, so this package adds an fsspec-backed file opener behind the s3 extra to bridge the gap.

table = scan_as_arrow_table("s3://my-bucket/plates/experiment.ome.tiff")
register_directory("s3://my-bucket/plates/", catalog, "myproject.images")

---

Namespace layout

A typical experiment in the catalog looks like:

myproject.images   → image_assets table  (one row per image file)
                   → chunk_index table   (optional; Zarr chunked arrays only)
myproject.profiles → profiles table      (pycytominer measurements)

---

OME-Arrow integration

Everything above deals with metadata — shape, dtype, file location. If you also need the pixel data itself stored as an Arrow table (for example, to pass raw pixel arrays into an ML pipeline without re-reading the original file), use the optional OME-Arrow integration (requires ome-arrow >= 0.0.11):

from iceberg_bioimage import (
    create_ome_arrow_from_tiff,
    create_ome_arrow_from_zarr,
    open_ome_arrow_dataset,
    write_ome_arrow_dataset,
)

Conversion note: create_ome_arrow_from_zarr/create_ome_arrow_from_tiff materialize source pixels to build Arrow values. For large images, expect the conversion to take roughly as long as reading the full pixel array once, plus Arrow encoding overhead. Peak memory and output size scale with the pixel payload, so a 10 GB uncompressed image should be treated as a multi-GB conversion. Prefer open_ome_arrow_dataset to read an already-converted OME-Arrow dataset without re-paying that cost.

Install with pip install 'iceberg-bioimage[ome-arrow]'.

---

DuckDB helpers

Optional SQL-style filtering and joins over registered metadata tables, useful when you want ad-hoc queries (e.g. "all images with cell_count > 10") without writing pandas/pyarrow code by hand:

from iceberg_bioimage import join_image_assets_with_profiles, query_metadata_table

# Arrow tables loaded from your catalog, warehouse export, or local Parquet files.
image_assets_table = ...
profiles_table = ...

# joined is an Arrow table containing profile rows plus matching image metadata.
joined = join_image_assets_with_profiles(image_assets_table, profiles_table)
filtered = query_metadata_table(joined, filters=[("cell_count", ">", 10)])

Install with pip install 'iceberg-bioimage[duckdb]'.

---

Troubleshooting

Problem	Fix
DuckDB helpers require the optional duckdb dependency	pip install 'iceberg-bioimage[duckdb]'
fsspec is required to open remote TIFF URIs	pip install 'iceberg-bioimage[s3]'
Profile fails join contract (missing dataset_id)	Pass profile_dataset_id= to join helpers, or check validate_microscopy_profile_table() for alias suggestions
Missing table: ... for catalog-backed paths	Check catalog config, namespace spelling, and table names

---

Documentation

• docs/src/getting-started.md — first-time setup
• docs/src/catalog-setup.md — catalog configuration
• docs/src/cytomining.md — warehouse export workflows
• docs/src/warehouse-spec.md — warehouse interoperability specification
• examples/quickstart.py — minimal scan, publish, validation
• examples/catalog_duckdb.py — catalog-backed query workflow
• examples/synthetic_workflow.py — catalog-free local workflow