Enable virtual dataset cataloging#25
Conversation
📝 WalkthroughWalkthroughThis PR expands image registration and deregistration workflows, adds catalog-free Arrow-table scanning and profile-table publishing, supports remote TIFF URIs, broadens OME-Arrow integration, updates package exports, and adds documentation, configuration, unit tests, and network-gated integration tests. ChangesIngestion and publishing workflow expansion
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant API
participant Adapter
participant Catalog
Caller->>API: register_directory(path, catalog, namespace)
API->>Adapter: scan discovered image URI
Adapter-->>API: ScanResult
API->>Catalog: delete existing dataset rows when replace=True
API->>Catalog: append image assets and optional chunk index
API-->>Caller: WarehouseIngestResult
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
src/iceberg_bioimage/api.py (1)
32-33: ⚡ Quick winAvoid relying on private
_dataset_idacross modules.At Line 32, importing
_dataset_idfrom another module couples this API to a private implementation detail. Please promote a public helper (or move it to a shared utility module) to avoid brittle cross-file contracts.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/iceberg_bioimage/api.py` around lines 32 - 33, The import of the private symbol `_dataset_id` into this module creates a fragile cross-module dependency; refactor by exposing a public helper (e.g., rename `_dataset_id` to `get_dataset_id` or `dataset_id_for` and place it in a shared utility module) and update callers (including `delete_dataset_image_assets`) to import and call the new public helper instead of referencing the private name; ensure any tests and all references to `_dataset_id` are updated to use the new public function and adjust docstrings/comments accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/iceberg_bioimage/api.py`:
- Around line 81-88: Currently the code calls
_deregister_by_dataset_id(_dataset_id(scan_result.source_uri), ...) when
replace=True before publishing, which can permanently drop dataset rows if
publish_image_assets(...) or publish_chunk_index(...) fails; change the flow so
deletions occur only after both publish_image_assets and publish_chunk_index
complete successfully (or implement an atomic swap/transactional approach),
i.e., defer the call to _deregister_by_dataset_id until after successful returns
from publish_image_assets and publish_chunk_index (or add rollback logic in the
exception path) and ensure _dataset_id(scan_result.source_uri) is the same id
used for the deferred deletion.
In `@src/iceberg_bioimage/publishing/image_assets.py`:
- Around line 76-83: In _build_profile_schema, replace the private import
pyiceberg.io.pyarrow._pyarrow_to_schema_without_ids with the public
pyiceberg.io.pyarrow.pyarrow_to_schema and call
pyarrow_to_schema(arrow_table.schema); if that raises or indicates missing
Iceberg field IDs, pass a generated minimal name_mapping (e.g., map each field
name to a unique id) to pyarrow_to_schema so it can assign IDs—wrap the import
and call in the same try/except and raise the same RuntimeError on ImportError,
ensuring the function returns the result of pyarrow_to_schema instead of the
private helper.
In `@tests/test_iris_datasets.py`:
- Around line 208-209: The test currently asserts that any .parquet exists in
the iris_nf1 directory (parquet_files = list(iris_nf1.glob("*.parquet")) ;
assert parquet_files...), which is too loose; update the assertion to
specifically check for the presence of "profiles.parquet" in the iris_nf1
directory (e.g., verify (iris_nf1 / "profiles.parquet").exists() or assert
"profiles.parquet" is in the list of file names from
iris_nf1.glob("*.parquet")), so the test explicitly ensures profiles.parquet is
present.
- Around line 166-171: The test currently allows vacuous success when there are
no NF1 images; before calling scan_as_arrow_table and before computing dataset
IDs ensure the image list is non-empty by asserting the tiff_files variable is
truthy. Specifically, add an assertion like "assert tiff_files" before the list
comprehension that builds tables (used with scan_as_arrow_table) and likewise
before the block that computes dataset IDs/uniqueness so the tests fail if no
.tif files are found.
In `@tests/test_profile_namespace.py`:
- Around line 212-220: The test
test_iris_nf1_profiles_alias_resolves_well_and_site currently asserts
resolved["plate_id"] but should verify site resolution; update the assertion for
the resolved mapping returned by resolve_microscopy_profile_columns to assert
resolved["site_id"] is not None (or add that assertion alongside the existing
plate_id check) so the test actually validates the "site" alias (refer to
variables well_id, plate_id, site_id and the function
resolve_microscopy_profile_columns).
---
Nitpick comments:
In `@src/iceberg_bioimage/api.py`:
- Around line 32-33: The import of the private symbol `_dataset_id` into this
module creates a fragile cross-module dependency; refactor by exposing a public
helper (e.g., rename `_dataset_id` to `get_dataset_id` or `dataset_id_for` and
place it in a shared utility module) and update callers (including
`delete_dataset_image_assets`) to import and call the new public helper instead
of referencing the private name; ensure any tests and all references to
`_dataset_id` are updated to use the new public function and adjust
docstrings/comments accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4a9f2ba9-4a8c-4b0b-979b-b807e41c827b
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (22)
.pre-commit-config.yamlREADME.mddocs/src/examples/basic-workflow.pydocs/src/examples/metadata-workflow.pypyproject.tomlsrc/iceberg_bioimage/__init__.pysrc/iceberg_bioimage/adapters/ome_tiff.pysrc/iceberg_bioimage/adapters/zarr_v2.pysrc/iceberg_bioimage/api.pysrc/iceberg_bioimage/integrations/ome_arrow.pysrc/iceberg_bioimage/publishing/chunk_index.pysrc/iceberg_bioimage/publishing/image_assets.pysrc/iceberg_bioimage/validation/contracts.pytests/__init__.pytests/conftest.pytests/fakes.pytests/test_iris_datasets.pytests/test_ome_arrow_integration.pytests/test_profile_namespace.pytests/test_registration_update.pytests/test_s3_support.pytests/test_virtual_datasets.py
gwaybio
left a comment
gwaybio
left a comment
There was a problem hiding this comment.
Exciting to see this! I made several comments which should be addressed prior to merging. You'll notice the comments are concentrated in the readme, which is currently quite terse. I would recommend paying very careful attention to language and clarity, especially at this stage in the project. Paying off this debt early will be a big win long-term!
Co-Authored-By: Gregory Way <gregory.way@gmail.com>
Co-authored-by: Gregory Way <gregory.way@gmail.com>
|
Thanks @gwaybio ! After working through your comments and making changes I think this is ready for another review when there's a chance. |
There was a problem hiding this comment.
A few more comments - looking good.
The README still needs a bit of work. It's still quite terse and I'm often wondering why would I want to do this or cool, but why should I care. The README is a very important document and we need to make sure it's accurately and understandably describing the motivation for the work. It needs to self-justify, at this moment.
More README improvements, might be beyond scope of this PR, so feel free to punt for now
Co-Authored-By: Gregory Way <gregory.way@gmail.com>
Co-Authored-By: Gregory Way <gregory.way@gmail.com>
Co-Authored-By: Gregory Way <gregory.way@gmail.com>
Co-Authored-By: Gregory Way <gregory.way@gmail.com>
Co-Authored-By: Gregory Way <gregory.way@gmail.com>
|
Thank you @gwaybio for this review! Merging this in after making changes here. |
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pyproject.toml (1)
145-150: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd a CI job for
pytest -m network
test_matrixruns plainpytest, soini_options.addopts = "-m 'not network'"excludes every@pytest.mark.networktest from CI. Add a separate job/step that runs the network-marked tests explicitly to keep those integration checks covered.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pyproject.toml` around lines 145 - 150, Add a separate CI job or step that explicitly runs pytest with the network marker, using the project’s existing test configuration and environment setup. Ensure it invokes pytest with “-m network” so tests excluded by the default addopts in the pytest configuration are executed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 39-40: Update the scan description in README.md to remove the
unconditional “constant-time regardless of image size” claim, and describe
instead that metadata-only scanning typically avoids reads proportional to pixel
data while acknowledging runtime may vary.
- Around line 58-59: Update the OME-Arrow version shown in the README install
commands for the `ome-arrow` extra from >=0.0.10 to >=0.0.11, and ensure the
corresponding OME-Arrow references in the integration section use the same
minimum version.
- Around line 281-291: Update the README S3 documentation to state that Zarr
access via S3 requires the `s3` extra, or an equivalent S3 backend, because
`s3fs` is provided only by `project.optional-dependencies.s3`. Remove the claim
that Zarr S3 works without an extra install and revise the surrounding
dependency explanation accordingly.
In `@src/iceberg_bioimage/publishing/image_assets.py`:
- Around line 92-127: Update image asset publishing around
_concretize_null_columns and the table.append flow to use the loaded destination
table schema when coercing all-null columns, preserving each existing field’s
type; retain the explicit string fallback only when creating a new table. In
tests/test_profile_namespace.py lines 168-188, replace the unconditional
string-type assertion with coverage that appends an all-null column to an
existing non-string field and verifies the append succeeds with the destination
type.
---
Outside diff comments:
In `@pyproject.toml`:
- Around line 145-150: Add a separate CI job or step that explicitly runs pytest
with the network marker, using the project’s existing test configuration and
environment setup. Ensure it invokes pytest with “-m network” so tests excluded
by the default addopts in the pytest configuration are executed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 518ae4b2-8762-479b-8c50-ea35c7ee900d
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (9)
.pre-commit-config.yamlREADME.mdpyproject.tomlsrc/iceberg_bioimage/api.pysrc/iceberg_bioimage/integrations/ome_arrow.pysrc/iceberg_bioimage/publishing/chunk_index.pysrc/iceberg_bioimage/publishing/image_assets.pytests/test_profile_namespace.pytests/test_virtual_datasets.py
🚧 Files skipped from review as they are similar to previous changes (4)
- tests/test_virtual_datasets.py
- src/iceberg_bioimage/publishing/chunk_index.py
- src/iceberg_bioimage/integrations/ome_arrow.py
- src/iceberg_bioimage/api.py
| - **scan** — read an image's metadata (shape, dtype, axes, chunk layout) without | ||
| loading pixel data. Fast and constant-time regardless of image size. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Avoid promising constant-time scans.
Metadata-only scanning avoids pixel-sized reads, but runtime can still depend on metadata/page count, directory enumeration, and remote-storage latency. Replace this with wording such as “typically avoids reads proportional to pixel data.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 39 - 40, Update the scan description in README.md to
remove the unconditional “constant-time regardless of image size” claim, and
describe instead that metadata-only scanning typically avoids reads proportional
to pixel data while acknowledging runtime may vary.
| pip install 'iceberg-bioimage[ome-arrow]' # OME-Arrow image pixel-data access (≥0.0.10) | ||
| pip install 'iceberg-bioimage[s3]' # S3 / remote URI support for OME-TIFF |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Use one OME-Arrow minimum version everywhere.
The install section says >=0.0.10, while the integration section says >=0.0.11 and the PR objective identifies 0.0.11 as the upgraded requirement. Update Lines 58-59 to >=0.0.11.
Also applies to: 314-317
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 58 - 59, Update the OME-Arrow version shown in the
README install commands for the `ome-arrow` extra from >=0.0.10 to >=0.0.11, and
ensure the corresponding OME-Arrow references in the integration section use the
same minimum version.
| Zarr stores on S3 work out of the box — no extra install needed — because `zarr` already pulls in `fsspec`. | ||
| OME-TIFF on S3 requires an extra install because `tifffile` does not: | ||
|
|
||
| ```bash | ||
| pip install 'iceberg-bioimage[s3]' | ||
| ``` | ||
|
|
||
| Install it with `uv sync --group ome-arrow` or | ||
| `pip install 'iceberg-bioimage[ome-arrow]'`. | ||
| The reason for the difference: `zarr` already depends on `fsspec`, so `s3://` | ||
| paths work immediately. `tifffile` (used for OME-TIFF) does not understand | ||
| `fsspec` URLs on its own, so this package adds an `fsspec`-backed file opener | ||
| behind the `s3` extra to bridge the gap. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n 'fsspec|s3fs|zarr|\[project\.optional-dependencies\]' pyproject.tomlRepository: WayScience/iceberg-bioimage
Length of output: 223
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- pyproject.toml (dependency sections) ---'
nl -ba pyproject.toml | sed -n '1,120p'
echo
echo '--- README.md (S3 wording area) ---'
nl -ba README.md | sed -n '270,300p'Repository: WayScience/iceberg-bioimage
Length of output: 250
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import tomllib
data = tomllib.loads(Path('pyproject.toml').read_text())
print('project.dependencies:')
for dep in data.get('project', {}).get('dependencies', []):
print(' ', dep)
print('\nproject.optional-dependencies:')
for extra, deps in data.get('project', {}).get('optional-dependencies', {}).items():
print(f' [{extra}]')
for dep in deps:
print(' ', dep)
PYRepository: WayScience/iceberg-bioimage
Length of output: 388
Document the s3 extra for Zarr S3 access
zarr is a base dependency, but s3fs only comes from project.optional-dependencies.s3, so “no extra install needed” is misleading. Call out that Zarr on S3 also needs the s3 extra (or another S3 backend).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 281 - 291, Update the README S3 documentation to
state that Zarr access via S3 requires the `s3` extra, or an equivalent S3
backend, because `s3fs` is provided only by `project.optional-dependencies.s3`.
Remove the claim that Zarr S3 works without an extra install and revise the
surrounding dependency explanation accordingly.
| publish_table = _concretize_null_columns(arrow_table) | ||
|
|
||
| def _build_profile_schema() -> object: | ||
| from pyiceberg.io.pyarrow import pyarrow_to_schema | ||
| from pyiceberg.table.name_mapping import MappedField, NameMapping | ||
|
|
||
| name_mapping = NameMapping( | ||
| [ | ||
| MappedField(field_id=i + 1, names=[field.name]) | ||
| for i, field in enumerate(publish_table.schema) | ||
| ] | ||
| ) | ||
| return pyarrow_to_schema(publish_table.schema, name_mapping=name_mapping) | ||
|
|
||
| table = _load_or_create_table( | ||
| catalog, | ||
| namespace, | ||
| table_name, | ||
| schema_builder=_build_profile_schema, | ||
| ) | ||
| table.append(publish_table) | ||
| return publish_table.num_rows | ||
|
|
||
|
|
||
| def _concretize_null_columns(arrow_table: pa.Table) -> pa.Table: | ||
| """Cast all-null columns to strings for Iceberg schema compatibility.""" | ||
|
|
||
| normalized = arrow_table | ||
| for index, field in enumerate(arrow_table.schema): | ||
| if not pa.types.is_null(field.type): | ||
| continue | ||
| normalized = normalized.set_column( | ||
| index, | ||
| pa.field(field.name, pa.string(), nullable=True, metadata=field.metadata), | ||
| arrow_table.column(index).cast(pa.string()), | ||
| ) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant file and inspect the surrounding implementation.
ast-grep outline src/iceberg_bioimage/publishing/image_assets.py --view expanded || true
echo "---- image_assets.py excerpt ----"
sed -n '1,220p' src/iceberg_bioimage/publishing/image_assets.py | cat -n
echo "---- tests/test_profile_namespace.py excerpt ----"
sed -n '1,260p' tests/test_profile_namespace.py | cat -n
echo "---- search for all-null / concretize helpers ----"
rg -n "_concretize_null_columns|all-null|null columns|pyarrow_to_schema|MappedField|NameMapping|register_profile_table|append\(" src testsRepository: WayScience/iceberg-bioimage
Length of output: 29431
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "---- image_assets.py _load_or_create_table section ----"
sed -n '221,320p' src/iceberg_bioimage/publishing/image_assets.py | cat -n
echo "---- api.py register_profile_table wrapper ----"
sed -n '420,475p' src/iceberg_bioimage/api.py | cat -n
echo "---- tests/fakes.py ----"
sed -n '1,120p' tests/fakes.py | cat -nRepository: WayScience/iceberg-bioimage
Length of output: 9495
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "---- dependency versions ----"
sed -n '1,220p' pyproject.toml | cat -n
echo "---- any tests around appending to existing tables or schema mismatch ----"
rg -n "append.*schema|schema mismatch|existing table|load_table|create_table|publish_profile_table|register_profile_table" tests src
echo "---- pyarrow null-to-string / schema inference probe ----"
python3 - <<'PY'
import pyarrow as pa
t = pa.table({"a": pa.nulls(2), "b": [1, 2]})
print("source schema:", t.schema)
print("casted schema:", t.set_column(0, pa.field("a", pa.string()), t.column(0).cast(pa.string())).schema)
print("source column types:", [f.type for f in t.schema])
PYRepository: WayScience/iceberg-bioimage
Length of output: 11750
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "---- tests/test_profile_namespace.py around the remaining profile tests ----"
sed -n '236,285p' tests/test_profile_namespace.py | cat -n
echo "---- source around schema derivation helper in chunk_index for comparison ----"
sed -n '1,110p' src/iceberg_bioimage/publishing/chunk_index.py | cat -nRepository: WayScience/iceberg-bioimage
Length of output: 6239
🌐 Web query:
PyIceberg append pyarrow schema type compatibility null column cast string existing table schema
💡 Result:
In PyIceberg, schema compatibility during an append operation is validated by comparing the PyArrow schema of the input DataFrame against the Iceberg table's schema [1][2]. When attempting to append a column containing only nulls (PyArrow pa.null() type) to an existing Iceberg table, several factors influence compatibility and successful execution [3][4]. Schema Compatibility and Null Columns: 1. PyArrow Null Type: For V1 and V2 Iceberg tables, pa.null() is often not automatically promoted to the target Iceberg column type because there is no direct Iceberg type mapping for a pure null type [3]. This can lead to validation errors during append [3][5]. 2. V3 Tables: In newer Iceberg specifications (V3+), pa.null() may be mapped to UnknownType, which is intended to be promotable to any type [3]. 3. Recommended Workaround: If you encounter compatibility errors when appending a column that consists entirely of nulls, explicitly define the schema of your PyArrow DataFrame before the append operation [3]. By creating the PyArrow table with the correct target type (e.g., pa.string()) even when all values are null, you ensure that the PyArrow schema matches the Iceberg table schema expectations [3]. Example of explicit schema definition: import pyarrow as pa # Define the schema explicitly to match the Iceberg table's expected column type schema = pa.schema([ pa.field("existing_column_name", pa.string, nullable=True), ]) # Create the table with this schema, even if the data is all None df = pa.Table.from_pylist([{"existing_column_name": None}], schema=schema) # Perform the append table.append(df) Additional Considerations: - Type Promotion: PyIceberg supports type promotion on write (e.g., pa.int32() to LongType), but this applies to compatible primitive types, not necessarily from a raw null type [2][6]. - Schema Projection/Ordering: Modern PyIceberg versions (0.7.x+) handle arbitrary column ordering and subsets of the table schema automatically, meaning you do not need to order your DataFrame columns to match the Iceberg table exactly [1][2]. - Metadata Errors: Historically, issues were reported where writing columns containing only nulls caused errors in metadata collection [4]. Using recent versions of PyIceberg is recommended to ensure these edge cases are handled [1][2].
Citations:
- 1: tbl.append(df): schema validation of tbl & df during compares the order & data types apache/iceberg-python#1088
- 2: Allow writing
pa.Tablethat are either a subset of table schema or in arbitrary order, and support type promotion on write apache/iceberg-python#921 - 3: Adding promotion for UnknownType per V3+ spec apache/iceberg-python#2155
- 4: Support Null Column Write in Unpartitioned Table apache/iceberg-python#348
- 5: When writing data from a PyArrow DataFrame, how should we handle 'null' Fields? apache/iceberg-python#2119
- 6: Allow writing dataframes that are either a subset of table schema or in arbitrary order apache/iceberg-python#829
Preserve existing field types for all-null profile columns. Casting null-typed columns to string is only safe for table creation; on an existing table it can make a valid append incompatible with the table’s actual field type. Coerce all-null columns from the loaded table schema when appending, and keep the creation-time fallback explicit.
src/iceberg_bioimage/publishing/image_assets.py#L92-L127: use the destination schema for null columns before append.tests/test_profile_namespace.py#L168-L188: replace the unconditional string assertion with coverage for appending an all-null column to an existing non-string field.
📍 Affects 2 files
src/iceberg_bioimage/publishing/image_assets.py#L92-L127(this comment)tests/test_profile_namespace.py#L168-L188
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/iceberg_bioimage/publishing/image_assets.py` around lines 92 - 127,
Update image asset publishing around _concretize_null_columns and the
table.append flow to use the loaded destination table schema when coercing
all-null columns, preserving each existing field’s type; retain the explicit
string fallback only when creating a new table. In
tests/test_profile_namespace.py lines 168-188, replace the unconditional
string-type assertion with coverage that appends an all-null column to an
existing non-string field and verifies the append succeeds with the destination
type.
2 participants
Description
This PR enables virtual dataset references. We also add OME-IRIS datasets for testing and upgrade the OME-Arrow requirements of the project.
What kind of change(s) are included?
Checklist
Please ensure that all boxes are checked before indicating that this pull request is ready for review.
Summary by CodeRabbit