Populate per-asset dataStandard for BIDS, HED, NWB, and extensions by yarikoptic · Pull Request #1811 · dandi/dandi-cli

yarikoptic · 2026-02-24T22:12:34Z

Summary

Populate per-asset dataStandard field (new in dandischema) for all standards:
- BIDS: from dataset_description.json with BIDSVersion
- HED: from HEDVersion in dataset_description.json, including library schemas (e.g. "sc:1.0.0") as extensions
- NWB: from NWB file metadata, with ndx-* extensions extracted from h5file["specifications"] group
- OME/NGFF: for .ome.zarr assets in BIDS datasets
Centralize backward-compat guard as _SCHEMA_BAREASSET_HAS_DATASTANDARD in bases.py — single bool gates all new dandischema features (dataStandard, version, extensions), with RuntimeError if dandischema >= 0.12.2 unexpectedly lacks the field
Add get_nwb_extensions() to pynwb_utils.py — reads HDF5 specifications group, filters core namespaces, returns {name: latest_version} for each extension
All new code is no-op against released dandischema (< 0.12.2)

Test plan

TestBIDSDatasetDescriptionDataStandard — 5 tests covering BIDS always set, HED detected/absent, HED list form, HED library schemas as extensions
test_get_nwb_extensions — verifies ndx-* extraction with version sorting, core namespace filtering
test_get_nwb_extensions_no_specs — empty dict when no specifications group
460 tests pass with dev dandischema, 426 pass with released dandischema 0.12.1 (new feature tests correctly skip)
TODO1: add matrix run to test against dandi-schema#371, and then remove that or do it properly as a dedicated run based on description here annotation

PR in dandi-schema:

Add per-asset dataStandard, HED standard, and extensions to StandardsType dandi-schema#371

🤖 Generated with Claude Code

codecov · 2026-02-24T23:00:05Z

Codecov Report

❌ Patch coverage is 79.62963% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.04%. Comparing base (0af950f) to head (f69a338).

Files with missing lines	Patch %	Lines
dandi/files/bids.py	69.81%	16 Missing ⚠️
dandi/files/bases.py	54.54%	10 Missing ⚠️
dandi/pynwb_utils.py	75.00%	5 Missing ⚠️
dandi/tests/test_files.py	95.55%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (0af950f) and HEAD (f69a338). Click for more details.

HEAD has 85 uploads less than BASE

Flag BASE (0af950f) HEAD (f69a338)

unittests 98 13

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #1811       +/-   ##
===========================================
- Coverage   76.27%   53.04%   -23.23%     
===========================================
  Files          87       87               
  Lines       12484    12642      +158     
===========================================
- Hits         9522     6706     -2816     
- Misses       2962     5936     +2974

Flag	Coverage Δ
unittests	`53.04% <79.62%> (-23.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bendichter · 2026-03-03T19:58:58Z

 BIDS_DATASET_ERRORS = ("BIDS.MANDATORY_FILE_MISSING_PLACEHOLDER",)


+def _add_standard(


line 542 of bases appears to have similar logic

+from .bases import (
+    _SCHEMA_BAREASSET_HAS_DATASTANDARD,
+    GenericAsset,
+    LocalFileAsset,
+    NWBAsset,
+)


To break the cycle with minimal behavioral change, keep only the class needed for immediate class definition (LocalFileAsset) as a top-level import, and lazily import the other .bases symbols (GenericAsset, NWBAsset, _SCHEMA_BAREASSET_HAS_DATASTANDARD) inside the methods/functions that use them.

In the shown snippet, we can safely reduce the top-level .bases import to only LocalFileAsset. This removes the cycle-triggering eager dependency footprint from module import time for three symbols. Then, in regions of dandi/files/bids.py (not shown here) where GenericAsset, NWBAsset, or _SCHEMA_BAREASSET_HAS_DATASTANDARD are referenced, add local imports in those functions/methods. For typing-only references, prefer from __future__ import annotations (already present) and/or if TYPE_CHECKING: imports.

Within the provided lines, the concrete edit is:

Replace lines 21–27 with from .bases import LocalFileAsset.

Replace the conditional block at lines 28–31 to avoid using _SCHEMA_BAREASSET_HAS_DATASTANDARD at module import time (set a conservative default and load actual flag lazily where needed).

bendichter · 2026-03-03T20:09:54Z

@yarikoptic lgtm! Let me just check on some of my own data

bendichter · 2026-03-03T20:23:11Z

Added NWB spec version extraction in 279c297.

get_nwb_version() already existed in pynwb_utils.py but wasn't being used when constructing the StandardsType. Now the version is populated from the HDF5 nwb_version attribute.

Tested against dandiset 001636 (346 NWB files, macaque electrophysiology):

Before:

Neurodata Without Borders (NWB) (version=None)
  ext: ndx-events v0.4.0
  ext: ndx-hed v0.2.0

After:

Neurodata Without Borders (NWB) (version=2.9.0)
  ext: ndx-events v0.4.0
  ext: ndx-hed v0.2.0

bendichter

ok, I made a minor correction. LGTM now!

- BIDSDatasetDescriptionAsset.get_metadata(): always sets BIDS standard (with BIDSVersion), adds HED when HEDVersion present in JSON, warns on read failure - NWBAsset.get_metadata(): sets NWB standard - ZarrBIDSAsset.get_metadata(): sets OME/NGFF for .ome.zarr assets - Guard for older dandischema without dataStandard on BareAsset; RuntimeError if dandischema >= 0.12.2 lacks it - Register ai_generated pytest marker in tox.ini Requires dandischema with per-asset dataStandard support (0.12.2+). Works silently with older dandischema. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add get_nwb_extensions() to pynwb_utils that reads h5file["specifications"] to discover ndx-* namespaces and their versions (filtering out core/hdmf) - NWBAsset.get_metadata() populates StandardsType.extensions with ndx-* extensions found in the NWB file - BIDSDatasetDescriptionAsset.get_metadata() extracts HED library schemas from list-valued HEDVersion (e.g. ["8.2.0", "sc:1.0.0"]) into extensions - Add tests for both NWB extension extraction and HED library schema parsing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…S_DATASTANDARD Single bool in bases.py guards all dataStandard/version/extensions features that ship together in dandischema >= 0.12.2, replacing scattered "field in Model.model_fields" checks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move json, h5py, get_nwb_extensions, and _SCHEMA_BAREASSET_HAS_DATASTANDARD imports to module top level in tests. Keep pynwb_utils import deferred in bases.py (heavy transitive deps: h5py/pynwb/hdmf/numpy) per existing convention. Add import guidance to CLAUDE.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

hed_standard does not exist in dandischema < 0.12.2, so gate the import on _SCHEMA_BAREASSET_HAS_DATASTANDARD (all new symbols ship together). HED detection is skipped when unavailable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move _SCHEMA_BAREASSET_HAS_DATASTANDARD guard block after all imports to fix E402. Use getattr/setattr for dataStandard access and type: ignore[call-arg] for StandardsType(version=...) to satisfy mypy against released dandischema that lacks these fields. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use the existing get_nwb_version() to populate the version field on the NWB StandardsType entry so per-asset metadata reports the NWB spec version (e.g. 2.9.0) alongside extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

This enables CI to test against the unreleased dandischema that has the dataStandard field and StandardsType enhancements. TO BE REVERTED before merge. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

yarikoptic mentioned this pull request Feb 24, 2026

Add per-asset dataStandard, HED standard, and extensions to StandardsType dandi/dandi-schema#371

Open

4 tasks

github-advanced-security AI found potential problems Feb 24, 2026

View reviewed changes

Comment thread dandi/files/bids.py Fixed

yarikoptic added the minor Increment the minor version when merged label Mar 3, 2026

bendichter reviewed Mar 3, 2026

View reviewed changes

yarikoptic force-pushed the hed branch from d000be8 to 4f7d154 Compare March 3, 2026 20:07

bendichter reviewed Mar 3, 2026

View reviewed changes

Comment thread dandi/files/bases.py Outdated

github-advanced-security AI found potential problems Mar 3, 2026

View reviewed changes

bendichter self-requested a review March 3, 2026 20:23

bendichter approved these changes Mar 3, 2026

View reviewed changes

yarikoptic and others added 9 commits April 13, 2026 14:39

chore: git ignore uv.lock

e4d9537

Extract NWB spec version into dataStandard

7b0416f

Use the existing get_nwb_version() to populate the version field on the NWB StandardsType entry so per-asset metadata reports the NWB spec version (e.g. 2.9.0) alongside extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

TEMP: install dandischema from dandi-schema#371 (enh-hed branch)

f69a338

This enables CI to test against the unreleased dandischema that has the dataStandard field and StandardsType enhancements. TO BE REVERTED before merge. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

yarikoptic force-pushed the hed branch from c17d42c to f69a338 Compare April 13, 2026 18:39

@@ -18,17 +18,12 @@
             import dandi
             from dandi.bids_validator_deno import bids_validate
-            from .bases import (
-                _SCHEMA_BAREASSET_HAS_DATASTANDARD,
-                GenericAsset,
-                LocalFileAsset,
-                NWBAsset,
-            )
+            from .bases import LocalFileAsset
-            if _SCHEMA_BAREASSET_HAS_DATASTANDARD:
-                from dandischema.models import hed_standard  # type: ignore[attr-defined]
-            else:
-                hed_standard = None  # type: ignore[assignment]
+            # Avoid eager access to .bases module state at import time to reduce cycle pressure.
+            # Resolve schema capability lazily at usage sites.
+            _SCHEMA_BAREASSET_HAS_DATASTANDARD = False
+            hed_standard = None  # type: ignore[assignment]
             from .zarr import ZarrAsset
             from ..consts import ZARR_MIME_TYPE, dandiset_metadata_file
             from ..metadata.core import add_common_metadata, prepare_metadata

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions#1811

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions#1811
yarikoptic wants to merge 9 commits intomasterfrom
hed

yarikoptic commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

codecov Bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

bendichter Mar 3, 2026

Uh oh!

Uh oh!

Check notice

Copilot Autofix

bendichter commented Mar 3, 2026

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

bendichter left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		BIDS_DATASET_ERRORS = ("BIDS.MANDATORY_FILE_MISSING_PLACEHOLDER",)


		def _add_standard(

Conversation

yarikoptic commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Uh oh!

codecov Bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bendichter Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Check notice

Uh oh!

Copilot Autofix

bendichter commented Mar 3, 2026

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

bendichter left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yarikoptic commented Feb 24, 2026 •

edited

Loading

codecov Bot commented Feb 24, 2026 •

edited

Loading