Skip to content

Convert ros3 streaming to remfile in User Guide Part I and NWBWidget demo #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bendichter merged 2 commits into from
Aug 20, 2026
Merged

Convert ros3 streaming to remfile in User Guide Part I and NWBWidget demo #185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2fccdaf
Convert ros3 streaming to remfile in User Guide Part I and NWBWidget …
bendichter Aug 20, 2026
4b9aa81
Split remfile wrapping, add DiskCache, backfill requirements.in
bendichter Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 26 additions & 77 deletions dandi/DANDI User Guide, Part I.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,145 +39,92 @@
" \"aiosignal==1.4.0\" \\\n",
" \"annotated-types==0.7.0\" \\\n",
" \"arrow==1.4.0\" \\\n",
" \"asciitree==0.3.3\" \\\n",
" \"attrs==26.1.0\" \\\n",
" \"backcall==0.2.0\" \\\n",
" \"bids-validator-deno==2.4.1\" \\\n",
" \"bidsschematools==1.2.2\" \\\n",
" \"bids-validator-deno==3.0.1\" \\\n",
" \"bidsschematools==1.2.7\" \\\n",
" \"blessed==1.38.0\" \\\n",
" \"bqplot==0.12.45\" \\\n",
" \"certifi==2026.4.22\" \\\n",
" \"cffi==2.0.0\" \\\n",
" \"charset-normalizer==3.4.7\" \\\n",
" \"ci-info==0.4.0\" \\\n",
" \"click==8.1.8\" \\\n",
" \"click==8.3.3\" \\\n",
" \"click-didyoumean==0.3.1\" \\\n",
" \"comm==0.2.3\" \\\n",
" \"contourpy==1.3.3\" \\\n",
" \"cryptography==43.0.3\" \\\n",
" \"cycler==0.12.1\" \\\n",
" \"dandi==0.74.3\" \\\n",
" \"dandischema==0.12.1\" \\\n",
" \"decorator==4.4.2\" \\\n",
" \"deprecated==1.3.1\" \\\n",
" \"dandi==0.77.0\" \\\n",
" \"dandischema==0.14.0\" \\\n",
" \"deno==2.9.5\" \\\n",
" \"dnspython==2.8.0\" \\\n",
" \"donfig==0.8.1.post1\" \\\n",
" \"email-validator==2.3.0\" \\\n",
" \"etelemetry==0.3.1\" \\\n",
" \"fasteners==0.20\" \\\n",
" \"fonttools==4.62.1\" \\\n",
" \"fqdn==1.5.1\" \\\n",
" \"frozenlist==1.8.0\" \\\n",
" \"fscacher==0.4.4\" \\\n",
" \"fsspec==2025.3.0\" \\\n",
" \"gast==0.7.0\" \\\n",
" \"google-crc32c==1.8.0\" \\\n",
" \"h5py==3.16.0\" \\\n",
" \"hdmf==3.14.6\" \\\n",
" \"hdmf-zarr==0.11.3\" \\\n",
" \"hdmf==6.2.0\" \\\n",
" \"humanize==4.15.0\" \\\n",
" \"idna==3.13\" \\\n",
" \"imageio==2.37.3\" \\\n",
" \"importlib-metadata==4.13.0\" \\\n",
" \"interleave==0.3.0\" \\\n",
" \"ipydatagrid==1.4.0\" \\\n",
" \"ipydatawidgets==4.3.2\" \\\n",
" \"ipyfilechooser==0.6.0\" \\\n",
" \"ipympl==0.10.0\" \\\n",
" \"ipython==7.34.0\" \\\n",
" \"ipython-genutils==0.2.0\" \\\n",
" \"ipyvolume==0.6.3\" \\\n",
" \"ipyvue==1.12.0\" \\\n",
" \"ipyvuetify==1.11.3\" \\\n",
" \"ipywebrtc==0.6.0\" \\\n",
" \"ipywidgets==8.1.8\" \\\n",
" \"isodate==0.7.2\" \\\n",
" \"isoduration==20.11.0\" \\\n",
" \"jaraco-classes==3.4.0\" \\\n",
" \"jaraco-context==6.1.2\" \\\n",
" \"jaraco-functools==4.4.0\" \\\n",
" \"jedi==0.20.0\" \\\n",
" \"jeepney==0.9.0\" \\\n",
" \"joblib==1.5.3\" \\\n",
" \"jsonpointer==3.1.1\" \\\n",
" \"jsonschema==4.26.0\" \\\n",
" \"jsonschema-specifications==2025.9.1\" \\\n",
" \"jupyterlab-widgets==3.0.16\" \\\n",
" \"keyring==25.7.0\" \\\n",
" \"keyrings-alt==5.0.2\" \\\n",
" \"kiwisolver==1.5.0\" \\\n",
" \"lazy-loader==0.5\" \\\n",
" \"matplotlib==3.10.0\" \\\n",
" \"matplotlib-inline==0.2.1\" \\\n",
" \"ml-dtypes==0.5.4\" \\\n",
" \"more-itertools==10.8.0\" \\\n",
" \"multidict==6.7.1\" \\\n",
" \"natsort==8.4.0\" \\\n",
" \"ndx-grayscalevolume==0.0.2\" \\\n",
" \"ndx-icephys-meta==0.1.0\" \\\n",
" \"ndx-spectrum==0.2.2\" \\\n",
" \"networkx==3.6.1\" \\\n",
" \"numcodecs==0.15.1\" \\\n",
" \"numcodecs==0.16.5\" \\\n",
" \"numpy==2.0.2\" \\\n",
" \"nwbinspector==0.6.3\" \\\n",
" \"nwbwidgets==0.11.3\" \\\n",
" \"nwbinspector==0.7.2\" \\\n",
" \"packaging==26.1\" \\\n",
" \"pandas==2.2.2\" \\\n",
" \"parso==0.8.6\" \\\n",
" \"pexpect==4.9.0\" \\\n",
" \"pickleshare==0.7.5\" \\\n",
" \"pillow==11.3.0\" \\\n",
" \"platformdirs==4.9.6\" \\\n",
" \"plotly==5.13.1\" \\\n",
" \"prompt-toolkit==3.0.52\" \\\n",
" \"propcache==0.4.1\" \\\n",
" \"ptyprocess==0.7.0\" \\\n",
" \"py2vega==0.7.0\" \\\n",
" \"pycparser==3.0\" \\\n",
" \"pycryptodomex==3.23.0\" \\\n",
" \"pydantic==2.12.3\" \\\n",
" \"pydantic-core==2.41.4\" \\\n",
" \"pydantic-settings==2.14.0\" \\\n",
" \"pygments==2.20.0\" \\\n",
" \"pynwb==2.8.3\" \\\n",
" \"pynwb==4.1.0\" \\\n",
" \"pyout==0.8.1\" \\\n",
" \"pyparsing==3.3.2\" \\\n",
" \"python-dateutil==2.9.0.post0\" \\\n",
" \"python-dotenv==1.2.2\" \\\n",
" \"pythreejs==2.4.2\" \\\n",
" \"pytz==2025.2\" \\\n",
" \"pyyaml==6.0.3\" \\\n",
" \"referencing==0.37.0\" \\\n",
" \"remfile==0.1.14\" \\\n",
" \"requests==2.32.4\" \\\n",
" \"rfc3339-validator==0.1.4\" \\\n",
" \"rfc3987==1.3.8\" \\\n",
" \"rpds-py==0.30.0\" \\\n",
" \"ruamel-yaml==0.19.1\" \\\n",
" \"scikit-image==0.25.2\" \\\n",
" \"scipy==1.16.3\" \\\n",
" \"secretstorage==3.5.0\" \\\n",
" \"semantic-version==2.10.0\" \\\n",
" \"setuptools==75.2.0\" \\\n",
" \"six==1.17.0\" \\\n",
" \"tenacity==9.1.4\" \\\n",
" \"tensorstore==0.1.82\" \\\n",
" \"threadpoolctl==3.6.0\" \\\n",
" \"tifffile==2026.4.11\" \\\n",
" \"tqdm==4.67.3\" \\\n",
" \"traitlets==5.7.1\" \\\n",
" \"traittypes==0.2.3\" \\\n",
" \"trimesh==4.12.2\" \\\n",
" \"typing-extensions==4.15.0\" \\\n",
" \"typing-inspection==0.4.2\" \\\n",
" \"tzdata==2026.1\" \\\n",
" \"uri-template==1.3.0\" \\\n",
" \"urllib3==2.5.0\" \\\n",
" \"wcwidth==0.6.0\" \\\n",
" \"webcolors==25.10.0\" \\\n",
" \"widgetsnbextension==4.0.15\" \\\n",
" \"wrapt==2.1.2\" \\\n",
" \"yarl==1.23.0\" \\\n",
" \"zarr==2.18.7\" \\\n",
" \"zarr-checksum==0.4.7\" \\\n",
" \"zipp==3.23.1\""
" \"zarr==3.1.5\" \\\n",
" \"zarr-checksum==0.4.7\""
]
},
{
Expand Down Expand Up @@ -456,9 +403,9 @@
"id": "7d980104-839e-44ae-8031-ae3d20517eb7",
"metadata": {},
"source": [
"## Visualization NWB files\n",
"## Inspecting NWB Files\n",
"\n",
"NWB Widgets is a library of interactive data visualizations that works automatically with any NWB file. This can be very useful for visually confirming any conversion."
"PyNWB objects render a rich, expandable summary in Jupyter, which is a quick way to visually confirm the contents of a conversion."
]
},
{
Expand All @@ -468,17 +415,15 @@
"metadata": {},
"outputs": [],
"source": [
"from nwbwidgets import nwb2widget\n",
"\n",
"nwb2widget(nwbfile)"
"nwbfile"
]
},
{
"cell_type": "markdown",
"id": "1fa27484-8e53-4a6d-a755-a5718806c69f",
"metadata": {},
"source": [
"It can also be used to explore any file shared on DANDI. You can use the DANDI API to access the s3 path of any file and stream it directly into NWB Widgets."
"The same works for any file shared on DANDI. You can use the DANDI API to access the S3 path of any file and stream it directly, inspecting it without downloading the whole file."
]
},
{
Expand Down Expand Up @@ -508,17 +453,21 @@
"metadata": {},
"outputs": [],
"source": [
"import h5py\n",
"import remfile\n",
"from dandi.dandiapi import DandiAPIClient\n",
"from pynwb import NWBHDF5IO\n",
"from nwbwidgets import nwb2widget\n",
"\n",
"with DandiAPIClient() as client:\n",
" asset = client.get_dandiset(dandiset_id, \"draft\").get_asset_by_path(filepath)\n",
" s3_url = asset.get_content_url(follow_redirects=1, strip_query=True)\n",
"\n",
"io = NWBHDF5IO(s3_url, mode='r', load_namespaces=True, driver='ros3')\n",
"# cache reads on disk so re-running cells doesn't re-fetch the same bytes\n",
"rem_file = remfile.File(s3_url, disk_cache=remfile.DiskCache(\"nwb-cache\"))\n",
"h5py_file = h5py.File(rem_file, \"r\")\n",
"io = NWBHDF5IO(file=h5py_file, load_namespaces=True)\n",
"nwb = io.read()\n",
"nwb2widget(nwb)"
"nwb"
]
}
],
Expand Down
6 changes: 6 additions & 0 deletions dandi/DANDI User Guide, Part I.requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
dandi>=0.74.0
h5py
numpy
pynwb
python-dateutil
remfile
Loading
Loading