Publish verified notebook environment images to GHCR; requirements.in submission contract

.github/docker/Dockerfile

@@ -0,0 +1,51 @@
+# syntax=docker/dockerfile:1
+# Environment image for DANDI example notebooks. Built by
+# .github/workflows/build-notebook-images.yml from a context prepared by
+# .github/scripts/build_notebook_image.py (requirements.txt + work/).
+#
+# Two separate Python environments by design:
+#   - the kernel env is the system Python (/usr/local): pins go in with
+#     `uv pip install --system`, matching both the Colab bootstrap cell and
+#     run_notebook.py, so the CI harness runs unmodified inside the image;
+#   - JupyterLab lives in an isolated `uv tool` env so its own dependency
+#     tree can never upgrade or downgrade anything in the pinned kernel env.
+
+ARG BASE_IMAGE=python:3.12-slim@sha256:2c941e860699f878900b0edc2403613c234d4b32eda3cc9fa7036991a2a63c4a
+FROM ${BASE_IMAGE}
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends curl git ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=ghcr.io/astral-sh/uv:0.12.5 /uv /uvx /usr/local/bin/
+
+ARG JUPYTERLAB_VERSION=4.6.3
+ENV UV_TOOL_BIN_DIR=/opt/uv-bin
+RUN uv tool install "jupyterlab==${JUPYTERLAB_VERSION}"
+# JUPYTER_PATH makes the pinned kernelspec under /usr/local win over the
+# kernelspec that ships inside the JupyterLab tool env.
+ENV PATH="/opt/uv-bin:${PATH}" \
+    JUPYTER_PATH=/usr/local/share/jupyter
+
+# ipykernel and nbformat resolve jointly with the pins in one invocation so
+# any conflict fails the build instead of silently changing a pinned version.
+COPY requirements.txt /tmp/requirements.txt
+RUN uv pip install --system --no-cache -r /tmp/requirements.txt ipykernel nbformat \
+    && python -m ipykernel install --name python3 --display-name "Python 3 (pinned)"
+
+WORKDIR /work
+COPY work/ /work/
+
+ARG DEFAULT_NOTEBOOK
+ARG BUILD_HASH
+ARG GIT_SHA=unknown
+ARG NOTEBOOKS=""
+ENV DEFAULT_NOTEBOOK=${DEFAULT_NOTEBOOK}
+LABEL org.opencontainers.image.source="https://github.com/dandi/example-notebooks" \
+      org.opencontainers.image.revision="${GIT_SHA}" \
+      org.dandiarchive.notebooks="${NOTEBOOKS}" \
+      org.dandiarchive.build-hash="${BUILD_HASH}"
+
+EXPOSE 8888
+CMD jupyter-lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root \
+    --ServerApp.default_url="/lab/tree/${DEFAULT_NOTEBOOK}"

.github/docker/README.md

@@ -0,0 +1,78 @@
+# Containerized Notebook Environments
+
+Each Colab-ready notebook in this repository is also published as a
+self-contained container image on the GitHub Container Registry. The image
+bundles the notebook, any helper files it fetches, a JupyterLab server, and
+the exact pinned dependency set from the notebook's install cell,
+preinstalled. Unlike the Colab path, the image does not depend on Colab's
+current Python version or on PyPI still serving the pinned packages, so it
+remains runnable long after the hosted environments have moved on.
+
+## Running a Notebook Image
+
+```
+docker run --rm -p 8888:8888 ghcr.io/dandi/example-notebooks/001550-paganlab:latest
+```
+
+Then open the `http://127.0.0.1:8888/lab?token=...` URL printed in the
+terminal. JupyterLab opens on the notebook with its dependencies already
+installed; the install cell at the top is a no-op and can be skipped. The
+notebooks stream data from the DANDI Archive, so network access is still
+required at run time.
+
+Images are built for `linux/amd64`, the platform the dependency pins were
+resolved for. On Apple Silicon, Docker Desktop runs them under emulation;
+pass `--platform linux/amd64` to silence the platform warning.
+
+Images are named after the notebook directory (lowercased, with `/` replaced
+by `-`). When notebooks in the same directory pin different dependency sets,
+the notebook name is appended, e.g.
+`ghcr.io/dandi/example-notebooks/000971-lernerlab-seiler-2024-optogenetics-example-notebook`.
+
+## Tags
+
+| Tag | Meaning |
+| --- | --- |
+| `latest` | most recent verified build |
+| `YYYY-MM-DD` | date-stamped snapshot, useful as a citable reference |
+| `sha-<short>` | git commit the image was built from |
+| `hash-<12 hex>` | content hash of the build inputs (pins, notebooks, helpers, Dockerfile); used by CI to skip unchanged rebuilds |
+
+## How Images Are Built and Verified
+
+The `Build notebook images` workflow
+(`.github/workflows/build-notebook-images.yml`) groups notebooks by directory
+and pin set (`.github/scripts/build_notebook_image.py`), builds one image per
+group from the parameterized `Dockerfile` in this directory, and then runs
+every notebook in the group **inside the candidate image** using the same
+harness the test workflows use (`.github/scripts/run_notebook.py`). An image
+is pushed only if all of its notebooks execute successfully, so `latest` is
+always a verified snapshot.
+
+Inside the image the kernel environment is the system Python, with pins
+installed via `uv pip install --system`, exactly matching the Colab bootstrap
+cell and the CI harness. JupyterLab runs from an isolated `uv tool`
+environment so its own dependencies cannot perturb the pinned kernel
+environment. `ipykernel` and `nbformat` are the only additions to the pinned
+set; they resolve jointly with the pins so a conflict fails the build rather
+than silently changing a pinned version.
+
+## Maintainer Notes
+
+- The workflow is currently `workflow_dispatch` only. The `filter` input is a
+  regex on the notebook directory or path (default: the `001550/PaganLab`
+  pilot); `push: false` gives a dry run (build + verify, no publish); `force`
+  rebuilds even when the build-hash check says the published image is current.
+- Rebuilds are skipped when a `hash-<...>` tag matching the current build
+  inputs already exists on the registry. Changing a notebook, its pins, its
+  helpers, or the Dockerfile changes the hash and triggers a rebuild. Pins are
+  refreshed with `.github/scripts/lock_notebook.py` from the `requirements.in`
+  committed next to the notebook (see `docs/adding-notebooks.md`).
+- The base image is digest-pinned in the Dockerfile (`ARG BASE_IMAGE`). To
+  bump it, update the digest, which changes every group's build hash, and
+  dispatch a full rebuild. The `BASE_IMAGE` arg is also the knob for a future
+  variant based on the official Colab runtime image.
+- Images run as root (`python:3.12-slim` has no unprivileged user) and keep
+  Jupyter's token auth enabled. The published port binding in the docs is
+  loopback-only via `-p 8888:8888` on a local machine; advise users not to
+  bind on public interfaces.

.github/scripts/build_notebook_image.py

@@ -0,0 +1,233 @@
+"""Group notebooks into container-image build units and prepare build contexts.
+
+Notebooks that carry the Colab-bootstrap install cell are grouped by
+(directory, pin-set): notebooks in the same directory whose install cells pin
+the identical dependency set share one image. The image for a group contains
+every file of that directory (minus notebooks belonging to other groups) plus
+any helper files the install cells fetch via `!curl`/`!wget`, with the pinned
+dependencies preinstalled into the system Python.
+
+Subcommands:
+    list-groups [--filter REGEX] [--names-only]
+        Print the groups as JSON. `--names-only` emits just the group names,
+        suitable for a GitHub Actions matrix.
+    prepare --name NAME --context-dir DIR [--image-prefix PREFIX]
+        Write a docker build context (requirements.txt + work/) for one group
+        and emit image name, build hash, default notebook, and the notebook
+        list to $GITHUB_OUTPUT (or stdout when unset).
+
+The pin extraction reuses `find_install_cell` from run_notebook.py, so the
+image contents stay in lockstep with what CI tests. Entries captured by that
+regex that are not actual pins (it also picks up the literal `form` from the
+`{ display-mode: "form" }` title line) are filtered out: only `pkg==ver` and
+`pkg @ url` requirements are baked into an image.
+
+Assumes `nbformat` is importable.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+
+import nbformat
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from list_notebooks import REPO_ROOT, is_excluded, load_exclusions  # noqa: E402
+from run_notebook import find_install_cell  # noqa: E402
+
+DOCKERFILE = REPO_ROOT / ".github" / "docker" / "Dockerfile"
+DEFAULT_IMAGE_PREFIX = "ghcr.io/dandi/example-notebooks"
+
+
+def slug(s: str) -> str:
+    return re.sub(r"[^a-z0-9]+", "-", s.lower()).strip("-")
+
+
+def real_pins(pins: list[str]) -> tuple[list[str], list[str]]:
+    """Split install-cell entries into actual requirements and strays."""
+    kept = [p for p in pins if "==" in p or " @ " in p]
+    dropped = [p for p in pins if p not in kept]
+    return kept, dropped
+
+
+@dataclass
+class Group:
+    directory: str  # repo-relative, e.g. "001550/PaganLab"
+    pin_hash: str
+    pins: list[str]
+    helpers: list[str] = field(default_factory=list)
+    notebooks: list[str] = field(default_factory=list)  # filenames within directory
+    name: str = ""
+
+    def as_dict(self) -> dict:
+        return {
+            "name": self.name,
+            "directory": self.directory,
+            "pin_hash": self.pin_hash,
+            "notebooks": sorted(self.notebooks),
+        }
+
+
+def collect_groups() -> list[Group]:
+    exclusions = load_exclusions()
+    groups: dict[tuple[str, str], Group] = {}
+    for path in sorted(REPO_ROOT.rglob("*.ipynb")):
+        if ".ipynb_checkpoints" in path.parts:
+            continue
+        rel = str(path.relative_to(REPO_ROOT))
+        if is_excluded(rel, exclusions):
+            continue
+        nb = nbformat.read(path, as_version=4)
+        try:
+            pins, helpers, _ = find_install_cell(nb)
+        except RuntimeError:
+            continue  # no Colab bootstrap -> no image
+        kept, _ = real_pins(pins)
+        pin_hash = hashlib.sha256("\n".join(sorted(kept)).encode()).hexdigest()
+        directory = str(path.parent.relative_to(REPO_ROOT))
+        key = (directory, pin_hash)
+        group = groups.setdefault(
+            key, Group(directory=directory, pin_hash=pin_hash, pins=sorted(kept))
+        )
+        group.notebooks.append(path.name)
+        for h in helpers:
+            if h not in group.helpers:
+                group.helpers.append(h)
+
+    per_dir: dict[str, list[Group]] = {}
+    for g in groups.values():
+        per_dir.setdefault(g.directory, []).append(g)
+    for dir_groups in per_dir.values():
+        for g in dir_groups:
+            g.name = slug(g.directory)
+            if len(dir_groups) > 1:
+                g.name += "-" + slug(Path(sorted(g.notebooks)[0]).stem)
+
+    result = sorted(groups.values(), key=lambda g: g.name)
+    names = [g.name for g in result]
+    assert len(names) == len(set(names)), f"group name collision: {names}"
+    return result
+
+
+def matches(group: Group, pattern: str) -> bool:
+    if not pattern:
+        return True
+    rx = re.compile(pattern)
+    return bool(
+        rx.search(group.directory)
+        or any(rx.search(f"{group.directory}/{n}") for n in group.notebooks)
+    )
+
+
+def cmd_list_groups(args: argparse.Namespace) -> int:
+    groups = [g for g in collect_groups() if matches(g, args.filter)]
+    if args.names_only:
+        print(json.dumps([g.name for g in groups]))
+    else:
+        print(json.dumps([g.as_dict() for g in groups], indent=2))
+    return 0
+
+
+def cmd_prepare(args: argparse.Namespace) -> int:
+    groups = [g for g in collect_groups() if g.name == args.name]
+    if not groups:
+        print(f"error: no group named {args.name!r}", file=sys.stderr)
+        return 1
+    group = groups[0]
+
+    context = Path(args.context_dir)
+    if context.exists():
+        shutil.rmtree(context)
+    work = context / "work"
+
+    group_notebooks = set(group.notebooks)
+    src_dir = REPO_ROOT / group.directory
+
+    def ignore(directory: str, names: list[str]) -> list[str]:
+        ignored = [n for n in names if n == ".ipynb_checkpoints"]
+        for n in names:
+            if not n.endswith(".ipynb"):
+                continue
+            in_group = (
+                Path(directory) == src_dir and n in group_notebooks
+            )
+            if not in_group:
+                ignored.append(n)
+        return ignored
+
+    shutil.copytree(src_dir, work, ignore=ignore)
+
+    (context / "requirements.txt").write_text("\n".join(group.pins) + "\n")
+
+    # Bake helper files the same way run_notebook.py fetches them in CI.
+    for h in group.helpers:
+        cmd_str = h.lstrip("!").strip()
+        m = re.search(r"-o\s+(\S+)", cmd_str)
+        if m:
+            target = work / m.group(1)
+            target.parent.mkdir(parents=True, exist_ok=True)
+        r = subprocess.run(
+            ["bash", "-c", cmd_str], cwd=str(work), capture_output=True, text=True
+        )
+        if r.returncode != 0:
+            print(f"error: helper failed: {cmd_str}\n{r.stderr[-1000:]}", file=sys.stderr)
+            return 1
+
+    h = hashlib.sha256()
+    for pin in group.pins:
+        h.update(pin.encode() + b"\n")
+    for helper in group.helpers:
+        h.update(helper.encode() + b"\n")
+    for name in sorted(group.notebooks):
+        h.update((REPO_ROOT / group.directory / name).read_bytes())
+    h.update(DOCKERFILE.read_bytes())
+    build_hash = h.hexdigest()
+
+    notebooks = sorted(group.notebooks)
+    outputs = {
+        "image": f"{args.image_prefix}/{group.name}",
+        "build_hash": build_hash,
+        "build_hash_short": build_hash[:12],
+        "default_notebook": notebooks[0],
+        "notebooks": json.dumps(notebooks),
+        "context_dir": str(context),
+    }
+    github_output = os.environ.get("GITHUB_OUTPUT")
+    if github_output:
+        with open(github_output, "a") as f:
+            for k, v in outputs.items():
+                f.write(f"{k}={v}\n")
+    print(json.dumps(outputs, indent=2))
+    return 0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_list = sub.add_parser("list-groups")
+    p_list.add_argument("--filter", default="", help="regex on directory or notebook path")
+    p_list.add_argument("--names-only", action="store_true")
+    p_list.set_defaults(func=cmd_list_groups)
+
+    p_prep = sub.add_parser("prepare")
+    p_prep.add_argument("--name", required=True)
+    p_prep.add_argument("--context-dir", required=True)
+    p_prep.add_argument("--image-prefix", default=DEFAULT_IMAGE_PREFIX)
+    p_prep.set_defaults(func=cmd_prepare)
+
+    args = parser.parse_args()
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())