Multi-arch notebook images (native arm64), non-root user, prebaked font cache - #182
Merged
Conversation
A bare -p 8888:8888 binds all interfaces, and on macOS it coexists with a local Jupyter server already listening on 127.0.0.1:8888, so the browser silently reaches the wrong server and shows a confusing 404. Binding 127.0.0.1 explicitly keeps the container off network interfaces and turns a port collision into a loud "address already in use" error. Found during the pilot pull-and-run validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pilot validation on Apple Silicon showed amd64-only images are unreliable under Rosetta: subprocess spawns from the threaded kernel process can deadlock, which surfaced as cells hanging forever on the first matplotlib import (its font-cache build shells out to fc-list) and as the server wedging while spawning a restarted kernel. Emulation is the problem, so the fix is native images for both architectures. The workflow now builds and verifies each group on native runners (ubuntu-latest for amd64, ubuntu-24.04-arm for arm64), pushes per-arch hash-tagged images, and merges them into one multi-arch manifest that carries the user-facing tags. No emulation anywhere in the pipeline. The image also gains a non-root jovyan user (uid 1000; the JupyterLab tool env moves to /opt so it stays readable) and builds the matplotlib font cache at image-build time so first imports don't stall on it. CI verification runs the container with --user root so the harness's transient installs can still write to the system Python; gcc is available during the pin install (purged in the same layer) for pins without a wheel on the target arch, such as old psutil on arm64. Verified natively on arm64: both pilot notebooks green inside the image, and kernel execution over the websocket completes in milliseconds, including subprocess spawns and the matplotlib import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Preview for this PR has been removed (PR closed). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validating the pilot image (#180) on Apple Silicon surfaced a failure mode worse than slowness: Docker's Rosetta emulation can deadlock on subprocess spawns from a threaded x86-64 process. In practice that meant the first
import matplotlib.pyplotfroze the kernel forever (its font-cache build shells out tofc-list), and restarting the kernel wedged the whole Jupyter server while it tried to spawn the replacement kernel process. py-spy stack dumps against the live container pinpointed both. Since a large share of our users are on Apple Silicon Macs, amd64-only images are not shippable.Changes
Multi-arch images, built and verified natively. The workflow now runs each group's build on two native runners (
ubuntu-latestfor amd64,ubuntu-24.04-armfor arm64, which is free for public repos), verifies every notebook inside each per-arch candidate with the unmodified harness, pusheshash-<h>-amd64/hash-<h>-arm64images, and a merge job assembles them into one multi-arch manifest carrying the user-facing tags (latest,sha-, date,hash-). The per-arch hash tags double as skip markers, so unchanged groups still cost nothing to rebuild. There is no emulation anywhere in the pipeline, anddocker runon a Mac now picks the native image automatically.Non-root user. The server now runs as
jovyan(uid 1000) rather than root; the JupyterLab tool env moves from/root/.localto/optso it stays readable. CI verification passes--user rootso the harness's transient installs can write to the system Python; the published default stays non-root.Font cache prebaked. Matplotlib's font cache is built at image-build time (as
jovyan), so the first import in a fresh container no longer stalls on it. This also removes the exact runtime subprocess spawn that deadlocked under Rosetta, for anyone who does still run the amd64 image emulated.Compiler during pin install.
gcc/libc6-devare installed for the pin layer and purged in the same layer, covering pins with no wheel on the target architecture (the pilot'spsutil==5.9.5has no arm64 wheel).This includes the docs commit from #181 (loopback port binding), which can be closed if this merges first. One correction to #180's description: the pilot image is about 1.7 GB uncompressed, not 380 MB; that figure was a misreading during the first local build.
Verification
Built the arm64 image natively on an M-series Mac with the new Dockerfile: both pilot notebooks verify green inside it (7 to 8 seconds each), and kernel execution over the websocket as
jovyancompletes in milliseconds where the emulated image hung indefinitely:a = 1in 0.05s, a subprocess spawn in 0.04s,import matplotlib.pyplotin 0.62s with the prebaked cache. The amd64 path reuses the already-exercised build and verify steps from #180's pilot runs.After merge: dispatch the workflow with
forcefor the pilot filter so both arches rebuild and the multi-arch manifest replaces the amd64-only tags, then continue the fleet rollout.🤖 Generated with Claude Code