ci: add k8s stack PR preview via labeled deploy + Cloudflare Tunnel - #205
Draft
aktech wants to merge 52 commits into
Draft
ci: add k8s stack PR preview via labeled deploy + Cloudflare Tunnel#205aktech wants to merge 52 commits into
aktech wants to merge 52 commits into
Conversation
Labeling a PR with deploy-preview spins up the full NIC platform stack plus this chart in an ephemeral k3d cluster, then exposes JupyterHub through a per-PR Cloudflare Tunnel gated by Cloudflare Access (GitHub org SSO), posting the link as a sticky PR comment. Includes a standalone tunnel-only smoketest workflow for validating the Cloudflare plumbing without the full stack deploy.
aktech
added
deploy-preview
Contributor
|
The K8s stack preview for this PR has expired.
Push a new commit or re-add the |
pull_request/synchronize on this one file, scoped narrowly, so the tunnel-only smoketest can run without waiting for main to see the workflow_dispatch trigger. Remove before merging.
--no-autoupdate is a tunnel-level flag, not a run-subcommand flag; placed after `run` it errored with "flag provided but not defined" and the tunnel process exited immediately without ever connecting.
Two-level hostnames (pr-<n>.dspack.iakte.ch) aren't covered by Cloudflare's default Universal SSL, which only auto-issues a cert for the zone apex plus one wildcard level. Switching to a dedicated single-level domain (pr-<n>.github.fyi) avoids needing the paid Advanced Certificate Manager add-on.
workflow_dispatch is now registered and reachable; the iteration workaround is no longer needed.
aktech
added
the
deploy-preview
charts/ is gitignored, so the vendored jupyterhub-4.3.2.tgz seen in local dev never reaches a fresh CI checkout. helm dependency build resolves it against the already-committed Chart.lock digest.
workflow_dispatch has no PR context to auto-detect, so the comment step uses number_force (sticky-pull-request-comment's any-event PR number override, distinct from number which is push-event only).
Contributor
|
Tunnel smoketest (static page, not the real stack): Deployed: 2026-08-31 12:26 UTC · Expires: 2026-08-31 12:41 UTC Sign-in via Cloudflare Access (GitHub SSO), then should show "Tunnel smoketest OK". |
helm dependency build resolves against locally-registered repos, not just the Chart.lock digest; a fresh runner has none configured. Added helm repo add before the build step. Verified against a clean clone.
…debugging nic-operator ignores any namespace without nebari.dev/managed=true, so the NebariApp CRD never reconciled. Also adds a static pod/job/event dump and an interactive tmate SSH session (actor-restricted, 20min cap) on deploy failure, since the cluster is deleted right after and static logs alone weren't enough to diagnose the last two failures.
00-gateway-auth.py reads /etc/oauth/issuer-url unconditionally at import time. That file only exists once the operator's async Keycloak client provisioning finishes, which Helm doesn't wait for, so the hub pod crash-loops immediately when nebariapp.auth.enabled=true. Confirmed via kubectl logs during a failed run. Dropping nebariapp entirely for this preview; operator/OIDC reconcile isn't exercised here.
Without it, 02-jhub-apps.py falls back to bind_url=http://0.0.0.0:8000, which JupyterHub then bakes into browser-facing OAuth redirect URLs for the jhub-apps service (client_id=service-japps), breaking the login flow after dummy-auth. Confirmed via a real login attempt on the deployed preview.
The preview URL is stable per-PR, so redeploys posted byte-identical comment text and looked like they never updated. Adding deployed-at and expires-at timestamps makes every redeploy visibly change the comment and tells reviewers exactly when the link goes dead.
…ri-data-science-pack into add-k8s-preview-workflow
aktech
added
deploy-preview
A workflow retry keeps the same github.run_id (only run_attempt changes), so a re-run after an attempt already created the tunnel but didn't reach its cleanup step hit a 409 name conflict. Now falls back to looking up and reusing the existing tunnel by name instead of failing outright. Confirmed via a real run_attempt=2 failure.
PREVIEW_DOMAIN now reads from vars.PREVIEW_DOMAIN (repo Settings -> Actions -> Variables), set to github.fyi, in both the real preview workflow and the smoketest. Repointing to a different domain no longer needs a code change.
Same account already holds both the tunnel (Cloudflare Tunnel:Edit) and the openteams.app zone (Zone:Read, DNS:Edit) on the existing CLOUDFLARE_TUNNEL_API_TOKEN, so no new token needed. Hostnames flatten to pr-<n>-data-science-pack.openteams.app (single label) to stay within Cloudflare's free Universal SSL one-wildcard-level coverage.
aktech
added
deploy-preview
Drops the removed profile/k3d-version/k8s-version inputs, installs kind via helm/kind-action, swaps k3d image import + manual cluster delete for kind load docker-image (v3 auto-tears-down the cluster on its own).
kind load docker-image failed with "unknown containerd config version: 4" against the sandbox's kind node image. Support for that format landed in kind v0.32.0; v0.27.0 (matching test.yaml, whose own kind-created cluster doesn't hit this) predates it.
A live browser login through the preview link 502s on /services/japps/oauth_callback. jhub-apps runs as a managed-service subprocess inside the hub pod, so a crash there doesn't fail helm --wait or the rollout status checks -- add an unconditional hub log dump so the cause shows up in the run's own output.
Both Keycloak and nebari-operator are ArgoCD Applications with selfHeal: true, reconciled against NIC's auto-created local GitOps repo. A direct kubectl set env DID apply (confirmed live: both workloads genuinely rolled to new pods), but ArgoCD's reconcile loop silently reverted it within a couple minutes once it noticed the drift from the GitOps repo -- hub's redirect still landed on the unreachable keycloak.nebari.local well after the "successful" patch. Edit the two source files in the GitOps repo directly (values/keycloak/base.yaml, manifests/nebari-operator/deployment-patch.yaml), commit, and force an ArgoCD hard refresh so selfHeal applies our change instead of fighting it -- the same pattern already used for this project's other ArgoCD-managed clusters. Poll for the live env var to actually change before trusting rollout status, since a stale refresh can make rollout status return instantly without ArgoCD having applied anything yet.
Cloudflare Access is the real security boundary for this preview -- only allow-listed accounts reach the tunnel at all -- so a simple, known Keycloak password is fine here rather than making reviewers hunt for real credentials on a throwaway cluster. Creates the user via Keycloak's admin API, piping the admin secret straight into the token request without ever echoing or logging it. PR comment now states the credentials directly.
…cess Confirmed live: after a real login, hub's OAuth callback hit json.decoder.JSONDecodeError: Expecting value (an empty/non-JSON body) on the token exchange. Keycloak's public hostname sits behind Cloudflare Access same as the JupyterHub hostname -- fine for the browser-facing authorize/login legs since the reviewer already has an Access session, but hub's own server-to-server token/userinfo calls have no such session and get blocked by Access instead of a JSON response. Set keycloak.backchannelURL to Keycloak's in-cluster Service so only token_url/userdata_url move there -- authorize_url and end_session_url stay on the public issuer, which is what the browser actually needs to reach. This is exactly the split-horizon path config/jupyterhub/00-gateway-auth.py already supported; just needed the value set.
jhub-apps runs as a managed-service subprocess inside the same pod as hub, but z2jh injects JUPYTERHUB_API_URL pointing at the `hub` Service (a ClusterIP self-reference). Routing same-pod traffic through a Service depends on the CNI supporting hairpin NAT for a pod reaching its own Service -- confirmed via a live preview to time out (httpcore.ConnectTimeout) on a kind/kindnet cluster, breaking /services/japps/oauth_callback with a 500 right after a successful Keycloak login. Rewrite the host to localhost (same port/path) before forwarding it into the japps service's subprocess environment. This is a general fix, not preview-specific: same-pod loopback is always reliable regardless of hairpin NAT support on whatever CNI a real deployment happens to run.
Confirmed live: setting svc["environment"]["JUPYTERHUB_API_URL"] (same pattern as the existing OIDC secret forwarding) had no effect -- the subprocess still saw the `hub` Service URL and still timed out. JupyterHub's own Spawner.get_env() computes env['JUPYTERHUB_API_URL'] = hub_api_url from self.hub.api_url *after* merging self.environment, unconditionally overwriting whatever a service config sets. Only something applied after JupyterHub finishes building the subprocess's env can actually win, so wrap the service's own command with a shell-level `env VAR=value` assignment instead -- that's scoped to jhub-apps' own process and can't be overridden by the parent.
Keycloak's default realm profile requires email/firstName/lastName, so creating the user without them triggered a first-login "Update Account Information" prompt every time. Set them at creation instead so the reviewer account logs straight through.
The previous fix read os.environ.get("JUPYTERHUB_API_URL") in the hub
process's own config-load-time environment to precompute a localhost
URL to inject. That variable is empty there -- JupyterHub only computes
and injects it into a service's own environment at spawn time -- so the
rewrite was silently a no-op and jhub-apps kept timing out against the
`hub` Service.
Wrap the service's command with a shell snippet that rewrites
$JUPYTERHUB_API_URL to localhost (via sed, keeping port/path) at the
moment the subprocess execs, using whatever JupyterHub has actually put
in its environment by then.
The tunnel step ran unbounded and relied on the job's timeout-minutes (90) to kill it, which makes GitHub mark the whole run "cancelled" with a "job has exceeded the maximum execution time" failure annotation -- looks like a real failure even though the preview is meant to expire. Bound the step itself to 20 minutes via `timeout`, treating its own deadline (exit 124) as success, so the job finishes normally well under the outer limit. Updates the reviewer-facing expiry timestamp and comments to match.
The sticky PR comment edits in place (it was created on the PR's first deploy), so it doesn't move in the thread and looked like it never updated even though its content was current. Add a GitHub Deployment/Environment (pr-<n>-preview) alongside it: GitHub renders that as its own status box pinned near the top of the PR with the live URL and timestamp, independent of the comment thread. Marked inactive once the tunnel closes or the deploy-preview label is removed, so it doesn't keep showing a dead link as green.
GitHub Actions runs run: steps as bash -e. A plain `timeout ...; ec=$?` doesn't work there -- -e aborts the step the instant timeout returns 124 (its own deadline hit), before ec=$? is ever reached, so the step showed as a real failure even on the expected timeout path (confirmed live on run 33499572113). Capture the exit code via `|| ec=$?` instead, which is a protected context -e doesn't abort on.
Drops the login walkthrough (obvious to anyone opening the link) and lays out project/status/links/timestamp as a compact table instead, so the comment reads at a glance like a Vercel-for-GitHub bot comment.
Was showing the PR's branch ref, not a project name -- doesn't match the Vercel-style table it's mimicking, where Project identifies the thing being deployed. Use the chart's actual name instead.
Reading "Expires: 2026-09-01 12:37 UTC" means doing timezone math by hand to know if the preview is about to die. Use GitHub's own <relative-time> web component (confirmed via gh api /markdown to survive comment sanitization) so it reads "in 17 minutes" and keeps ticking client-side, no re-editing the comment required. Falls back to the plain UTC string as its text content before the element hydrates.
The comment was only ever written once, at deploy time. <relative-time> kept the clock ticking but the surrounding wording and green status dot never changed, so an expired preview still read "Ready" next to "Expires 39 minutes ago". Edit the same comment again during teardown with the correct wording and an inactive status marker. Also strips stray em dashes from workflow comments and the PR-facing text.
Adding extend-preview any time before the preview expires pushes the tunnel step's deadline back 20 minutes from that moment (a reset, not a cumulative add), without tearing down and rebuilding the cluster. The already-running tunnel step polls for the label every 15s and consumes it by deleting it, so it can be reused any number of times before the job's own 90min timeout-minutes catches up. Scopes the job's labeled/unlabeled trigger to the deploy-preview label specifically, since a labeled event fires for any label add -- without this, adding extend-preview would start a second run of the job, which concurrency: cancel-in-progress then uses to cancel the very run it was meant to extend.
github-actions
Bot
removed
the
extend-preview
The job-level if: condition added earlier only skips deploy-preview's job for a labeled event on any label but deploy-preview -- it doesn't stop concurrency: cancel-in-progress from cancelling the current run, since that operates on the workflow run itself regardless of whether the job inside it ends up running. Confirmed live: adding extend-preview queued a new run that was about to cancel the one it was supposed to extend. cancel-in-progress now evaluates to false specifically for a labeled event on extend-preview, so that new (skipped) run just queues behind the current one instead of cancelling it.
aktech
added
deploy-preview
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
deploy-previewlabel-triggered workflow that spins up the full NIC platform stack plus this chart in an ephemeral k3d cluster, then exposes JupyterHub through a per-PR Cloudflare Tunnel gated by Cloudflare Access (GitHub org SSO), posting the link as a sticky PR comment.Also adds a standalone
workflow_dispatchsmoketest workflow that validates just the Cloudflare Tunnel/DNS/Access plumbing against a trivial static page, for fast iteration without paying for the full stack deploy each time.Draft PR to test the preview workflow on itself — add the
deploy-previewlabel to try it.