fix(supply-chain): attest the real published tags, and make the GPU image mirrorable

Review was right on both blocking points, and both needed code rather than
wording.

`imagetools create` always writes an index, even from a single manifest — so
`:<version>-amd64` is a single-entry index whose digest differs from the
`:<sha>-amd64` manifest it wraps. Verified against a local registry: pushing a
manifest at bd44eb13 and retagging it produced e857a74f. Attesting the manifest
therefore left the tag people actually pin unverifiable, and a static matrix
could not fix it: resolving the live tags for one image returns five distinct
digests, and which of them exist depends on whether the run is a release and
whether the latest guard passed.

A new attest-subjects job resolves the published tags to digests, de-duplicates,
and feeds attest-images as a dynamic matrix. Tags this run did not publish are
skipped rather than failing it.

The NVIDIA device plugin was hardcoded in the template, so the previous
guidance — mirror it to the same path, or patch the DaemonSet — was not durable:
copying to another registry does not redirect the pull, and a patch is reverted
by the next helm upgrade. It now takes its image from
ollama.gpu.devicePlugin.image and goes through the shared helper, so it honors
global.imageRegistry with useRegistryForAllImages like every other third-party
image. Default renders byte-identical; chart minor bumped for the new key.

Docs corrections from the same round, several of them regressions from my own
previous commit:

- The ADFS example set SSO_ISSUER to the IdP identifier, and SSO_SAML_AUDIENCE
  defaults to SSO_ISSUER — so it silently broke audience validation. The example
  now sets both.
- Compose does not pass MIGRATION_DATABASE_URL through from .env: its migrations
  service declares an explicit environment list.
- CLI authentication requires Redis; it does not fall back, and redis.mdx
  already said so.
- The rollback snippet assigned SIM_VERSION in a subshell that Compose never saw.
- Custom blocks hide their internals only when tracing is off.
- A new workspace is still governed by the organization's default group.
- doctor is not a Kubernetes tool, so it cannot be the first step for Helm users.
- ENTERPRISE_ENABLED needs its NEXT_PUBLIC twin.
- The 100 MB response cap does not cover MCP's standalone SSE stream.
- Generating CRON_SECRET in a shell configures nothing by itself.

Renaming two headings broke published anchors that an in-repo library article
links to; both now point at the new slugs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt

Author: waleedlatif1

.github/workflows/ci.yml

@@ -620,26 +620,93 @@ jobs:
   # Attaching attestations here instead leaves the index itself untouched — they
   # are stored as separate referrer manifests that point at it.
   #
-  # Every pullable tag is covered. `create-ghcr-manifests` publishes both
-  # multi-arch tags (`:sha`, `:version`, `:latest`) and single-arch ones
-  # (`:version-amd64`, `:latest-arm64`, …), and those resolve to different
-  # digests — an index digest and the two per-arch manifest digests. The matrix
-  # therefore attests all three subjects per image. `imagetools create` is
-  # deterministic, so the version and latest indexes share the sha index's
-  # digest and need no separate attestation.
+  # Resolve the set of digests that actually got published, so the attestation
+  # job below covers every tag a customer can pull.
   #
-  # Per-arch subjects also give a truthful SBOM: the amd64 and arm64 images
-  # contain different packages, and a single SBOM attached to the index cannot
-  # describe both.
+  # A static list is not enough. `imagetools create` always writes an INDEX, so
+  # `:<version>-amd64` is a single-entry index whose digest differs from the
+  # `:<sha>-amd64` manifest it wraps — attesting the manifest leaves the tag
+  # people actually pin unverifiable. Which tags exist also varies per run:
+  # version tags only on a release, and the latest tags only when the monotonic
+  # guard in create-ghcr-manifests passed. Resolving tag -> digest here and
+  # de-duplicating is what keeps the two in step without hardcoding that logic
+  # twice.
+  attest-subjects:
+    name: Resolve Attestation Subjects
+    runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
+    timeout-minutes: 10
+    needs: [create-ghcr-manifests, detect-version]
+    if: >-
+      !cancelled() &&
+      needs.create-ghcr-manifests.result == 'success' &&
+      needs.detect-version.result == 'success' &&
+      github.event_name == 'push' && github.ref == 'refs/heads/main'
+    permissions:
+      contents: read
+      packages: read
+    outputs:
+      subjects: ${{ steps.resolve.outputs.subjects }}
+    steps:
+      - name: Login to GHCR
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Resolve published tags to digests
+        id: resolve
+        env:
+          IS_RELEASE: ${{ needs.detect-version.outputs.is_release }}
+          VERSION: ${{ needs.detect-version.outputs.version }}
+          SHA: ${{ github.sha }}
+        run: |
+          set -euo pipefail
+
+          IMAGES="simstudio migrations realtime pii cron"
+          TAGS="${SHA} ${SHA}-amd64 ${SHA}-arm64 latest latest-amd64 latest-arm64"
+          if [ "${IS_RELEASE}" = "true" ]; then
+            TAGS="${TAGS} ${VERSION} ${VERSION}-amd64 ${VERSION}-arm64"
+          fi
+
+          : > /tmp/subjects.jsonl
+          for name in $IMAGES; do
+            image="ghcr.io/simstudioai/${name}"
+            seen=""
+            for tag in $TAGS; do
+              # A tag that this run did not publish (no release, or the latest
+              # guard held it back) is expected to be missing — skip it rather
+              # than fail the run.
+              digest="$(docker buildx imagetools inspect "${image}:${tag}" \
+                --format '{{json .Manifest}}' 2>/dev/null | jq -r '.digest // empty')"
+              [ -n "$digest" ] || continue
+              case " $seen " in *" $digest "*) continue ;; esac
+              seen="$seen $digest"
+              jq -nc --arg image "$image" --arg digest "$digest" \
+                '{image: $image, digest: $digest}' >> /tmp/subjects.jsonl
+            done
+          done
+
+          if [ ! -s /tmp/subjects.jsonl ]; then
+            echo "::error::Resolved no image digests to attest"
+            exit 1
+          fi
+
+          echo "Resolved $(wc -l < /tmp/subjects.jsonl) distinct subjects:"
+          cat /tmp/subjects.jsonl
+          echo "subjects=$(jq -sc . /tmp/subjects.jsonl)" >> "$GITHUB_OUTPUT"
+
+  # One leg per distinct published digest. Attesting each subject separately is
+  # also what makes the SBOMs truthful: the amd64 and arm64 images contain
+  # different packages, and one SBOM attached to the index cannot describe both.
   attest-images:
     name: Attest Images
     runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
     timeout-minutes: 15
-    needs: [create-ghcr-manifests]
+    needs: [attest-subjects]
     if: >-
       !cancelled() &&
-      needs.create-ghcr-manifests.result == 'success' &&
-      github.event_name == 'push' && github.ref == 'refs/heads/main'
+      needs.attest-subjects.result == 'success'
     permissions:
       contents: read
       packages: write
@@ -649,15 +716,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        image:
-          - ghcr.io/simstudioai/simstudio
-          - ghcr.io/simstudioai/migrations
-          - ghcr.io/simstudioai/realtime
-          - ghcr.io/simstudioai/pii
-          - ghcr.io/simstudioai/cron
-        # The published tag suffixes, and so the distinct digests a customer can
-        # pull. Empty is the multi-arch index.
-        suffix: ['', '-amd64', '-arm64']
+        include: ${{ fromJSON(needs.attest-subjects.outputs.subjects) }}
 
     steps:
       - name: Login to GHCR
@@ -667,30 +726,10 @@ jobs:
           username: ${{ github.repository_owner }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      # Resolved once and reused by every step below: signing a tag would sign
-      # whatever that tag points at when the step runs, which is not necessarily
-      # what this run published.
-      #
-      # `{{json .Manifest}}` piped through jq, not a bare `{{.Manifest.Digest}}`:
-      # buildx renders a format string consisting only of a `.Manifest` field as
-      # the human-readable inspect block rather than the field value, so the bare
-      # form does not produce a parseable digest.
-      - name: Resolve digest
-        id: digest
-        run: |
-          REF="${{ matrix.image }}:${{ github.sha }}${{ matrix.suffix }}"
-          DIGEST="$(docker buildx imagetools inspect "$REF" \
-            --format '{{json .Manifest}}' | jq -r '.digest')"
-          if [ -z "$DIGEST" ] || [ "$DIGEST" = "null" ]; then
-            echo "::error::Could not resolve a digest for ${REF}"
-            exit 1
-          fi
-          echo "value=${DIGEST}" >> "$GITHUB_OUTPUT"
-
       - name: Generate SBOM
         uses: anchore/sbom-action@3ad7283483fc7af8ff2b4ea19663c2d5ca935e26 # v0.24.2
         with:
-          image: ${{ matrix.image }}@${{ steps.digest.outputs.value }}
+          image: ${{ matrix.image }}@${{ matrix.digest }}
           format: spdx-json
           output-file: sbom.spdx.json
           # The action's own release upload is for workflows triggered by a
@@ -702,7 +741,7 @@ jobs:
         uses: actions/attest-sbom@c604332985a26aa8cf1bdc465b92731239ec6b9e # v4.1.0
         with:
           subject-name: ${{ matrix.image }}
-          subject-digest: ${{ steps.digest.outputs.value }}
+          subject-digest: ${{ matrix.digest }}
           sbom-path: sbom.spdx.json
           # Stored alongside the image so a mirrored registry carries the
           # attestation with it, rather than only being retrievable from GitHub.
@@ -712,7 +751,7 @@ jobs:
         uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
         with:
           subject-name: ${{ matrix.image }}
-          subject-digest: ${{ steps.digest.outputs.value }}
+          subject-digest: ${{ matrix.digest }}
           push-to-registry: true
 
       - name: Install Cosign
@@ -722,7 +761,7 @@ jobs:
       # signature that admission controllers (Kyverno, the Sigstore policy
       # controller) verify before admitting a pod.
       - name: Sign image
-        run: cosign sign --yes "${{ matrix.image }}@${{ steps.digest.outputs.value }}"
+        run: cosign sign --yes "${{ matrix.image }}@${{ matrix.digest }}"
 
   # Check if docs changed
   # Smallest runner on purpose: a depth-2 checkout plus a path filter, no

apps/docs/content/docs/platform/enterprise/access-control.mdx

@@ -160,7 +160,7 @@ The **Chat Deployment** row also carries an **auth-mode allowlist** — *Auth mo
 | Feature | What clearing it withholds |
 |---------|---------------------------|
 | Invitations | Prevents inviting anyone to a workspace or to the organization. |
-| Workspace Creation | Prevents creating new workspaces. A new workspace falls outside every existing group's scope. |
+| Workspace Creation | Prevents creating new workspaces. A new one is not covered by any workspace-scoped group until you add it, though the organization's default group still governs it. |
 | Member Directory | Withholds the member directory. Members cannot see the names or email addresses of other members. |
 
 **Credentials & Access**

apps/docs/content/docs/platform/enterprise/custom-blocks.mdx

@@ -14,7 +14,7 @@ A custom block always runs the **latest deployed version** of its source workflo
 
 ## Common uses
 
-The block's author keeps the credentials and the workflow logic; consumers see only the inputs and outputs. Common patterns:
+The block's author keeps the credentials and the workflow logic; by default consumers see only the inputs and outputs, unless the author enables **Trace runs in consumer logs**. Common patterns:
 
 - **Internal API gateway.** Wrap an authenticated internal or partner endpoint — "Create Ticket", "Charge Account", "Provision User" — behind a block that takes only the business inputs. Teammates call it without the base URL, API key, or auth headers, and when the endpoint changes you update one workflow instead of every consumer's.
 - **Blessed knowledge lookup.** Package a vetted retrieval pipeline — chunking, filters, reranking — as "Search Company Docs" with a single query input, so teams reuse the approved retrieval instead of each rebuilding it.

apps/docs/content/docs/platform/enterprise/data-drains.mdx

@@ -250,9 +250,11 @@ GET /api/cron/run-data-drains
 It authenticates with a bearer token equal to `CRON_SECRET` and returns `401` when that variable is unset, so a self-hosted deployment must set it:
 
 ```bash
-CRON_SECRET=$(openssl rand -hex 32)
+openssl rand -hex 32
 ```
 
+Set the same value as `CRON_SECRET` on both the app and whatever invokes the endpoint. Generating it in your shell does not configure either one.
+
 The Helm chart schedules this endpoint hourly for you (`cronjobs.jobs.runDataDrains`). Outside Helm, schedule it yourself:
 
 ```bash

apps/docs/content/docs/platform/enterprise/data-retention.mdx

@@ -213,9 +213,10 @@ PII redaction runs against a standalone [Presidio](https://microsoft.github.io/p
 
 ```bash
 # The Presidio service exposing /analyze and /anonymize
-# Helm: http://<release>-pii.<namespace>.svc.cluster.local:5001
-# Docker Compose: http://<pii service name>:5001
-PII_URL=http://sim-pii.simstudio.svc.cluster.local:5001
+# Helm — substitute your release name and namespace
+PII_URL=http://<release>-pii.<namespace>.svc.cluster.local:5001
+# Docker Compose — the PII service name on your network
+# PII_URL=http://pii:5001
 ```
 
 All PII stages are configurable under **Settings → Organization → Data retention**.

apps/docs/content/docs/platform/enterprise/sso.mdx

@@ -387,6 +387,7 @@ NEXT_PUBLIC_APP_URL=https://your-instance.com \
 SSO_PROVIDER_TYPE=saml \
 SSO_PROVIDER_ID=adfs \
 SSO_ISSUER=https://adfs.company.com/adfs/services/trust \
+SSO_SAML_AUDIENCE=https://your-instance.com \
 SSO_DOMAIN=company.com \
 SSO_USER_EMAIL=admin@company.com \
 SSO_SAML_ENTRY_POINT=https://adfs.company.com/adfs/ls \

apps/docs/content/docs/platform/enterprise/verified-domains.mdx

@@ -71,6 +71,6 @@ SSO_ENABLED=true
 NEXT_PUBLIC_SSO_ENABLED=true
 ```
 
-`ENTERPRISE_ENABLED` turns both on together. See the [self-hosted enterprise guide](/platform/enterprise/self-hosted).
+`ENTERPRISE_ENABLED` turns both on together, but it needs its own browser twin — set `NEXT_PUBLIC_ENTERPRISE_ENABLED` alongside it, or the server enables SSO while the browser still hides it. See the [self-hosted enterprise guide](/platform/enterprise/self-hosted).
 
 Once enabled, verify domains from **Settings → Organization → Single sign-on**, in the **Verified domains** section above the identity provider configuration. The older `/workspace/<workspaceId>/settings/domains` path still resolves to the same page.

apps/docs/content/docs/platform/self-hosting/architecture.mdx

@@ -52,7 +52,7 @@ This holds essentially all durable state: workflows, runs, logs, users, organiza
 
 ### redis
 
-Backs pub/sub, the Socket.IO adapter, the idempotency store, execution progress markers, distributed execution limits, and the CLI-auth approval store. The idempotency store, progress markers, and approval store fall back to Postgres or in-process state. Pub/sub falls back to a **process-local** emitter, which is fine on one replica and drops every cross-pod event on more than one. See [Redis](/platform/self-hosting/redis).
+Backs pub/sub, the Socket.IO adapter, the idempotency store, execution progress markers, distributed execution limits, and the CLI-auth approval store. The idempotency store and execution progress markers fall back to Postgres or in-process state. The CLI-auth approval store does not — CLI authentication requires Redis at any replica count. Pub/sub falls back to a **process-local** emitter, which is fine on one replica and drops every cross-pod event on more than one. See [Redis](/platform/self-hosting/redis).
 
 ### cron
 

apps/docs/content/docs/platform/self-hosting/security.mdx

@@ -220,7 +220,7 @@ The allowlist replaces four separate escape hatches, so a few deployments that w
 
 Every guarded outbound response is bounded. A caller that does not set its own limit gets the default of **100 MB**; exceeding it rejects the request with a payload-size error and destroys the socket rather than buffering the rest.
 
-This is an easily misread cause of "a large download from an integration fails" — the failure looks like a broken connection to the third-party service rather than a limit Sim imposed. It applies to the guarded provenances above, not to the presigned object-storage upload path.
+This is an easily misread cause of "a large download from an integration fails" — the failure looks like a broken connection to the third-party service rather than a limit Sim imposed. It applies to the guarded provenances above, not to the presigned object-storage upload path, and not to MCP's standalone SSE stream, which is deliberately unbounded so a long-lived stream is not cut off.
 
 ## Client IP and forwarded headers
 

apps/docs/content/docs/platform/self-hosting/troubleshooting.mdx

@@ -5,7 +5,7 @@ description: Common issues and solutions
 
 ## Start here: `sim-setup doctor`
 
-Before working through anything below, run the built-in checker. It catches most self-hosting failures without you having to guess which subsystem is broken.
+On a Docker Compose or source install, run the built-in checker first. It catches most failures without you having to guess which subsystem is broken. It reads env files, so it is not a Kubernetes tool — on Helm, skip to the checks below.
 
 ```bash
 npx sim-setup doctor