fix(webapp,helm): require control-plane auth secrets and remove weak bundled-datastore defaults (#57)

* fix(webapp): require control-plane auth secrets

* fix(helm): remove weak default credentials for bundled datastores

Empty the shipped postgres/clickhouse/minio/registry credential defaults so an
unconfigured install fails closed instead of booting with publicly-known
passwords. Add fail-closed validation guards for the bundled (deploy=true)
datastores and feed CI throwaway render-time values so chart linting stays green.

* fix(webapp): reject known-insecure control-plane secret defaults

Add a blocklist refine to PROVIDER_SECRET, COORDINATOR_SECRET and
MANAGED_WORKER_SECRET so the webapp rejects the publicly-known
placeholder strings at boot, matching the value the coordinator already
refuses at startup. Add the three now-required vars to the root
.env.example and align the supervisor MANAGED_WORKER_SECRET example with
the Docker example so local setups authenticate consistently.

* fix(helm): fail closed on missing control-plane secrets and fix datastore guard guidance

Blank the bundled provider/coordinator/managed-worker secret defaults and
add a render-time guard so an unconfigured install fails closed instead of
shipping repo-published tokens, matching the datastore fail-closed pattern.
Require both postgres.auth.postgresPassword and postgres.auth.password, and
point the Postgres/ClickHouse guard messages at the key paths the webapp
actually consumes so operators are not sent to a crash loop. Add placeholder
credentials to values-production-example.yaml and the CI lint values so the
documented install paths still render, and update the README to reflect that
a values file with credentials is now required.

* test(webapp): provide required control-plane secrets in e2e harness

The e2e webapp harness (startTestServer) spawns the production webapp
bundle, which now fails closed when PROVIDER_SECRET / COORDINATOR_SECRET
/ MANAGED_WORKER_SECRET are unset. Supply strong test values (not the
blocklisted known-insecure defaults) so the webapp boots and the
E2E Tests: Webapp check passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(webapp,hosting): fail closed on empty app secrets and stop shipping working ones

Extend the fail-closed convention to the two remaining app secrets and the
Docker Compose self-host path:

- env.server: SESSION_SECRET and MAGIC_LINK_SECRET now require .min(1), so an
  empty value fails the schema parse at boot rather than running with an empty
  secret (SESSION_SECRET is the JWT signing key) or falling through to the
  emailAuth runtime guard.
- hosting/docker/.env.example and apps/supervisor/.env.example: ship the
  crypto/auth secrets empty with a short 'generate with openssl rand -hex 16'
  comment instead of the previous working public values. A bare
  'cp .env.example .env && docker compose up' now fails closed on the first
  missing secret; the documented setup already generates all six.
- Tests: cover empty and unset rejection for all five required secrets.

Addresses SEC-387 / GHSA-pqxw-g93w-hj9x (the Docker path the earlier commits
in this PR did not cover).

* fix(webapp,helm): reject published application secret defaults

* fix(helm): reject unsupported bundled datastore secret references

* format

* .env

* feat(helm): auto-generate app and control-plane secrets instead of failing closed

Leave secrets.* empty to have the chart generate a strong value on first install, retained across upgrades via lookup so ENCRYPTION_KEY/SESSION_SECRET are never rotated. Explicit values and existingSecret still win. Removes the now-redundant fail-closed guards for these secrets (the webapp still rejects known-insecure values at startup).

* feat(hosting): add docker generate-secrets.sh to fill self-hosting secrets

Fills empty required secrets in .env with openssl-random values. Safe to re-run: never overwrites an already-set value, so restarts and re-runs don't rotate ENCRYPTION_KEY/SESSION_SECRET. Docs updated to run it during setup.

* ci(helm): lint and render release-helm with ci lint-values

The bundled-datastore passwords are now fail-closed, so a values.yaml-only lint/template render fails. Supply ci/lint-values.yaml like the prerelease workflow already does, keeping the chart release pipeline green.

* feat(webapp): allow opting out of the insecure-default secret blocklist

ALLOW_INSECURE_DEFAULT_SECRETS lets a deployment boot while still using a known-published default it cannot safely rotate yet (e.g. ENCRYPTION_KEY protects existing data, SESSION_SECRET rotation logs everyone out). It only bypasses the known-insecure blocklist; the min-length and 32-byte ENCRYPTION_KEY checks still apply. A loud warning is logged at boot naming any secret still on a published default.

* feat(hosting): auto-generate bundled-datastore passwords for docker self-hosting

Remove the shipped weak defaults for postgres/clickhouse/minio/registry. generate-secrets.sh now fills each datastore password (openssl rand), and for the registry writes a matching bcrypt htpasswd via docker. Connection URLs are derived from the single password var by compose interpolation, so server and client always match; datastore services fail closed with a helpful message if a password is unset. Never clobbers an existing value - use --force to rotate.

* docs(self-hosting): document secret auto-generation, rotation gotchas, and the insecure-default opt-out

Note that generate-secrets.sh also fills the bundled datastore passwords; add ALLOW_INSECURE_DEFAULT_SECRETS to the webapp env list; add a Helm secret generation/rotation section covering upgrade retention, the GitOps regeneration caveat, and the lack of a clean ENCRYPTION_KEY migration.

* feat(helm): auto-generate bundled-datastore passwords

Generate the postgres/clickhouse/minio passwords once into a chart-managed datastore Secret (retained across upgrades via lookup) and point each bundled subchart at it via auth.existingSecret; the webapp reads them back through secretKeyRef and $(VAR) URL interpolation, so server and client always match. The registry password is generated and retained inside secrets.yaml (consumed at render time by htpasswd + dockerconfigjson). Drops the deploy=true datastore password guards - a bare helm install now renders and deploys with strong, unique credentials, and explicit values or an existingSecret still win.

* fix(helm): start the s2 container via args, not command

The s2 image ENTRYPOINT is ["./s2"] with the subcommand/flags as CMD. Setting Kubernetes command overrode the entrypoint and tried to exec "lite" directly, so the pod failed with StartError and realtime streams v2 (the default) never came up. Passing them as args preserves the entrypoint (./s2 lite ...). Verified live in kind: the s2 pod now reaches Running.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>

Author: 3 people

.env.example

@@ -2,6 +2,7 @@
 SESSION_SECRET=abcdef1234
 MAGIC_LINK_SECRET=abcdef1234
 ENCRYPTION_KEY=ae13021afef0819c3a307ad487071c06 # Must be a random 16 byte hex string. You can generate an encryption key by running `openssl rand -hex 16` in your terminal
+MANAGED_WORKER_SECRET=abcdef1234 # Must match the supervisor's MANAGED_WORKER_SECRET
 LOGIN_ORIGIN=http://localhost:3030
 DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema=public
 # This sets the URL used for direct connections to the database and should only be needed in limited circumstances

.github/workflows/helm-prerelease.yml

@@ -52,12 +52,14 @@ jobs:
 
       - name: Lint Helm Chart
         run: |
-          helm lint ./hosting/k8s/helm/
+          helm lint ./hosting/k8s/helm/ \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml
 
       - name: Render templates
         run: |
           helm template test-release ./hosting/k8s/helm/ \
             --values ./hosting/k8s/helm/values.yaml \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml \
             --output-dir ./helm-output
 
       - name: Validate manifests

.github/workflows/release-helm.yml

@@ -47,12 +47,14 @@ jobs:
 
       - name: Lint Helm Chart
         run: |
-          helm lint ./hosting/k8s/helm/
+          helm lint ./hosting/k8s/helm/ \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml
 
       - name: Render templates
         run: |
           helm template test-release ./hosting/k8s/helm/ \
             --values ./hosting/k8s/helm/values.yaml \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml \
             --output-dir ./helm-output
 
       - name: Validate manifests

.server-changes/require-webapp-auth-secrets.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: breaking
+---
+
+Self-hosted deployments no longer ship shared default credentials; fresh installs generate their own. If yours still uses a previously published default, set a unique value before upgrading, or set `ALLOW_INSECURE_DEFAULT_SECRETS=true` to keep booting while you migrate.

apps/supervisor/.env.example

@@ -1,8 +1,8 @@
 # This needs to match the token of the worker group you want to connect to
 TRIGGER_WORKER_TOKEN=
 
-# This needs to match the MANAGED_WORKER_SECRET env var on the webapp
-MANAGED_WORKER_SECRET=managed-secret
+# Must match the webapp's MANAGED_WORKER_SECRET. Generate with: openssl rand -hex 16
+MANAGED_WORKER_SECRET=
 
 # Point this at the webapp in prod
 TRIGGER_API_URL=http://localhost:3030

apps/webapp/app/env.server.ts

@@ -85,6 +85,29 @@ const S2EnvSchema = z.preprocess(
   ])
 );
 
+// Previously published secret values must never be accepted, including when
+// an existing deployment or external secret manager still supplies one.
+const INSECURE_SECRET_VALUES = [
+  "managed-secret",
+  "2818143646516f6fffd707b36f334bbb",
+  "44da78b7bbb0dfe709cf38931d25dcdd",
+  "f686147ab967943ebbe9ed3b496e465a",
+  "447c29678f9eaf289e9c4b70d3dd8a7f",
+];
+
+// Escape hatch for deployments that can't rotate a published default yet (e.g.
+// ENCRYPTION_KEY protects existing data). Read raw: a refine can't see the
+// sibling parsed flag.
+const allowInsecureDefaultSecrets = ["true", "1"].includes(
+  (process.env.ALLOW_INSECURE_DEFAULT_SECRETS ?? "").toLowerCase().trim()
+);
+
+const isNotInsecureSecret = (value: string) =>
+  allowInsecureDefaultSecrets || !INSECURE_SECRET_VALUES.includes(value);
+
+const INSECURE_SECRET_MESSAGE =
+  "must not be a known-insecure published default; set a strong, unique value. If you cannot rotate it yet (e.g. it protects existing encrypted data or active sessions), set ALLOW_INSECURE_DEFAULT_SECRETS=1 to boot while you migrate.";
+
 const EnvironmentSchema = z
   .object({
     NODE_ENV: z.union([z.literal("development"), z.literal("production"), z.literal("test")]),
@@ -188,14 +211,15 @@ const EnvironmentSchema = z
     // Control-plane cache relax knobs. Unset -> defaults (DEFAULT_CP_CACHE_TTL_MS / _MAX_ENTRIES).
     CONTROL_PLANE_CACHE_TTL_MS: z.coerce.number().int().optional(),
     CONTROL_PLANE_CACHE_MAX_ENTRIES: z.coerce.number().int().optional(),
-    SESSION_SECRET: z.string(),
-    MAGIC_LINK_SECRET: z.string(),
+    SESSION_SECRET: z.string().min(1).refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
+    MAGIC_LINK_SECRET: z.string().min(1).refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
     ENCRYPTION_KEY: z
       .string()
       .refine(
         (val) => Buffer.from(val, "utf8").length === 32,
         "ENCRYPTION_KEY must be exactly 32 bytes"
-      ),
+      )
+      .refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
     WHITELISTED_EMAILS: z
       .string()
       .refine(isValidRegex, "WHITELISTED_EMAILS must be a valid regex.")
@@ -684,7 +708,11 @@ const EnvironmentSchema = z
     EVENTS_LOAD_SHEDDING_THRESHOLD: z.coerce.number().int().default(100000),
     EVENTS_LOAD_SHEDDING_ENABLED: z.string().default("1"),
 
-    MANAGED_WORKER_SECRET: z.string().default("managed-secret"),
+    MANAGED_WORKER_SECRET: z.string().min(1).refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
+
+    // Allow booting with a known-insecure published default secret. Temporary
+    // bridge for deployments that can't rotate yet; rotate as soon as possible.
+    ALLOW_INSECURE_DEFAULT_SECRETS: BoolEnv.default(false),
 
     // Tenant scoping on worker actions is header-driven (folded into the engine snapshot read) and
     // needs no flag. This is only the no-header fallback: when "1", a worker action on a run created
@@ -2113,3 +2141,24 @@ const EnvironmentSchema = z
 
 export type Environment = z.infer<typeof EnvironmentSchema>;
 export const env = EnvironmentSchema.parse(process.env);
+
+if (env.ALLOW_INSECURE_DEFAULT_SECRETS) {
+  const insecure = (
+    [
+      ["SESSION_SECRET", env.SESSION_SECRET],
+      ["MAGIC_LINK_SECRET", env.MAGIC_LINK_SECRET],
+      ["ENCRYPTION_KEY", env.ENCRYPTION_KEY],
+      ["MANAGED_WORKER_SECRET", env.MANAGED_WORKER_SECRET],
+    ] as const
+  )
+    .filter(([, value]) => INSECURE_SECRET_VALUES.includes(value))
+    .map(([name]) => name);
+
+  if (insecure.length > 0) {
+    console.warn(
+      `⚠️  ALLOW_INSECURE_DEFAULT_SECRETS is enabled and these secrets still use a known-insecure published default: ${insecure.join(
+        ", "
+      )}. This is insecure - rotate them as soon as you can.`
+    );
+  }
+}

apps/webapp/test/env.server.test.ts

@@ -0,0 +1,84 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+
+const originalEnv = process.env;
+
+const requiredEnv = {
+  NODE_ENV: "test",
+  DATABASE_URL: "postgresql://test:test@localhost:5432/test",
+  DIRECT_URL: "postgresql://test:test@localhost:5432/test",
+  SESSION_SECRET: "test-session-secret",
+  MAGIC_LINK_SECRET: "test-magic-link-secret",
+  ENCRYPTION_KEY: "test-encryption-keeeeey-32-bytes",
+  CLICKHOUSE_URL: "http://localhost:8123",
+  DEPLOY_REGISTRY_HOST: "registry.example.com",
+  MANAGED_WORKER_SECRET: "test-managed-worker-secret",
+};
+
+describe("webapp environment secrets", () => {
+  afterEach(() => {
+    process.env = originalEnv;
+    vi.resetModules();
+  });
+
+  it.each(["SESSION_SECRET", "MAGIC_LINK_SECRET", "ENCRYPTION_KEY", "MANAGED_WORKER_SECRET"])(
+    "requires %s to be explicitly set",
+    async (key) => {
+      process.env = { ...requiredEnv };
+      delete process.env[key];
+
+      await expect(import("../app/env.server")).rejects.toThrow(key);
+    }
+  );
+
+  it.each(["SESSION_SECRET", "MAGIC_LINK_SECRET", "ENCRYPTION_KEY", "MANAGED_WORKER_SECRET"])(
+    "rejects an empty %s",
+    async (key) => {
+      process.env = { ...requiredEnv, [key]: "" };
+
+      await expect(import("../app/env.server")).rejects.toThrow(key);
+    }
+  );
+
+  it.each([
+    ["SESSION_SECRET", "2818143646516f6fffd707b36f334bbb"],
+    ["MAGIC_LINK_SECRET", "44da78b7bbb0dfe709cf38931d25dcdd"],
+    ["ENCRYPTION_KEY", "f686147ab967943ebbe9ed3b496e465a"],
+    ["MANAGED_WORKER_SECRET", "managed-secret"],
+    ["MANAGED_WORKER_SECRET", "447c29678f9eaf289e9c4b70d3dd8a7f"],
+  ])("rejects the known-insecure default value for %s", async (key, insecureValue) => {
+    process.env = { ...requiredEnv, [key]: insecureValue };
+
+    await expect(import("../app/env.server")).rejects.toThrow(key);
+  });
+
+  it("accepts explicitly configured secrets", async () => {
+    process.env = { ...requiredEnv };
+
+    const { env } = await import("../app/env.server");
+
+    expect(env.SESSION_SECRET).toBe(requiredEnv.SESSION_SECRET);
+    expect(env.MAGIC_LINK_SECRET).toBe(requiredEnv.MAGIC_LINK_SECRET);
+    expect(env.ENCRYPTION_KEY).toBe(requiredEnv.ENCRYPTION_KEY);
+    expect(env.MANAGED_WORKER_SECRET).toBe(requiredEnv.MANAGED_WORKER_SECRET);
+  });
+
+  it("allows a known-insecure default when ALLOW_INSECURE_DEFAULT_SECRETS is set", async () => {
+    process.env = {
+      ...requiredEnv,
+      ALLOW_INSECURE_DEFAULT_SECRETS: "1",
+      ENCRYPTION_KEY: "f686147ab967943ebbe9ed3b496e465a",
+      MANAGED_WORKER_SECRET: "managed-secret",
+    };
+
+    const { env } = await import("../app/env.server");
+
+    expect(env.ENCRYPTION_KEY).toBe("f686147ab967943ebbe9ed3b496e465a");
+    expect(env.MANAGED_WORKER_SECRET).toBe("managed-secret");
+  });
+
+  it("still rejects an empty secret even with ALLOW_INSECURE_DEFAULT_SECRETS", async () => {
+    process.env = { ...requiredEnv, ALLOW_INSECURE_DEFAULT_SECRETS: "1", SESSION_SECRET: "" };
+
+    await expect(import("../app/env.server")).rejects.toThrow("SESSION_SECRET");
+  });
+});

apps/webapp/test/registryConfig.test.ts

@@ -10,6 +10,9 @@ describe("getRegistryConfig", () => {
     MAGIC_LINK_SECRET: "test-magic-link-secret",
     ENCRYPTION_KEY: "test-encryption-keeeeey-32-bytes",
     CLICKHOUSE_URL: "http://localhost:8123",
+    PROVIDER_SECRET: "test-provider-secret",
+    COORDINATOR_SECRET: "test-coordinator-secret",
+    MANAGED_WORKER_SECRET: "test-managed-worker-secret",
   };
 
   beforeEach(() => {

apps/webapp/test/setup.ts

@@ -13,6 +13,9 @@ config({ path: path.resolve(__dirname, "../.env") });
 // the pair — the ioredis mock below forces lazyConnect, so nothing ever dials.
 process.env.REDIS_HOST ??= "localhost";
 process.env.REDIS_PORT ??= "6379";
+process.env.PROVIDER_SECRET ??= "test-provider-secret";
+process.env.COORDINATOR_SECRET ??= "test-coordinator-secret";
+process.env.MANAGED_WORKER_SECRET ??= "test-managed-worker-secret";
 
 // Worker singletons construct a RedisWorker at import time whose ioredis client
 // connects eagerly, so any test importing the service graph opens real Redis

docs/self-hosting/docker.mdx

@@ -75,12 +75,23 @@ git clone --depth=1 https://github.com/triggerdotdev/trigger.dev
 cd trigger.dev/hosting/docker
 ```
 
-2. Create a `.env` file
+2. Create a `.env` file and generate secrets
 
 ```bash
 cp .env.example .env
+
+# Fills the required secrets in .env with strong, unique values.
+# Safe to re-run - it never overwrites a secret you've already set.
+./generate-secrets.sh
 ```
 
+<Note>
+  The stack ships no working default credentials. `generate-secrets.sh` fills the
+  application secrets and the bundled datastore passwords with strong, unique values.
+  Keep them safe - rotating the encryption key or session secret later will invalidate
+  existing sessions and encrypted data.
+</Note>
+
 3. Start the webapp
 
 ```bash
@@ -130,6 +141,13 @@ docker compose up -d
 
 4. Configure the supervisor using the [environment variables](/self-hosting/env/supervisor) in your `.env` file, including the [worker token](#worker-token).
 
+<Warning>
+  For a split webapp/worker setup, set `MANAGED_WORKER_SECRET` on the worker to
+  the **same** value as the webapp's `MANAGED_WORKER_SECRET`. Don't run
+  `generate-secrets.sh` on the worker host - it would create a mismatched value
+  and the worker would fail to authenticate.
+</Warning>
+
 5. Apply the changes:
 
 ```bash