fix: stop the Hub API from requiring enrichment credentials, and honor LOG_LEVEL in hub-worker (ENG-1916)

[xernobyl]

What does this PR do?

Local Hub crashes on boot when the sentiment/emotion providers are configured but the Google credentials file isn't reachable inside the container — the API exits 1 with sentiment config: ... failed to find default credentials, nothing listens on :8080, and every web-side Hub call fails with a generic fetch failed.

Linear: https://linear.app/formbricks/issue/ENG-1916/dev-hub-crashes-on-boot-when-gemini-enrichment-creds-arent-mounted

The root cause turned out to be a bit more interesting than a dev-env problem. The API was building sentiment, emotions, and translation LLM clients at startup and passing each to exactly one consumer: river.AddWorker. But the API's River client is never Start()ed — hub-worker runs the workers — so the API cannot work those jobs even in principle. It was constructing credentialed Vertex clients it can never call, and any one of them failing took the whole HTTP API down with it.

The registration existed to satisfy River's insert-time unknown-kind check, but that check only runs when Workers is non-nil (client.go:2228-2239), and declaring a queue is never required in order to insert onto it (client.go:1693-1695). So the requirement was self-imposed: building the client insert-only (river.Config{}, both Workers and Queues omitted — River rejects setting one without the other) makes it disappear, and with it the API's need for credentials it can't use.

This is not the graceful degradation the ticket proposed, on purpose. hub-worker still fails fast when a credential is missing, because silently disabling a provider somebody explicitly configured turns a visible boot failure into an unnoticed data-quality outage — you'd deploy green and quietly enrich nothing for weeks. Config errors should fail loudly; what was wrong here was the blast radius, not the fail-fast. This narrows it to the process that actually needs the credential.

It's worth flagging that this is production-relevant and not only a local-dev annoyance: enable google-gemini in Helm values without the matching IAM binding and today the entire API CrashLoopBackOffs, taking down all feedback ingestion rather than just enrichment.

Embeddings are deliberately untouched — the API keeps its embedding client, because search uses it synchronously to embed the incoming query. So EMBEDDING_PROVIDER=google-gemini does still require credentials in the API, and that's correct.

Trade-off

Going insert-only gives up River's insert-time UnknownJobKindError, so a kind added to the insert side without a matching worker would strand jobs silently. internal/workers had no test for its wiring at all, so this adds one and puts that invariant in it, checked against a new service.JobKindSpecs. River exposes no way to enumerate a bundle's registered kinds (workersMap is unexported with no accessor), so registration is observed via AddWorkerSafely, which errors only on a duplicate kind. Net effect: the guard moves from boot time (where it needed credentials) to test time (where it doesn't).

JobKindSpecs also collapses riverDepthQueues, which was a third hand-maintained copy of the queue set. wiring.go still names its own queues, since it pairs each with a per-enrichment MaxWorkers that the list has no notion of — the two are kept in step by assertion rather than derivation, which is what the parity test is for.

Dropping placeholderMaxWorkers is fallout: it existed only for the API (which passed 1 for workers it never ran), so with the API registering nothing, hub-worker is the only caller and it always passed 0.

One thing for reviewers to weigh

A deployment can now have a healthy API happily enqueueing jobs while hub-worker crash-loops — enrichment stops while ingestion continues. That's the intended narrowing, but it does move detection off "the API is down" and onto the queue signals. Both are preserved: the River queue-depth gauge (now derived from JobKindSpecs, so a new kind can't be added without its queue appearing on the gauge) and the enrichment-backlog gauge. Happy to add an explicit alert suggestion if that feels thin.

Not closed by this PR

ENG-1916 also asks for the dev docker-compose.dev.yml credential mount and an .env.example guardrail. Those live in the formbricks monorepo and come in a follow-up — and that follow-up is the part that actually gets Gemini enrichment running locally, which is what the ticket was filed for. This PR stops the API dying; it doesn't make enrichment work.

Also here: hub-worker now honors LOG_LEVEL

This came up while debugging the above. hub-worker never configured slog, so it ignored LOG_LEVEL entirely and ran on Go's default handler, while hub-api honored it. That matters more after this change, because once the API stops dying the worker's logs are the only place the actionable credential error appears — and its debug lines were unreachable.

The level parsing and handler setup move into internal/observability, which already owns the logging handler, and both entrypoints call it. Living there also means the logic is covered by make test-unit, which runs ./cmd/api and ./internal/... but not ./cmd/worker.

Heads-up for operators — the worker's output changes, not just its level. It previously wrote to stderr as:

2026/07/28 10:37:10 INFO Worker running client_id=...

and now matches hub-api, writing to stdout as:

time=2026-07-28T10:37:10.962Z level=INFO msg="Worker running" client_id=...

Container log collection is unaffected (both streams are captured), but anything parsing hub-worker output or relying on stderr/stdout separation needs updating. Worth a line in the release notes.

hub-api is unchanged, including that it configures logging before reporting a config-load failure so a broken config still produces output — the worker now does the same. Unrecognized levels still fall back to info rather than silencing logs. One pre-existing oddity preserved rather than fixed: only the exact string warn matches, so LOG_LEVEL=warning falls back to info. Pinned in a test; happy to make it accept warning if that reads as a trap.

How should this be tested?

Unit, lint and integration:

make fmt && make lint && make build && make test-unit && make tests

Note on make tests locally: it sources .env, and if your DATABASE_URL points at a Postgres that isn't running you'll get a wall of connection failures unrelated to this change. Against a live test_db it's green:

DATABASE_URL="postgresql://postgres:postgres@localhost:5432/test_db?sslmode=disable" \
  go test ./tests/... -timeout 180s

End-to-end, reproducing the ticket. Build this branch, point HUB_IMAGE_TAG at it, and set the enrichment providers with a credentials path that does not exist inside the container:

SENTIMENT_PROVIDER=google-gemini
SENTIMENT_MODEL=gemini-2.5-flash
SENTIMENT_GOOGLE_CLOUD_PROJECT=some-project
SENTIMENT_GOOGLE_CLOUD_LOCATION=us-central1
EMOTIONS_PROVIDER=google-gemini          # same shape
TRANSLATION_PROVIDER=google-gemini       # same shape, plus TRANSLATION_DEFAULT_LANGUAGE
EMBEDDING_PROVIDER=openai
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_PROVIDER_API_KEY=sk-anything-unused-at-boot
GOOGLE_APPLICATION_CREDENTIALS=/does/not/exist/in/container.json

Expected, and what I saw:

	ghcr.io/formbricks/hub:0.8.1	this branch
API boot	ERROR Failed to create application → exit 1	INFO Starting server port=8080
GET /health	nothing listening	200
POST /v1/feedback-records	—	201, record persisted
hub-worker	exit 1	exit 1 (unchanged, on purpose)

Then confirm inserts still land correctly with no Workers/Queues declared — this is the load-bearing bit, so worth checking rather than trusting:

select kind, queue, state from river_job order by kind;

All six kinds should appear on their own queues, matching service.JobKindSpecs exactly:

 feedback_embedding          | embeddings            | available
 feedback_emotions           | emotions              | available
 feedback_sentiment          | sentiments            | available
 feedback_translation        | translations          | available
 tenant_translation_backfill | translation_backfills | available
 webhook_dispatch            | default               | available

(webhook_dispatch needs a webhook registered first; tenant_translation_backfill is triggered by a PATCH /v1/tenants/{tenant_id}/settings writing target_language.)

Both new guards were checked by breaking them deliberately, so they aren't just passing decoration: dropping the sentiment registration from wiring.go fails the parity test on its missing queue (queue "sentiments" for kind "feedback_sentiment" missing from queue config), and sourcing a queue's MaxWorkers from the wrong enrichment fails the concurrency test (MaxWorkers = 5, want 4).

For the LOG_LEVEL part, run the worker briefly with LOG_LEVEL=error against any valid DATABASE_URL/API_KEY:

docker run --rm -e API_KEY=... -e DATABASE_URL=... -e LOG_LEVEL=error \
  --entrypoint sh <image> -c '/app/hub-worker & sleep 3; kill %1'

ghcr.io/formbricks/hub:0.8.1 still prints INFO lines in the default-handler format (2026/07/28 ... INFO Worker running ...), i.e. the level is ignored. This branch prints nothing at INFO, and with LOG_LEVEL unset prints time=... level=INFO msg="Worker running" ..., matching hub-api. ParseLogLevel also has a table test covering casing and the unknown-value fallback.

Beyond that I exercised the wider surface to check nothing regressed: full CRUD on feedback records (create/get/list/count/patch/delete, plus 401 unauthenticated and 404 after delete), tenant settings get/patch, semantic search returning 503 with embeddings disabled and reaching the provider when enabled, the worker draining a real webhook_dispatch job (attempted and retried on a non-2xx), and SIGTERM giving a clean exit 0 with Server stopped / Worker stopped on both binaries.

Enrichment itself was verified end to end against a stubbed OpenAI-compatible endpoint (no credentials, no external egress): feedback_sentiment and feedback_emotions both reached completed on the first attempt and the values persisted on the record. Disabling sentiment for the tenant then produced an emotions job but no sentiment job, so the per-tenant enrichment gate still applies.

Checklist

Required

• Filled out the "How to test" section in this PR
• Read Repository Guidelines
• Self-reviewed my own code
• Commented on my code in hard-to-understand bits
• Ran make build
• Ran make tests (integration tests in tests/) — green against a live test_db, see the note above
• Ran make fmt and make lint; no new warnings
• Removed debug prints / temporary logging
• Merged the latest changes from main onto my branch with git pull origin main
• If database schema changed: added migration in migrations/ — n/a, no schema change

Appreciated

• If API changed: added or updated OpenAPI spec — n/a, no API surface change (routes, request and response shapes are all untouched)
• If API behavior changed: added request/response examples — n/a, externally observable behavior is unchanged except that the API now boots where it previously exited
• Updated docs in docs/ if changes were necessary — n/a here; the env-var guardrail lands in the formbricks follow-up
• Ran make tests-coverage for meaningful logic changes — 74.3% total, comfortably over the 15% gate

[coderabbitai]

Walkthrough

The API now creates an insert-only River client and performs embedding search synchronously without registering embedding workers. Shared job specifications derive queue names, while worker wiring uses configured concurrency and optional-client registration. API and worker startup logging now uses centralized observability helpers with case-insensitive log-level parsing and info fallback. Tests cover River client validation, embedding handler setup, worker registration and concurrency, and log-level parsing.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main changes: making the API insert-only for enrichment jobs and updating worker log-level handling.
Description check	✅ Passed	The description is thorough and matches the template with purpose, testing steps, and checklist, even if the issue reference is a Linear link.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[xernobyl]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/service/job_kinds.go`:
- Around line 41-56: Add a focused unit test for JobQueueNames that exercises
duplicate-queue handling by using or arranging job-kind specs with at least two
entries sharing the same queue, then assert the returned queue names contain
that queue only once. Keep the test centered on the seen-map deduplication path.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c66e63a4-a113-4f27-9afe-e5bbe9be1bc2

📥 Commits

Reviewing files that changed from the base of the PR and between ee60d7f and d98b843.

📒 Files selected for processing (10)

• cmd/api/app.go
• cmd/api/app_test.go
• cmd/api/main.go
• cmd/worker/app.go
• cmd/worker/main.go
• internal/observability/logging.go
• internal/observability/logging_test.go
• internal/service/job_kinds.go
• internal/workers/wiring.go
• internal/workers/wiring_test.go

[xernobyl]

Fixed in 699ac69 — you were right, the dedup branch was unreachable because every declared kind owns its own queue.

Rather than test it indirectly, I split it into distinctQueues(specs) so a test can pass specs that share a queue, and pinned the kind-to-queue pairing while there (the API's queue-depth gauge and the worker-registration parity test both read it, so drift there is worth catching).

[BhagyaAmarasinghe]

LGTM!