fix(self-hosted): honest send outcomes, real per-provider SES credentials, and a reconciliation path for uncertain sends

[andrei-hasna]

Supersedes #53 (its two commits are included verbatim as the first two commits here) and adds the two causes that PR left untouched, plus the reconciliation path.

1. Error semantics, up to date with origin/main (criterion 3)

PR #53's commits a4ae6d9 + 88c8381 are cherry-picked unchanged; their merge base was already current origin/main (1ac2d15), so there was nothing to rebase.

PR #53's verify job was red and is fixed here. scripts/generate-selfhost-sdk.ts anchored its nullable-message patches on whole interface literals, so the new reason/provider_error/sent fields on SendMessageError stopped the needle matching and the generator threw generated SDK shape changed; nullable send failure message was not injected. The patches are now anchored on the interface name plus the property, and src/selfhost.ts is regenerated and committed.

2. Genuine per-provider credentials (cause 2)

buildSelfHostedSender hardcoded access_key: null, secret_key: null, so the SES client always fell through to the AWS default chain — the deployment IAM role — no matter what was configured.

• Server: signs with EMAILS_SES_ACCESS_KEY_ID + EMAILS_SES_SECRET_ACCESS_KEY when both are present, deployment IAM role otherwise (unchanged default for OSS self-hosters). Half a pair is a hard startup error. The names are scoped on purpose — injecting the generic AWS_ACCESS_KEY_ID would re-point the entire AWS SDK default chain in the container (S3 inbound, SQS ingest), not just SES.
• A secret-free boot line now reports which identity outbound mail is signed with (provider=ses credentials=environment|ambient_aws_env|deployment_role), resolved through the same function the adapter signs with, so the reported identity cannot drift from the real one.
• Adapter: SESAdapter used to pair provider.access_key with the ambient AWS_SECRET_ACCESS_KEY when only one half was present — signing with an identity belonging to neither source. Resolution is now one exported rule (resolveSesCredentials): a provider pair is used exclusively, a partial pair is rejected, only a complete ambient pair (with session token) is a fallback.
• provider add/update validate WITH the supplied credentials, before anything is persisted — the old order created the row, validated the stored (credential-free) row against the CLI machine's ambient chain, and printed Provider credentials are invalid: Could not load credentials from any providers. Reproduced on main and asserted gone.
• Against a self-hosted server the credential flags are now refused with an error naming the server-side variables to set instead, and no row is created — because the self-hosted providers resource has no credential columns and the values were being dropped silently on the wire.
• Optional EMAILS_SES_CONFIGURATION_SET is threaded into SendEmailCommand, so SES metrics are attributable in a shared account.
• emails send --provider was parsed and then discarded in both modes. Now honoured locally, explicitly rejected in self_hosted mode.
• Corrected the comment in auth/mailer.ts claiming the sender "already targets" a specific SES account via the deployment role. It does not, and there is no cross-account assume-role anywhere in this codebase.

This PR does not put any credential value in the repo, in a task definition environment block, or in a log. Where the values go (Secrets Manager → ECS secrets by ARN) is deployment work, not code.

3. Both directions of honesty, as tests

src/server/self-hosted/send-honesty-and-reconciliation.integration.test.ts asserts each direction over the whole response — status class, sent, presence of an error key, retry_safe, the ledger row, and what the CLI's own serializer renders — not one field:

• A. provider accepted ⇒ nothing may read as failure: fresh success, provider-accepted-then-ledger-write-failed, idempotent replay; plus emails log rendering it as sent.
• B. provider rejected ⇒ nothing may read as sent: SES sandbox MessageRejected (the literal incident error), indeterminate network failure, policy block; plus emails log/--json rendering failed, never blank.
• Client seam: SelfHostedMailDataSource.send() resolves for every accepted path and throws for reject / indeterminate.

4. Reconciliation for uncertain (criterion 6)

Seven rows sat in send_state = 'uncertain' with no command able to list them and none able to resolve them.

• GET /v1/messages/send-intents/uncertain — exactly the rows whose outcome was never established.
• POST /v1/messages/send-intents/reconcile — close out one row against evidence. Guarded on send_state = 'uncertain' (a proven outcome is never overwritten, a live send is never raced); asserting sent requires the provider message id that proves it; the evidence note is mandatory; outcome + evidence + timestamp + resolving principal are persisted on the row; ids resolve through the same prefix/404/409 path as every other message route; tenant-scoped.
• CLI: emails send-intent uncertain, emails send-intent reconcile <id> --outcome sent|not-sent [--provider-message-id …] --evidence "…".

Nothing infers an outcome and nothing is bulk-applied.

5. Real test results

Environment: Bun 1.3.14, Postgres 16.4 in a dedicated container.

check	result
bunx tsc --noEmit	clean
local-mode suite (CI env-scrubbing replicated)	2088 pass / 0 fail / 108 skip (2196 across 210 files). Baseline on origin/main: 2059 pass / 0 fail
bun run build	ok
regenerate SDK + git diff --exit-code -- src/selfhost.ts	in sync
canonical package identity assertion	ok
no-cloud:source / no-cloud:pack	6 pass / 0 fail, no hosted markers
git diff --check	clean
postgres.integration	15 pass / 0 fail
message-id-resolution.integration	5 pass / 0 fail
rls.integration	10 pass / 0 fail
send-failure-semantics.integration	6 pass / 0 fail
send-honesty-and-reconciliation.integration (new)	17 pass / 0 fail
multi-tenancy.integration	32 pass / 1 fail — pre-existing

Pre-existing failures, reported honestly and NOT caused by this PR:

• multi-tenancy.integration.test.ts › tenant-scoped key issuance (WI-2e) › an owner mints a tenant key that resolves back to the same tenant fails identically on origin/main (1ac2d15) — verified by running it in a detached worktree at that commit against the same database.
• CI's container-runtime job (trivy HIGH/CRITICAL on the patched Bun base) is red on main too and is not touched here.

6. Fails-before / passes-after evidence

Verified by copying the new tests onto a detached worktree at origin/main and at PR #53's branch:

• src/providers/ses-credentials.test.ts — on main: SyntaxError: Export named 'resolveSesCredentials' not found. Here: 7 pass.
• sender.test.ts credential block — on main: SyntaxError: Export named 'classifyProviderSendError' not found. Here: 13 pass.
• db/providers.test.ts credential refusals — on main: 3 fail (expect(received).toThrow(expected) — credentials were silently accepted and dropped).
• cli/commands/provider.test.ts › refuses SES credentials — on main the received message is literally Provider credentials are invalid: Could not load credentials from any providers. Provider was not saved. Here it names EMAILS_SES_ACCESS_KEY_ID / EMAILS_SES_SECRET_ACCESS_KEY and echoes no credential value.
• cli/commands/provider.test.ts › registers credential-free metadata — on main this hangs, because provider add validated the stored credential-free row against a live AWS call. That hang is the defect.
• send-honesty-and-reconciliation.integration.test.ts — on main: fails at import. On PR fix(self-hosted): 502 la trimiteri externe deși mesajul pleacă — semantică corectă de eșec + idempotență la retry #53's branch: 8 pass / 8 fail, and every one of the 8 failures is a C: reconciliation case, i.e. exactly this PR's delta beyond fix(self-hosted): 502 la trimiteri externe deși mesajul pleacă — semantică corectă de eșec + idempotență la retry #53.

7. What this PR deliberately does not do

• No AWS/Secrets Manager/ECS change, no publish, no deploy. Cause 1 (wrong AWS account) is unblocked by the code here but is closed by putting the alumia key pair into Secrets Manager and referencing it from the task definition secrets block by ARN.
• No reconciliation of the seven live production rows — this PR ships the mechanism; running it against production with per-message evidence is a separate, evidence-recorded step.
• The @hasna/mailery vs @hasna/emails package-identity mismatch is untouched (ci.yml still enforces @hasna/mailery); it must be resolved before any npm publish.

8. Adversarial self-review (no second reviewer available in this session)

Findings raised and fixed during the pass, not after:

1. Reported vs real identity could drift. The boot label was first derived from "did we set EMAILS_SES_*?", while SESAdapter independently falls back to a static AWS_* pair from process.env. That is the same category of defect as the incident. Fixed by resolving the label through resolveSesCredentials — the function the adapter signs with — which also surfaces a malformed pair at boot instead of at first send (getAdapter builds lazily).
2. reconcile could 500 on a malformed id. getMessage hits a uuid column directly. Routed through resolveMessageIdOrError, matching every other message route: short prefixes resolve, unknown ids 404, ambiguous ids 409. Test added.
3. Cross-tenant reconciliation. Explicitly tested: another tenant sees count: 0 and gets 404, and the row is unchanged.
4. Quota semantics. not_sent → failed (excluded from the outbound quota window), sent → sent (included). Matches markSendFailed / completeSendIntent.
5. Secret leakage. Staged diff scanned for credential patterns before commit; only test placeholders (AKIA-AMBIENT, provider-secret) matched. The refusal error names variables, never values, and tests assert the supplied secret is not echoed. No Co-Authored-By trailer.
6. Behaviour change flagged: a local SQLite provider row carrying only one half of a key pair used to work by borrowing the ambient secret; it now fails with an explicit error. That is intentional — the old behaviour signed with a mixed identity — and is called out in the CHANGELOG.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[andrei-hasna]

CI on this PR head (c527d08) vs main (1ac2d15)

job	this PR	main	verdict
verify	success	success	PR #53's verify regression (SDK generator) is fixed
selfhost-postgres	failure	failure	pre-existing, identical single test
container-runtime	failure	failure	pre-existing (trivy HIGH/CRITICAL on the patched Bun base)
validate (terraform-aws-validate)	failure	failure	pre-existing — red on main on 2026-07-21 and 2026-07-22; no terraform touched here

selfhost-postgres detail, now that all six suites actually run in one invocation:

(fail) tenant-scoped key issuance (WI-2e) > an owner mints a tenant key that resolves back to the same tenant
 85 pass
 1 fail
Ran 87 tests across 6 files.

Byte-identical to the local run against a dedicated Postgres 16.4 container, and to main when that one test is run at 1ac2d15. Before the ci.yml change the job aborted at that test under set -e, so the send-semantics suites (including the one PR #53 added) had never executed in CI.

This PR introduces no new CI failure.