Custom background tasks: deduplication

[2chanhaeng]

Second sub-issue of #206.

Background

Background tasks frequently need at-most-once-per-key enqueue semantics: a digest mailer should not send twice if a request is retried, a cleanup job should coalesce duplicate triggers. The maintainer asked that this mirror the nativeRetrial capability flag introduced in #250 — backends that deduplicate natively own the check; otherwise Fedify provides a best-effort KV fallback, with the race-condition tradeoff documented explicitly.

This is kept separate from the core API so the first PR stays small and the deduplication semantics (including the documented best-effort limitation) get their own reviewable boundary.

Public API

MQ-layer primitives

// mq.ts
export interface MessageQueue {
  readonly nativeRetrial?: boolean;       // existing, #250
  readonly nativeDeduplication?: boolean; // new — backend dedups same deduplicationKey
  // …
}

export interface MessageQueueEnqueueOptions {
  readonly delay?: Temporal.Duration;     // existing
  readonly orderingKey?: string;          // existing
  readonly deduplicationKey?: string;     // new
}

These are MQ-layer primitives, not task-layer concepts, so they survive the Approach 2 Worker extraction unchanged and are reusable by any future enqueue path.

Task-API surface

• Add deduplicationKey?: string to TaskEnqueueOptions (the core sub-issue ships TaskEnqueueOptions without it; adding an optional field is non-breaking).
• FederationOptions.taskDeduplicationTtl?: Temporal.DurationLike (default 1 hour) — TTL for the KV fallback entry.
• FederationOptions.taskDeduplicationFallback?: "open" | "closed" (default "open") — behavior when deduplicationKey is set but the queue does not declare nativeDeduplication and the KV adapter exposes no conditional-write primitive: "open" logs at debug and proceeds; "closed" throws TypeError synchronously.
• New taskDeduplication KV prefix (default ["_fedify", "taskDeduplication"]), separate from activityIdempotence.

Resolution path

Inside #enqueueTasks, after the queue is resolved, when deduplicationKey is supplied:

1. If queue.nativeDeduplication === true: forward deduplicationKey in MessageQueueEnqueueOptions; the backend owns the check; Fedify does not touch KV.
2. Otherwise: attempt a conditional KV write under taskDeduplication with the TTL, using an onlyIfNotExists-style guard where supported (Deno KV atomic().check(), Postgres INSERT … ON CONFLICT DO NOTHING, Redis SET NX, SQLite INSERT OR IGNORE). Key present → skip the enqueue; write succeeded → proceed.
3. KV adapter has no conditional-write primitive → branch on taskDeduplicationFallback ("open" proceeds, "closed" throws).

For enqueueTaskMany, a single deduplicationKey applies to the whole batch (documented restriction; per-item dedup means calling enqueueTask in a loop). This preserves the atomicity guarantee of nativeDeduplication backends, which accept one key per call.

Documented limitation

The check-then-enqueue sequence in the KV fallback is not atomic: two concurrent enqueuers can both observe a missing key and both write. This is best-effort and stated in the public JSDoc for deduplicationKey; production deployments needing strict guarantees use a backend with nativeDeduplication: true. Cleanup is by TTL expiry, not active deletion on handler success (active cleanup introduces a success→crash-before-delete window; deferred to a later enhancement).

Out of scope

• Active KV cleanup on handler success (TTL-only for v1).
• Adding nativeDeduplication: true to the first-party adapter packages (packages/postgres, packages/redis, etc.) — track per-adapter follow-ups; this sub-issue ships the core flag + KV fallback, and each adapter opts in separately.

Acceptance criteria

• deduplicationKey on a nativeDeduplication: true queue is forwarded; Fedify does not write KV.
• deduplicationKey on a default queue: a second enqueue inside the TTL is skipped; re-enqueue after TTL expiry succeeds.
• taskDeduplicationFallback: "closed" throws synchronously when no conditional write is available; "open" proceeds with a debug log.
• taskDeduplication KV prefix does not collide with activityIdempotence.
• enqueueTaskMany applies one batch-level deduplicationKey.
• Best-effort race limitation documented in JSDoc and docs/manual/tasks.md; CHANGES.md updated; AI usage disclosed per AI_POLICY.md.

References

• Umbrella issue (design discussion + background): Support custom background tasks in worker #206 · depends on the core API sub-issue
• nativeRetrial prior art (the native-capability flag pattern this mirrors): Add automatic retry capability flag to MessageQueue interface #250