Scheduled job sends concurrent settlements causing batch nonce conflicts

[whoabuddy]

Problem

Payment settlement errors from x402-api-host cluster at exact 4-hour intervals with 3-5 failures per burst, all from concurrent requests hitting the relay simultaneously. This pattern is consistent with a scheduled/cron job that fires multiple API calls at once.

Evidence from production logs (2026-03-25)

Error pattern — exact 4-hour intervals

Time (UTC)	Errors	Error Reason
01:00:12-01:00:16	2	conflicting_nonce
05:00:04-05:00:11	4	conflicting_nonce
09:00:04-09:00:11	5	conflicting_nonce
13:00:05-13:00:08	4	conflicting_nonce

Total: 17 errors today (up from 8 yesterday, 1 the day before, 0 the three days before that).

All errors are nonce conflicts from concurrent requests

{
  "path": "/inference/openrouter/chat",
  "errorReason": "conflicting_nonce",
  "transaction": "",
  "network": "stacks:1"
}

Paths hit: /inference/openrouter/chat and /storage/paste — both x402-protected endpoints called within the same second.

The trend is worsening

Date	Errors
Mar 19-21	0
Mar 22	0
Mar 23	1
Mar 24	8
Mar 25	17

Whatever scheduled job is calling these endpoints appears to be increasing in concurrency or frequency.

Root cause

The relay's nonce pool is designed to handle concurrent requests (atomic assignment via blockConcurrencyWhile), but when multiple settlements arrive in the same second, they each get a nonce, both broadcast, and the node rejects one with ConflictingNonceInMempool because the first transaction hasn't been indexed yet.

The relay returns NONCE_CONFLICT with retryAfter: 30, but the calling code doesn't retry — it logs the error and moves on.

Proposed fix

Option A: Serialize settlements (recommended)

If the scheduled job calls multiple x402 endpoints, serialize them — wait for each settlement to complete before starting the next:

// Instead of:
await Promise.all([
  fetch("/inference/openrouter/chat", { headers: paymentHeaders }),
  fetch("/storage/paste", { headers: paymentHeaders }),
]);

// Do:
await fetch("/inference/openrouter/chat", { headers: paymentHeaders });
await fetch("/storage/paste", { headers: paymentHeaders });

Option B: Add jitter between calls

If parallelism is needed for performance, add 2-5 second random jitter between calls:

for (const endpoint of endpoints) {
  await fetch(endpoint, { headers: paymentHeaders });
  await new Promise(r => setTimeout(r, 2000 + Math.random() * 3000));
}

Option C: Retry on NONCE_CONFLICT

This is already tracked in #84. When the relay returns retryAfter: 30 and code: "NONCE_CONFLICT", wait and retry. This would recover from the conflict without changing the call pattern.

Related

• Retry settlement on conflicting_nonce with backoff #84 — Retry settlement on conflicting_nonce with backoff (general retry mechanism)
• Review and improve error responses for agent self-correction #85 — Review and improve error responses for agent self-correction
• aibtcdev/x402-sponsor-relay#216 — relay chaining limit undercounts in-flight nonces

[tfireubs-ui]

PR #89 (fix(x402): retry settlement on conflicting_nonce with exponential backoff) directly addresses the root cause identified here.

The middleware (src/middleware/x402.ts) currently returns 402 immediately when settleResult.success is false with errorReason: 'conflicting_nonce' — no retry. PR #89 adds 3-attempt exponential backoff (1s → 2s → 4s) with jitter before giving up, which should absorb the concurrent-submission race condition at the 4h cron boundary without requiring changes to the calling agent.

The concurrent caller pattern (two endpoints settling within ~1s of each other) doesn't need to change — the relay just needs the client to back off and retry. #89 does that.

1x APPROVED (arc0btc) + CI green. Blocking: needs a second review.