fix(sfmc): stop async poll hanging on large /results payloads

[rokatyal]

Summary

performPoll for SFMC asyncDataExtension hangs indefinitely whenever a batch completes with errors, until the caller's deadline expires and reports a synthetic 504. No records are ever delivered, and the poll retries into the identical hang forever.

The /results call now sets skipResponseCloning: true.

Root cause

performPoll calls /results only on the Complete + Has Errors path — the resultStatus === 'OK' path returns early and never touches it. That response carries one item per failed record, so for a batch of any real size it exceeds the 16KB highWaterMark of the PassThrough tee that response.clone() sets up in prepare-response.ts:

const clone = response.clone()
content = await clone.text()   // never resolves once the tee fills

Both tee branches must drain. The middleware awaits the clone while the original response body goes unread, so back-pressure stalls the pipe and clone.text() waits forever. It is not an error and produces no log line or span — the HTTP span closes normally because the network did finish; the hang is userland body consumption afterward.

skipResponseCloning (declared at request-client.ts:67, honored at prepare-response.ts:10) takes the response.text() path instead, with no tee and no deadlock.

Prior art

Same root cause, same file, same line, same fix as #2461 ("Iterable Lists bugfix: cloned response hangs operations between this Destination and Iterable", merged 2024-10-01), which reported it as "hangs operations whose response payload is large, creating timeouts during audience syncs". That PR set the flag at Iterable's own call sites; SFMC was never covered.

Reproduction

Against cross-fetch@3.2.0 → node-fetch@2.7.0 (the pinned client), with a /results-shaped body served gzipped + chunked, replicating prepare-response.ts exactly (clone, read the clone, never read the original):

items=1     raw=170B     gzip=148B -> resolved 170 chars in 7ms
items=50    raw=4631B    gzip=184B -> resolved 4631 chars in 2ms
items=1640  raw=149325B  gzip=686B -> *** HUNG (never resolved) ***

With skipResponseCloning: true, the identical 1640-item body:

skipResponseCloning=true  items=1640  decompressed=149325B -> OK, 1640 items parsed (4ms)
skipResponseCloning=true  items=1     decompressed=170B    -> OK, 1 item parsed     (2ms)

The gzip ratio is what makes this easy to miss: a 1.59KB compressed body is ~145KB decompressed, and the tee buffers decompressed bytes.

Observed in stage

Every poll for a 1640-record SFMC batch, across 4 pods and 3 aggregation IDs:

• /status → 200 in 0.316s, /results → 200 in 0.449s
• then 239 seconds of no logs and no spans
• destination call timed out / destination call aborted before starting, status 504, success_count: 0
• caller re-polls on RETRYABLE_ERROR and hits the same hang

Regression test

asyncDataExtension.async.test.ts already covers the Complete but Has Errors path, but with a
4-item fixture that stays under the threshold and passes with or without the fix. This PR adds a
1640-item case — the batch size that surfaced the hang in stage.

Verified the test is load-bearing (same nock/cross-fetch@3.2.0 path the suite uses):

New regression test, 1640 items:
  WITHOUT the fix (current staging code):
  skipResponseCloning=false items=1640 -> FAIL  *** test times out (poll hangs) *** (5005ms)
  WITH the fix (this PR):
  skipResponseCloning=true  items=1640 -> PASS  errorCount=820 successCount=820 (4ms)

Existing test fixture, 4 items (passes either way -- why a large body is needed):
  skipResponseCloning=false items=4 -> PASS  errorCount=2 successCount=2 (3ms)
  skipResponseCloning=true  items=4 -> PASS  errorCount=2 successCount=2 (3ms)

Worth flagging for reviewers: without the fix this test times out rather than failing an
assertion, which is exactly how the bug manifests in production — a poll that hangs until the
caller's deadline expires. If the suite has a per-test timeout shorter than the hang, that is what
will trip.

Testing

• Reproduced the hang and verified the fix against the pinned fetch client (numbers above)
• Added a regression test, and verified it fails without the fix
• Ran the repo's full suite locally (sparse checkout without monorepo deps — needs CI)
• [Segmenters] Tested in the staging environment

Notes for reviewers

Targeting staging rather than main because index.async.ts and the asyncActions registration exist only on staging.

Two adjacent issues found while tracing, deliberately not fixed here to keep this reviewable:

1. agent.destroy() sits after the awaited clone.text() in prepare-response.ts, so every hung request also leaks its socket. Fixed as a side effect here, but the ordering is still wrong for any other caller that hangs.
2. /results is paginated — page, pageSize, and count are declared in AsyncUpsertRowsPollResultsResponse and never used, so only the first page is consumed. Separately, the error branch indexes MultiStatusResponse by position within items while the success branch indexes by position within the original batch; those index spaces do not correspond.

[rokatyal]

Deep code review — 3 independent passes

Ran three independent bug-hunting passes over this diff. Two findings were significant enough to fix in 2b917e94; the rest are flagged below for reviewers.

Fixed in this PR

1. The original fix was incomplete — /status had the identical deadlock (found by all 3 passes)

performPoll makes two requests, and only /results got skipResponseCloning. But /status carries resultMessages: AsyncUpsertRowsPollResultMessage[] — an unbounded array of 4-string objects. Roughly 100–160 messages clears the same 16KB tee threshold, and /status is the call that runs first on every poll cycle, so it would hang before /results was ever reached, with an identical production symptom.

This was only visible from the type declaration — in the incident logs /status was always ~211 bytes because resultMessages happened to be empty. Reasoning from observed traffic alone missed it.

2. Missing/empty items — silent data-integrity bug (found by pass 2)

response.jobStatus = 'SUCCEEDED' is assigned before the /results fetch, and resultsResponse.data.items.length was dereferenced unguarded. Two bad outcomes:

• Missing items (malformed JSON — note prepare-response.ts swallows JSON.parse failures and leaves data undefined — or a non-JSON content-type leaving data as a string): TypeError thrown inside the try, caught at the bottom, reported as a bare FAILED/400 with no error message.
• Empty items: []: the loop never runs, multiStatusResponse stays empty, and a batch SFMC explicitly flagged resultStatus: 'Has Errors' is reported SUCCEEDED with zero records accounted for — every event silently dropped from the delivery report.

Both now return RETRYABLE_ERROR. Verified across four shapes: undefined data, data without items, items: [], and a non-JSON body; the normal path is unchanged.

Flagged, deliberately not fixed here

#	Finding	Why not here
3	asyncUpsertRowsV2 (sfmc-operations.ts:88) has the same exposure on the performBatch path	Shared by other callers; needs its own audit
4	TimeoutError isn't an HTTPError, so it maps to permanent FAILED rather than retryable	This fix makes the path more reachable — an unblocked large read can now time out instead of hanging. Deserves its own change
5	agent.destroy() runs after the awaited body read in prepare-response.ts, so a rejecting read still leaks the agent	Core middleware, not SFMC. This PR does improve it — previously the hang made destroy() unreachable entirely
6	/results pagination: page/pageSize/count declared but never read; only page 1 is consumed	Needs SFMC API semantics confirmed first

One unresolved disagreement worth a maintainer's eye: pass 1 flagged an index-space mismatch between the success path (indexes by payload.uploadCount) and the error path (indexes by position in items), calling it live per-record misattribution. Pass 3 checked the code and found both use the loop index i, so the current code is self-consistent. But pass 1's underlying concern stands and neither pass could settle it from this checkout: if /results returns only failed rows, positional indexing attributes errors to the wrong records. That needs SFMC's documented semantics (failures-only vs all-records, and the server-side pageSize cap) — items 6 and this are the same question.

CI status

The regression test passes in CI:

PASS src/destinations/salesforce-marketing-cloud/_tests_/asyncDataExtension.async.test.ts

All SFMC suites green. Two job failures on that run appear unrelated to this diff:

• Unit tests — sole failure is google-enhanced-conversions/__tests__/functions.test.ts › .getConversionActionId › should return error message and code if dynamic fetch fails. This branch touches only the two SFMC files (git diff --name-only confirms).
• Validate — "Assert yarn.lock is up-to-date". No commit on this branch touches yarn.lock.

Both look pre-existing on staging, but I'd appreciate confirmation from someone with more context — staging's recent runs show skipped, with the last success on 2026-08-14, so I couldn't establish a clean baseline myself.

Note that run was on 28bbd301, which predates the /status fix and the items guard. CI hasn't yet exercised 2b917e94.