Skip to content

Backport(v1.19): out_forward: drop keepalive sockets with failed or mismatched acks (#5436)#5445

Open
github-actions[bot] wants to merge 1 commit into
v1.19from
backport-to-v1.19/pr5436
Open

Backport(v1.19): out_forward: drop keepalive sockets with failed or mismatched acks (#5436)#5445
github-actions[bot] wants to merge 1 commit into
v1.19from
backport-to-v1.19/pr5436

Conversation

@github-actions

Copy link
Copy Markdown

Which issue(s) this PR fixes:
Fixes #4057

What this PR does / why we need it:

In Forwarder-Aggregator mode with keepalive true and flush_thread_count > 1, when the aggregator restarts/reconnects the forwarder starts logging

[warn]: ack in response and chunk id in sent data are different chunk_id="..." ack="...\n"

endlessly, and never recovers on its own until the pod is killed.

Root cause: when a chunk's ack times out (FAILED) or comes back with a chunk id that does not match the one tracked for its socket (CHUNKID_UNMATCHED), ack_check returned the keepalive socket to the reuse pool via ConnectionManager#close (→ SocketCache#checkin). Such a connection may still carry an in-flight/stale ack for a previous chunk. When the socket is checked out again for a new chunk, that stale ack is read first and compared against the new chunk's id, so it never matches. From then on every ack on that connection is offset by one chunk, producing an unbounded stream of mismatch warnings.

This PR adds ConnectionManager#discard, which revokes a keepalive socket (or closes a non-keepalive one) instead of checking it back in, and uses it from ack_check on the FAILED and CHUNKID_UNMATCHED paths. The tainted connection is torn down, so the next chunk uses a fresh connection with a clean ack stream and the warning storm stops. Successful acks keep reusing the socket exactly as before.

Docs Changes:
None.

Release Note:
out_forward: stop the endless "ack in response and chunk id in sent data are different" warning storm by discarding (instead of reusing) a keepalive socket whose ack failed or came back with a mismatched chunk id.


Tests (ruby 3.4.10):

  • test/plugin/out_forward/test_connection_manager.rb — new #discard cases: closes a non-keepalive socket, revokes a keepalive socket instead of checking it in, and confirms a discarded keepalive socket is not reused (a checked-in one is).
  • test/plugin/test_out_forward.rb — new ack_check cases: the socket is discarded on CHUNKID_UNMATCHED and FAILED, and reused on SUCCESS.
$ bundle exec ruby -Itest test/plugin/out_forward/test_connection_manager.rb
10 tests, 16 assertions, 0 failures, 0 errors

$ bundle exec ruby -Itest test/plugin/test_out_forward.rb
74 tests, 147 assertions, 0 failures, 0 errors, 3 omissions (Windows-only)

The new tests fail on master (the mismatched/failed ack socket is reused) and pass with this change.

…5436)

**Which issue(s) this PR fixes**:
Fixes #4057

**What this PR does / why we need it**:

In Forwarder-Aggregator mode with `keepalive true` and
`flush_thread_count` > 1, when the aggregator restarts/reconnects the
forwarder starts logging

```
[warn]: ack in response and chunk id in sent data are different chunk_id="..." ack="...\n"
```

endlessly, and never recovers on its own until the pod is killed.

Root cause: when a chunk's ack times out (`FAILED`) or comes back with a
chunk id that does not match the one tracked for its socket
(`CHUNKID_UNMATCHED`), `ack_check` returned the keepalive socket to the
reuse pool via `ConnectionManager#close` (→ `SocketCache#checkin`). Such
a connection may still carry an in-flight/stale ack for a previous
chunk. When the socket is checked out again for a new chunk, that stale
ack is read first and compared against the new chunk's id, so it never
matches. From then on every ack on that connection is offset by one
chunk, producing an unbounded stream of mismatch warnings.

This PR adds `ConnectionManager#discard`, which **revokes** a keepalive
socket (or closes a non-keepalive one) instead of checking it back in,
and uses it from `ack_check` on the `FAILED` and `CHUNKID_UNMATCHED`
paths. The tainted connection is torn down, so the next chunk uses a
fresh connection with a clean ack stream and the warning storm stops.
Successful acks keep reusing the socket exactly as before.

**Docs Changes**:
None.

**Release Note**:
out_forward: stop the endless "ack in response and chunk id in sent data
are different" warning storm by discarding (instead of reusing) a
keepalive socket whose ack failed or came back with a mismatched chunk
id.

---

Tests (`ruby 3.4.10`):

- `test/plugin/out_forward/test_connection_manager.rb` — new `#discard`
cases: closes a non-keepalive socket, revokes a keepalive socket instead
of checking it in, and confirms a discarded keepalive socket is not
reused (a checked-in one is).
- `test/plugin/test_out_forward.rb` — new `ack_check` cases: the socket
is discarded on `CHUNKID_UNMATCHED` and `FAILED`, and reused on
`SUCCESS`.

```
$ bundle exec ruby -Itest test/plugin/out_forward/test_connection_manager.rb
10 tests, 16 assertions, 0 failures, 0 errors

$ bundle exec ruby -Itest test/plugin/test_out_forward.rb
74 tests, 147 assertions, 0 failures, 0 errors, 3 omissions (Windows-only)
```

The new tests fail on `master` (the mismatched/failed ack socket is
reused) and pass with this change.

Signed-off-by: JSup <hanjisang0914@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@Watson1978 Watson1978 added this to the v1.19.4 milestone Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants