fix(deps): update go modules (non-major)#36
Open
renovate[bot] wants to merge 1 commit intomainfrom
Open
Conversation
1fc5118 to
1daacdf
Compare
b0f229e to
ba1f692
Compare
Author
ℹ️ Artifact update noticeFile name: go.modIn order to perform the update(s) described in the table above, Renovate ran the
Details:
|
ba1f692 to
46c0ede
Compare
798a8e0 to
30de259
Compare
30de259 to
6e5fc27
Compare
c811adf to
33fbe26
Compare
fc795a0 to
27e404f
Compare
41f046f to
b6991ea
Compare
29993dd to
ced979d
Compare
ced979d to
13067db
Compare
13067db to
e3d8932
Compare
e3d8932 to
8d49fc8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
v0.19.2→v0.24.0v2.4.0→v2.5.0v1.2.0→v1.2.1v2.3.0→v2.3.4v1.20.6→v1.21.0v1.17.1→v1.18.0v1.12.0→v1.13.1v1.27.1→v1.28.0v0.19.0→v0.20.0Release Notes
adobe/ims-go (github.com/adobe/ims-go)
v0.24.0Compare Source
v0.23.0Compare Source
v0.22.0Compare Source
v0.21.0Compare Source
v0.20.0Compare Source
go-viper/mapstructure (github.com/go-viper/mapstructure/v2)
v2.5.0Compare Source
What's Changed
New Contributors
Full Changelog: go-viper/mapstructure@v2.4.0...v2.5.0
knadh/koanf (github.com/knadh/koanf/providers/file)
v1.2.1Compare Source
changelog for v1.2.0 -> v1.2.1
29cce50Merge pull request #101 from e-nikolov/fix-pflag-map-types0202243posflag: add support for pflag map typestwmb/franz-go (github.com/twmb/franz-go)
v1.21.0Compare Source
===
This is a relatively "major" minor release. It adds support for Kafka 4.2,
adds full support for KIP-932 share groups, adds
KIP-881 rack-aware partition assignment, adds a handful of other
features / options, and fixes several niche bugs.
The companion
kfakepackage has also been significantly extended; it nowsupports everything except delegation tokens, streams APIs, and broker
internal APIs. kfake can be used as a dumb localhost broker; it has an option
to persist to disk to tolerate restarts (and it even handles quick restarts
without interrupting any client state). See the run_tests.sh script and the
main.go file in pkg/kfake if you want to see about bootstrapping this yourself.
I may create some tiny 'dumbkafka' binary that supports running on localhost
with a few options. Regardless, kfake is quite neat.
The
kadmpackage has been extended with new share APIs. See theincoming
kadmtag for full details.As a meta note, this was a significant time investment (>4w most evenings and
weekends for KIP-932 alone). I hope future releases require less work; 932 is
the last major feature this library has been missing for a while, and of
upcoming KIPs, only transactional support for 932 looks to maybe be some
effort. That said, if you get a lot of value from this and have a spare
quarter, please consider sponsoring.
API additions
Share groups (KIP-932)
franz-go now fully supports KIP-932 share groups for consuming.
Share groups are the "queue-like" alternative to consumer groups: many
consumers can share a single partition, records are individually
acknowledged, and unacknowledged records are automatically redelivered.
The new share group API mirrors the existing consumer group shape; see
full documentation on pkg.go.dev:
Two new
RecordFormatterverbs were added alongside this:%D- share group delivery count%A- share group acquisition deadline (timestamp; supports the samestrftime/Go formatting as
%d)Rack-aware partition assignment (KIP-881)
Both the range and sticky balancers now understand consumer racks. If you
set
kgo.Rack, the leader will preferentially assign you partitions whoseleader is in the same rack, preserving the existing priority of balance
over locality over stickiness. I was originally not planning to support this
in franz-go since the next generation rebalancer was releasing at a similar
time, but I suspect it's worth it to keep the client-driven rebalancing for
a while.
Custom balancers that use the consumer protocol can use the new
(*ConsumerBalancer).PartitionRacks()method to access the computedpartition-rack map.
Other new kgo APIs
AllowIdempotentProduceCancellationpermits cancellation of in-flightidempotent records, at the cost of breaking idempotency's duplicate
guarantee. When a record is in-flight, the client cannot tell "never
written" from "written but reply lost". Cancelling leaves the client's
sequence window inconsistent with the broker: the next produce either
silently gap-accepts (broker wrote the cancelled records) or hits
OUT_OF_ORDER_SEQUENCEand forces a producer ID reload (broker didnot). Any application-level retry of a cancelled record that the
broker actually stored will duplicate on the broker - idempotent
dedupe cannot help because the window has reset. By default the
client refuses to cancel in this state and waits for the record's
outcome. Use this when time-bounded delivery matters more than
duplicate-avoidance. Incompatible with a transactional id.
ProducerBatchMaxBytesFntakes a topic name and returns the max batchsize for that topic, following the
RetryTimeout/RetryTimeoutFnpattern. Useful when you produce to multiple topics with different
broker-side
max.message.bytes.AlwaysRetryEOFkeeps retrying EOF errors indefinitely for users whoseinfrastructure considers EOF always transient. This option is actually
generally recommended for all users, BUT you really have to ensure your
SASL and TLS is setup correctly when using this option. Invalid SASL or
TLS is only visible as an EOF error, so the client by default uses
heuristics to hard fail requests without retrying if the first write triggers
an EOF. This has bit some users over time due to an EOF ALSO being seen
during restarts; this new option allows you to say "trust me, I know
my configuration is correct: keep retrying".
HookPollStartfires at the start of eachPollFetches/PollRecordscall. Thanks @rarguellof91!Misc additions
kadmgained aRequireStableoption for offset fetching.kfake.VirtualNetworkplus a newkgoxsync package enable fulltesting/synctestsupport against an in-process kfake cluster. Seeexamples/testing_with_kfake_and_synctest. Thanks@cupcicm!
Behavior changes
ApiVersionsis now sent on every new broker connection, not justthe first one per broker. The broker uses the ApiVersions request's
ClientSoftwareName/ClientSoftwareVersionfields to scope KIP-714metric subscriptions; caching and skipping ApiVersions meant later
connections (fetch, group, etc.) registered as
software=unknown, andone
kgo.Clientappeared as two software entities to the broker.Versions are still cached for request-version selection, so this is
only a one-extra-request-per-connection cost.
Group consumer
OffsetFetchnow hardcodesRequireStable=true. Theprior behavior could return stale committed offsets during a
rebalance, and in the worst case, result in duplicate messages.
You no longer need to use the
RequireStableFetchOffsetsoption.ErrRecordTimeoutnow wraps the last retry error seen while waitingon metadata in
waitUnknownTopic.errors.Iscontinues to work forErrRecordTimeout; you can now alsoerrors.Isthe underlying cause(e.g. SASL auth errors that previously disappeared).
BlockRebalanceOnPollno longer gates the assign-side callback if theuser did not register
OnPartitionsAssigned. Assign only addspartitions, so a user's in-flight commit cannot reference anything
they don't still own; the gate only existed to serialize user
callbacks with poll.
MessageTooLargeerrors now include the uncompressed and compressedmessage sizes. Thanks @anubhav21sharma!
Bug fixes
Fixed a KIP-951 bug where
ensureBrokerscould destroy unrelatedbroker objects when the broker set changed. This would manifest to you
as "the broker chosen for the request is dead" (or does not exist).
listOffsetshad two related bugs: a nil cursor panic in the drainpath, and a race where a metadata signal could be lost. These were
possible to encounter on extremely fast setups (the bugs were years
old, and I only encountered via localhost kfake testing).
The KIP-848 heartbeat path had several bugs that were shaken out by
long spin loops against the new kfake share/848 support (that said,
I am still not opting into KIP-848 by default, and will only allow
an option-based opt-in once KIP-1251 is released in Kafka 4.3).
fetchOffsetsnow validatesOffsetFetchresponses against what wasrequested (brokers have been seen to omit partitions) and correctly
handles the group-level error code.
Fixed a TOCTOU race between
producerIDandcreateReqthat couldconstruct a produce request with a stale producer ID.
failDialnow actually clears the stale coordinator / controllercache, so a single bad dial does not poison subsequent discovery.
Various smaller fixes: retrying when
broker.gowrote 0 bytes (treatas "didn't try"), transient dial errors now retry for the full
configured retry budget rather than the hard-coded ~1.5s cap (which
previously caused failures across 5-30s rolling restarts).
Improvements
batchPromisesis now backpressured: the ring is a dynamically-sizedcircular buffer that can block pushes at
max(maxBufferedRecords, 8192). Previously it could grow unboundedly and OOM the client whenrecords failed faster than
finishPromisescould process them.Two rounds of allocation reductions in hot paths: the produce blocking
path no longer forces its closure onto the heap,
recBuf's lingertimer is reused,
brokerCxn's 4-byte read buffer is reused,headerless records no longer allocate an empty header slice, and some
deprecated APIs were swapped out.
InitProducerIDfailures caused by transient broker errors (dialrefused, EOF across a broker restart, etc.) no longer surface as a
fatal "unrecoverable producer ID" error. The client now marks the
producer ID for reload on these errors so the next produce or begin
re-runs
InitProducerIDagainst the (probably now-available) broker.Relevant commits
8854973cimprovement kgo: recover producer ID from transient broker errors in maybeRecoverProducerIDc9a91d11feature kgo: add xsync package + kfake: add VirtualNetwork for synctest (thanks @manuc-conf!)a5a7c6f2kgo: send ApiVersions on every new broker connection14346ddcfeature kgo: add rack-aware partition assignment (KIP-881)40bf3e52feature kgo: add HookPollStart hook (thanks @rarguellof91!)68c85147behavior change kgo: skip BlockRebalanceOnPoll gate on assign with no OnPartitionsAssigned6d9a0188feature kgo: add %D (delivery count) and %A (acquisition deadline) to RecordFormatter0f0bca22feature kgo: add support for share groups (KIP-932)8762d567kgo: retry if we failed at 0 bytes written3e2f61a8kgo: retry transient dial errors for the configured retry budget857ed6dcbehavior change kgo: wrap last retry error into ErrRecordTimeoutd922b883kgo: fix failDial to actually clear stale coordinator/controller cacheaf5abf66bugfix kgo: fix KIP-951 ensureBrokers destroying unrelated broker objects6c7aab54bugfix kgo: fix listOrEpoch drain and nil cursor panic in listOffsets416e8269kgo: do not cancel prior in-flight offset commits2f666797kgo: simplify STALE_MEMBER_EPOCH retry in offset commit9dde6ad8behavior change kgo: hardcode RequireStable for group consumer OffsetFetch8a9400dfbugfix kgo: fix listOrEpoch race where metadata signal is lost0338467dfeature kgo: add AllowIdempotentProduceCancellation optionbc46151afeature kgo: add ProducerBatchMaxBytesFn for per-topic batch size limitsde2dff52bugfix kgo: handle group-level error code in fetchOffsets45ec9cfbbugfix kgo: fix TOCTOU race between producerID and createReq63cf8fa6kgo: add backpressure to batchPromises ring buffer9f15841bfeature kgo: add AlwaysRetryEOF option1de163c3include uncompressed/compressed message size in MessageTooLarge error (thanks @anubhav21sharma!)v1.20.7Compare Source
===
This patch release fixes numerous niche bugs - some user reported, some found
while investigating other things - contains a few behavior improvements,
extensive kfake additions, and many test additions / improvements.
There have been extensive additions to kfake over the course of the past
month; kfake now supports transactions, the next generation consumer group
protocol, and more Kafka APIs. franz-go integration tests now run and pass
against kfake in CI.
Testing has further been extensively improved: integration tests now run
against the latest patch version of Kafka for all major Kafka versions going
back to 0.11.0. The existing test suite has been extended to run while opting
into the next generation consumer group. An integration test against Kerberos
has been added. For ~roughly the past year (maybe half year), all bugs found
have had regression tests added in kfake. This release massively extends the
kfake test suite -- both with behavior tests that were ported via Claude
directly to franz-go (with license attribution!) and with behavior tests that
Claude generated specifically for kfake.
The "next generation" (KIP-848) consumer group code in franz-go itself has some
improvements. These improvements were found after adding 848 code to kfake,
which allowed for much faster integration test looping. This looping was still
very slow for what it's worth; towards the end, integration tests would pass
~40+ times with race mode over the course of two hours before failing once.
Claude was instrumental with adding appropriate log lines and tracing logs for
diagnosing extremely niche failures; things also got slower when a few specific
log lines that would've helped weren't added the first time...
Anyway,
Bug fixes
Returns from PollRecords / PollFetches that contained ONLY an error (context
cancellation or something) previously did not block rebalances, even if you
opted into BlockRebalanceOnPoll.
If, while idempotently producing, the client encountered
TIMED_OUTwhileproducing (retryable), the client considered this a "we definitively did not
produce" state, and allowed you to cancel the records. Well, maybe the records
actually did get produced broker side eventually, and now you re-produce new
records - the NEW records could be "deduplicated" due to how idempotency works.
This one is a bit niche, if you're interested, you should read #1217 and the
two PRs that address it.
My original implementation of how Kerberos handled authentication was
correct... for the time. I missed how it should have been touched up years
ago and now 4.0 hard deprecates the old auth flow. So, that's been found,
reported, and now fixed.
If a partition returned a retryable error in a metadata request (odd behavior
already) the first time the client is discovering the partition (i.e. on
startup), the client would not retry loading the partition right away (it
would, but much later on standard metadata refresh).
There was a very subtle, basically un-encounterable data race while
consuming. That was fixed in
5caaa1e0. It's so niche it's not worthwriting more about here.
RequestCachedMetadata, broken for a few releases, has been fixed. I plan toswitch kadm back to using it in the next kadm release.
There was a data race on client shutdown while leaving the group if you canceled
the client context.
Improvements
ConnIdleTimeoutis now obeyed more exactly, allowing you to more reliably reasonabout the maximum idle time. Thanks @carsonip!
At the end of a group transact session, I force a heartbeat to kinda "force
detect" the group is still alive (with some timing bounds). If the heartbeat
detected any error - including
REBALANCE_IN_PROGRESS- the session wouldabort no matter what. This has been changed to actually allow a commit (if
that's what you're doing) in certain scenarios (notably: the client detects
KIP-447 support on the broker and you are using RequireStable).
SetOffsets now does not let you set offsets for partitions that are not being
consumed. Previously, you could, but it'd just create entries in a map that were
never used. Those entries are no longer created.
ProduceSync now automatically un-lingers any partition that is produced to,
causing more immediate flushes. This should resolve lag that was introduced
from v1.20.0 where linger was set to 10ms by default. There are many usages of
franz-go I've seen in the wild where ProduceSync is used in random places to
produce one message before going back to other things.
764eb29dimprovement kgo: unlinger partitions in ProduceSync to avoid linger delayd996bf71bugfix kgo: fix data race during client shutdown with canceled context85b2c855behavior change kgo: fix applySetOffsets to skip partitions not being consumed05de202bimprovement kgo: allow commit despite rebalance with KIP-447 and RequireStablee8a00cd6improvement kgo: do not reuse idle connection after connIdleTimeout (thanks @carsonip!)787ab9febugfix kgo: fix and enhance RequestCachedMetadata5caaa1e0bugfix kgo: fix data race in allowUsable reading c.source after Swap40c144d3bugfix kgo: check loadErr for new partitions on first metadata loade430ea93bugfix kgo: send SaslHandshake for GSSAPI mechanism702c4f75bugfix kgo: error returns from polling should block rebalance32e1580a0c086b05bugfix kgo: poison batches to prevent future cancelation on two errorsuber-go/zap (go.uber.org/zap)
v1.28.0Compare Source
Enhancements:
zapcore.CheckPreWriteHookandCheckedEntry.Beforemethod for transforming entries before they are written to any Cores.Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.
This PR was generated by Mend Renovate. View the repository job log.