What would you like to happen?
Currently in KafkaCommitOffset.CommitOffsetDoFn there is no visibility into how often offset commits are failing or how many retries are being attempted. If commits are silently failing in production you have no way to know without digging through logs.
Adding Beam metrics here would make it easy to monitor commit health through existing monitoring dashboards.
Proposed approach
Add a counter for total commit failures
Add a counter for retry attempts
Add a counter for commits that exhausted all retries
Use Beam's built in Metrics.counter() so it works with any runner
Why this is useful
If you're running a high throughput pipeline and offset commits are struggling, right now you'd have no idea until you see duplicate processing. This gives you an early warning signal.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
What would you like to happen?
Currently in
KafkaCommitOffset.CommitOffsetDoFnthere is no visibility into how often offset commits are failing or how many retries are being attempted. If commits are silently failing in production you have no way to know without digging through logs.Adding Beam metrics here would make it easy to monitor commit health through existing monitoring dashboards.
Proposed approach
Add a counter for total commit failures
Add a counter for retry attempts
Add a counter for commits that exhausted all retries
Use Beam's built in
Metrics.counter()so it works with any runnerWhy this is useful
If you're running a high throughput pipeline and offset commits are struggling, right now you'd have no idea until you see duplicate processing. This gives you an early warning signal.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components