[improve][broker]Increase the number of delayed messages sent whose delay time exceeds the TTL statistical metric

[zjxxzjwang]

Motivation

Currently, Pulsar Broker lacks a metric to track instances where delayed messages exceed their TTL. This results in messages set with delayed delivery times exceeding the TTL expiring before being consumed by users, with no mechanism to detect this occurrence. Consequently, there is a significant risk of message loss.

Modifications

1. Add a new member variable Rate exceedTTLDelayMessage = new Rate() to PersistentTopic to track the number of delayed messages exceeding the TTL.
2. Within the message sending logic of handleSend, add a check specifically for delayed messages whose delay time exceeds the TTL. If exceeded, call exceedTTLDelayMessage.recordEvent().
3. In TopicStatsImpl, add the exceedTTLDelayMessages variable to retrieve and display the count of delayed messages exceeding the TTL.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[Denovo1998]

Left some comments.

[zjxxzjwang]

@Denovo1998 Hello, could you please help me review again

[zjxxzjwang]

@Denovo1998 Hello, could you please help me review again

[Denovo1998]

There is no guarantee that messageTTLInSeconds > 0.

I suggest wrapping the counting logic in

if (messageTTLInSeconds != null && messageTTLInSeconds > 0)

[lhotari]

exceedTTLDelayedMessage doesn't read well. It seems that ttlExceededDelayedMessagesRate would be more accurate for this field.

In addition, it would be necessary to add a comment to this field to explain it.

[lhotari]

similar comment about exceedTTLDelayedMessages as in the previous comment. It doesn't read well. I think that ttlExceededDelayedMessages would be better. In the comment, it should be made explicit that this is the number of delayed messsages that exceeded TTL at publish time.

[lhotari]

Please check the review comments.
In addition, please revisit the PR title. It's currently [improve][broker]Increase the number of delayed messages sent whose delay time exceeds the TTL statistical metric which is not very clear. Something like [improve][broker] Add counter to topic stats for delayed messages which exceed TTL at publish time would be more accurate.