fix: Restart DSX exchange consumer when MQTT stays disconnected#1825
fix: Restart DSX exchange consumer when MQTT stays disconnected#1825kfelternv wants to merge 3 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
kfelternv
force-pushed
the
fix-dsx-consumer-wedge-watchdog
branch
from
aefa86b to
3d10a79
Compare
kfelternv
force-pushed
the
fix-dsx-consumer-wedge-watchdog
branch
from
3d10a79 to
766d39c
Compare
|
How does that help compared to reconnecting within the app? It sounds like the only thing this does is lead to a crashlooping service in the case of networking events which then leads to other issues (e.g. no metrics anymore). cc @FrankSpitulski for advise |
|
@Matthias247 This only happens if the entire message bus is down, which would be a much bigger problem. So I think its a good way to solve the problem to try to reconnect and if it can't reconnect after 30s then very obviously fail by crashing. |
|
I still don't understand how this is better. We lose metrics that could tell us that the connection is down and we confuse operators by getting them guessing why something is crashing. Just make sure the right metrics exist and an alarm is defined for them. |
| # How long the MQTT event loop is allowed to stay disconnected from the | ||
| # broker before the service exits, relying on Kubernetes to restart the | ||
| # pod with a fresh MQTT session (default: 30s). Recovery path for the | ||
| # consumer wedge where TCP reconnects but the MQTT subscription is |
There was a problem hiding this comment.
there are no MQTT subscriptions across connections. subscriptions are renewed on connection
3 participants
When the mqtt client can't connect to the message bus after 30s it exits the consumer task and relies on the orchestration layer ( like kubernetes ) to restart the service, this will cause the consumer to attempt to reconnect to the message bus and keep restarting until connection is established.