fix(agent): dedup detect-mode emits within a window - #248
Open
nghiadaulau wants to merge 1 commit into
Open
Conversation
A sustained anomaly re-clusters into the same pattern on every poll, and the worker re-emitted (and re-notified) on each tick — even the cache-hit path still called send(). The result was incident spam and wasted notifications for one ongoing issue. Add a DedupStore that gates emission per (service, pattern) for agent.emit_dedup_window (default 1h; "0" disables). Redis SETNX EX makes the "first in window" check atomic across replicas; an in-memory map is the fallback when Redis is unavailable, mirroring CursorStore. Suppressed emits record outcome="deduped" in the detect log; a failed send releases the window so the next tick retries instead of being silently suppressed. - pkg/agent/dedup.go — DedupStore (Allow / Release / lazy sweep). - pkg/agent/worker.go — emit() gate wraps both send sites. - config triple-touch (struct + clone_config + config.yaml) + Helm (agent.emitDedupWindow) + docs. Tests: DedupStore allow/release/expiry/disable; worker emitDetect suppresses the repeat and retries after a failed send.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In detect mode a sustained anomaly re-clusters into the same pattern on
every poll, and the worker re-emitted an incident on every tick — even
the cache-hit path (which skips the AI call) still called
send(). Oneongoing issue therefore produced a new incident + notification every
poll_interval, spamming channels and the incident store.Fix
A
DedupStoregates emission per(service, pattern)for a configurablewindow (
agent.emit_dedup_window, default1h;"0"restores the oldone-per-tick behavior):
SETNX EXmakes the "first emit in the window" check atomicacross replicas; an in-memory map is the fallback when Redis is
unavailable, mirroring the existing
CursorStoredegrade-don't-crashpattern.
emit()helper that applies the gate.
outcome="deduped"in the detect log (visible inthe Detect audit view).
Release) so the next tickretries rather than being suppressed for the whole window.
Scope
This is the windowed-dedup correctness fix only. Full incident lifecycle
(burst re-arm on frequency escalation, auto-resolve with hysteresis) is
intentionally out of scope here.
Changes
pkg/agent/dedup.go—DedupStore(Allow/Release/ lazy sweep).pkg/agent/worker.go—emit()gate; both send paths refactored through it.AgentConfig.EmitDedupWindow+clone_config+config/config.yaml; Helmagent.emitDedupWindow(values + configmap);docs (
configuration.md,ai-detect-mode.md).Testing
dedup_test.go— allow/release, window expiry,"0"disables, defaultparsing.
worker_dedup_test.go—emitDetectsuppresses the repeat (deduped,emitter called once) and retries after a failed send.
gofmt,go vet,go build ./...,go test -race ./pkg/agent ./pkg/configgreen;
helm templaterendersemit_dedup_window(omitted when empty).Checklist
"0"disables; default fixes the spam)fix:prefix); docs + CHANGELOG