Skip to content

fix(agent): dedup detect-mode emits within a window - #248

Open
nghiadaulau wants to merge 1 commit into
VersusControl:mainfrom
nghiadaulau:fix/agent-emit-dedup
Open

fix(agent): dedup detect-mode emits within a window#248
nghiadaulau wants to merge 1 commit into
VersusControl:mainfrom
nghiadaulau:fix/agent-emit-dedup

Conversation

@nghiadaulau

Copy link
Copy Markdown
Member

Problem

In detect mode a sustained anomaly re-clusters into the same pattern on
every poll, and the worker re-emitted an incident on every tick — even
the cache-hit path (which skips the AI call) still called send(). One
ongoing issue therefore produced a new incident + notification every
poll_interval, spamming channels and the incident store.

Fix

A DedupStore gates emission per (service, pattern) for a configurable
window (agent.emit_dedup_window, default 1h; "0" restores the old
one-per-tick behavior):

  • Redis SETNX EX makes the "first emit in the window" check atomic
    across replicas; an in-memory map is the fallback when Redis is
    unavailable, mirroring the existing CursorStore degrade-don't-crash
    pattern.
  • Both emit sites (cache-hit and fresh AI call) go through one emit()
    helper that applies the gate.
  • Suppressed emits record outcome="deduped" in the detect log (visible in
    the Detect audit view).
  • A failed send releases the window (Release) so the next tick
    retries rather than being suppressed for the whole window.

Scope

This is the windowed-dedup correctness fix only. Full incident lifecycle
(burst re-arm on frequency escalation, auto-resolve with hysteresis) is
intentionally out of scope here.

Changes

  • pkg/agent/dedup.goDedupStore (Allow / Release / lazy sweep).
  • pkg/agent/worker.goemit() gate; both send paths refactored through it.
  • Config triple-touch: AgentConfig.EmitDedupWindow + clone_config +
    config/config.yaml; Helm agent.emitDedupWindow (values + configmap);
    docs (configuration.md, ai-detect-mode.md).

Testing

  • dedup_test.go — allow/release, window expiry, "0" disables, default
    parsing.
  • worker_dedup_test.goemitDetect suppresses the repeat (deduped,
    emitter called once) and retries after a failed send.
  • gofmt, go vet, go build ./..., go test -race ./pkg/agent ./pkg/config
    green; helm template renders emit_dedup_window (omitted when empty).

Checklist

  • Tests for the new behavior (unit + worker path)
  • Config triple-touch + Helm mirrored
  • Backward compatible ("0" disables; default fixes the spam)
  • Redis-with-memory-fallback; single-replica safe
  • House conventions (fix: prefix); docs + CHANGELOG

A sustained anomaly re-clusters into the same pattern on every poll, and
the worker re-emitted (and re-notified) on each tick — even the cache-hit
path still called send(). The result was incident spam and wasted
notifications for one ongoing issue.

Add a DedupStore that gates emission per (service, pattern) for
agent.emit_dedup_window (default 1h; "0" disables). Redis SETNX EX makes
the "first in window" check atomic across replicas; an in-memory map is the
fallback when Redis is unavailable, mirroring CursorStore. Suppressed emits
record outcome="deduped" in the detect log; a failed send releases the
window so the next tick retries instead of being silently suppressed.

- pkg/agent/dedup.go — DedupStore (Allow / Release / lazy sweep).
- pkg/agent/worker.go — emit() gate wraps both send sites.
- config triple-touch (struct + clone_config + config.yaml) + Helm
  (agent.emitDedupWindow) + docs.

Tests: DedupStore allow/release/expiry/disable; worker emitDetect suppresses
the repeat and retries after a failed send.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant