[simpleanalytics-agent]

Summary

Fixes #151

pnpm run traffic-drop analyze flagged ezgif.com after the v12 deploy, but ezgif.com had removed the script hours earlier. The check now also looks at the hour before the deploy, compared with the same hour in the earlier weeks, and says when each drop started:

Label	Meaning
after the deploy	normal in the hour before
worse after the deploy	already down before, but clearly worse after it (binomial test of the after share)
unclear, little data before	too few pageviews in the hour before to tell
already down	down before and not clearly worse; listed separately, because the deploy didn't cause it

Follow-up requests are printed for the first three labels. The hour before the deploy is used instead of the full length of the comparison window, so a script removed hours earlier isn't diluted in a long window.

These commits were pushed to #144 after it was merged, so they're opened again here.

Security implications

• No security impact
• Has security impact - described as:

Testing

• Synthetic data in Elasticsearch 8.15 (3,000 websites, a day before and after the deploy, 5 weeks):
  • Dropped to zero after the deploy: all 15 flagged (11 "after the deploy", 4 "unclear").
  • Dropped to half: 14 of 15 flagged (9 "after the deploy", 5 "unclear").
  • Already down a day before: 7 of 15 labeled "already down" and 8 "unclear". None is blamed on the deploy.
  • False positives: none.
• Hand-made cases:
  • normal then zero → "after the deploy"
  • 650 of 1,000 then zero → "worse after the deploy"
  • 0 of 4 before → "unclear"
  • 18 of 1,000 before and 20 of 1,000 after → "already down"
• pnpm run typecheck passes.

GPT-6 Astra reviewed the first version. Its three findings are fixed:

• Further drops after an earlier dip are flagged.
• The before window is limited to an hour.
• Too little data before is reported as "unclear".

---

Open workspace in Conductor

Summary by CodeRabbit

• New Features
  • Traffic analysis now compares post-deploy activity with matching periods from prior weeks and evaluates a separate period immediately before the deploy.
  • Reports include pre-deploy traffic counts and classify drops as occurring after deployment, already present beforehand, worse after deployment, or unclear.
  • Sites with low traffic before deployment that did not clearly worsen are reported separately.

[coderabbitai]

Walkthrough

The traffic-drop analysis now checks traffic before a deploy as well as after it. It compares each period with matching prior-week windows and reports whether a drop occurred after the deploy, was already present, worsened after the deploy, or is unclear.

Changes: Deploy-time traffic drop analysis

Layer / File(s)	Summary
Build deploy-aligned query windows traffic-drop.mts	The query adds a pre-deploy window of up to one hour and matching weekly windows for both periods. Post-deploy baselines use only the matching post-deploy windows.
Compare and report traffic drops traffic-drop.mts	The analysis compares each period with its own weekly baseline, classifies qualifying drops, and reports already-down sites separately from flagged sites. The report includes pre-deploy counts and timing classifications.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related issues

• simpleanalytics/scripts#143 — This change updates per-site post-deploy drop analysis, but does not add the full alerting or coverage described in the issue.

Possibly related PRs

• simpleanalytics/scripts#144 — This PR introduced traffic-drop.mts, which this change modifies.
• simpleanalytics/scripts#148 — This PR changes deployment records that traffic-drop reads to determine deploy time.
• simpleanalytics/scripts#140 — This PR adds the deployment records used by traffic-drop.

Suggested reviewers: adriaandotcom

Merge Risk: Low · up to 89c79

A site already losing traffic could be labelled “worse after the deploy” for a small additional change. The practical-size threshold is not established as a requirement, so this is a bounded attribution concern rather than a confirmed merge blocker.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title "Only blame a deploy for drops that started after it" matches the main change. The PR changes traffic-drop analyze so it compares the hour before a deploy with earlier weeks, reports when a …
Linked Issues check		The change satisfies issue #151. windows() adds the hour before the deploy and matching weekly windows. compare() evaluates pre-deploy and post-deploy traffic against weekly baselines. when() re…
Out of Scope Changes check		The diff changes only traffic-drop.mts. The added windows, classification, output, and follow-up filtering directly support issue #151. No unrelated product or repository changes are present.

Full details: Title check

Explanation

The title "Only blame a deploy for drops that started after it" matches the main change. The PR changes traffic-drop analyze so it compares the hour before a deploy with earlier weeks, reports when a drop started, and labels drops that began before the deploy as already down or not caused by it. The title is specific, concise, and states the primary behavior change. It does not cover the supporting details (pre-deploy counts, timing classifications, follow-up output), which is acceptable for a title.

Full details: Linked Issues check

Explanation

The change satisfies issue #151. windows() adds the hour before the deploy and matching weekly windows. compare() evaluates pre-deploy and post-deploy traffic against weekly baselines. when() reports after the deploy, worse after the deploy, unclear, little data before, or already down. Follow-up requests use only the first three categories. The query also retains complete window counts and reports the timing classification.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Usage-based review receipt

• Mode: Continue automatically
• Reviewed files: 1
• Waived: $0.25 (charged $0.00)
• View usage details

Note

This review exceeded your plan’s limits and used usage-based reviews—free during trial, billed after paid activation unless disabled. Manage usage-based reviews.

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

SDLC label: change: routine because The change only adjusts an internal traffic-drop.mts analysis tool so it also checks the hour before a deploy and labels when a drop started, using the same aggregate per-hostname pageview counts that the tool already reads and making no change to data collection, access, storage, or the deployed tracking script.

[coderabbitai]

🔇 Additional comments: 1

traffic-drop.mts-388-390: 1

388-390: Minor · Functional Correctness · Quick win

⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Require a practical-size worsening, not only statistical significance, before reporting "worse after the deploy".

The test at Line 388 assumes that after.baseline and before.baseline are exact. It ignores the error in each median-of-weeks baseline, and week-to-week traffic often varies more than Poisson noise. The after window is also much longer than the before hour, so its count is large. For large counts, the binomial spread is small. A tiny deviation from the baseline ratio can then give z <= -minZ.

Example, with large-site numbers:

• after.baseline = 24000 and before.baseline = 1000.
• before.count = 400, a 60% drop.
• after.count = 8800, a 63% drop.
• This gives z ≈ -11.8, so the site is labelled "worse after the deploy" for a 3-point difference.

A site that was already down is then flagged and blamed on the deploy. This is the false attribution that issue #151 aims to remove.

Add a minimum relative worsening that reuses minDrop. For example, require after.count to be at least minDrop below the expected share of the combined count. Use a threshold that is inclusive and matches the >= minDrop check in compare.

Proposed fix

       const total = s.after.count + s.before.count;
       const share = s.after.baseline / (s.after.baseline + s.before.baseline);
       const spread = Math.sqrt(total * share * (1 - share));
-      return spread > 0 && (s.after.count - total * share) / spread <= -minZ
+      const expected = total * share;
+      return spread > 0 &&
+        (s.after.count - expected) / spread <= -minZ &&
+        1 - s.after.count / expected >= minDrop
         ? "worse after the deploy"
         : "already down";

Check the threshold against the hand-made "already underway" cases from the PR description.

---

Review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: QUIET
• Plan: Advanced
• Run ID: 8663ef4d-2258-4240-b455-372c2693590f

Commits

Reviewing files that changed from the base of the PR and between 3b73122 and 89c7995.

Files selected for processing: 1

• traffic-drop.mts

Limit details: You’ve used all 10 included reviews currently available.