diff --git a/.github/skills/oncall-weekly-telemetry-report/SKILL.md b/.github/skills/oncall-weekly-telemetry-report/SKILL.md index 70f57e54..0336dac6 100644 --- a/.github/skills/oncall-weekly-telemetry-report/SKILL.md +++ b/.github/skills/oncall-weekly-telemetry-report/SKILL.md @@ -5,9 +5,9 @@ description: Generate the weekly Android Broker on-call (OCE) WoW + 60-day trend # OCE Weekly Report -Produce the weekly Android Broker on-call (OCE) telemetry report as a self-contained HTML file at `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the reporting-week Sunday — see "Inputs to confirm" §4). Writes to the user's home folder, **outside the workspace**, so reports never accidentally get committed. +Produce the weekly Android Broker on-call (OCE) telemetry report as a self-contained HTML file at `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the **end-date of the rolling 7-day window** — see "Inputs to confirm" §1). Writes to the user's home folder, **outside the workspace**, so reports never accidentally get committed. -The output mirrors the structure of the canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html). The Step 1 bootstrap script copies the template into `~/android-oce-reports/oncall-wow-report-.html` and you edit it in place from there. Do **not** redesign the layout each week. +The output mirrors the structure of the canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html). The Step 1 bootstrap script copies the template into `~/android-oce-reports/oncall-wow-report-.html`, **stamps the resolved rolling-7-day window into the title / meta line / Generated banner**, and you edit it in place from there. Do **not** redesign the layout each run. **Before writing any KQL, read [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md).** It captures the canonical view names, helper functions, the HLL device-count gotcha, week-alignment rules, and ready-to-paste query templates — distilled from the production Android Broker Dashboard. @@ -21,33 +21,43 @@ Reusable helpers in [`assets/`](assets/): | [`code-attribution-template.md`](assets/docs/code-attribution-template.md) | Per-card checklist for the deep code-attribution block (Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step) | | [`queries/`](assets/queries/) | Canonical KQL templates, one file per query — see [`queries/README.md`](assets/queries/README.md). Highlights: [`attr-union-by-dim.kql`](assets/queries/attr-union-by-dim.kql) (NEW — all 7 dims in one round-trip), [`error-message-and-location.kql`](assets/queries/error-message-and-location.kql) (now accepts BOTH `` and `` in one call) | | [`templates/`](assets/templates/) | Copy-paste HTML snippets (`spike-card.html`, `traffic-attr-card.html`, `sparkline-footer.html`) | -| [`bucket-trends.js`](assets/scripts/bucket-trends.js) | Bucket all error codes into 60-day regression / spike / improvement / flat. Run with `--metric=devs` AND `--metric=reqs`. Pass `--end=YYYY-MM-DD` (Sunday after the reporting week, exclusive) to drop the partial in-progress bucket. **`--summary` suppresses the verbose header; `--json=` emits a structured sidecar for programmatic consumption.** | +| [`bucket-trends.js`](assets/scripts/bucket-trends.js) | Bucket all error codes into 60-day regression / spike / improvement / flat. Run with `--metric=devs` AND `--metric=reqs`. Pass `--end=YYYY-MM-DD` (the Sunday that OPENS the current in-progress week, exclusive — i.e. `startofweek(today)`) to drop the partial in-progress bucket. **`--summary` suppresses the verbose header; `--json=` emits a structured sidecar for programmatic consumption.** | | [`agg.js`](assets/scripts/agg.js) | Per-error per-dim top-N rollup with WoW deltas. Workhorse for filling spike-attribution dim blocks. | | [`summarize-attribution.js`](assets/scripts/summarize-attribution.js) | Roll up 7-dim attribution slices for spike-attribution cards. Supports BOTH `--union ` (preferred for 2-week WoW; pairs with `attr-union-by-dim.kql`) AND legacy `--label= file.json` per-dim mode. **Auto-detects the array-form schema produced by `assets/scripts/run-kql.ps1` — no schema-transformer step needed.** | | [`find-suspect-prs.ps1`](assets/scripts/find-suspect-prs.ps1) | Parallel `git log -S` + `--grep` across broker/ + common/ for a class/method symbol, with PR numbers + URLs. Run *only after* the Originator pre-check has identified a specific throw-site class — the unscoped 4-week PR window is small enough (<30 PRs) to scan with plain `git log` first. | | [`validate-report.ps1`](assets/scripts/validate-report.ps1) | Pre-publish validator. Catches stale tokens, devs/reqs leaks, mojibake (U+FFFD), unbalanced `
` depth in Section 2 (the nested-callout bug), KPI/trend sparkline coverage, code-attribution depth, layout-guard CSS presence, and suspicious low-peak fabricated `data-trend` arrays. Run as part of Step 7. | | [`scripts/run-kql.ps1`](assets/scripts/run-kql.ps1) | **Direct-REST Kusto helper — drop-in fallback for the Azure Kusto MCP server when the MCP times out** (frequent on per-error-code queries). Acquires a token via `az`, POSTs to `/v2/rest/query`, writes a JSON file the JS helpers can consume directly. | -| [`scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) | Bootstrap a new week's report from the canonical template. Auto-computes the reporting Sunday, creates `_data//`, prunes `_data` folders older than 60 days, and detects "unfilled template stub" vs "real prior report" collisions using a multi-marker fingerprint (title + meta date + first KPI value + size ratio). | +| [`scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) | Bootstrap a new report from the canonical template. Auto-computes the rolling 7-day window (curEnd = today UTC; override with `-EndDate YYYY-MM-DD`), creates `_data//`, prunes `_data` folders older than 60 days, **stamps the resolved window into the `` / `<div class="meta">` block / Generated banner**, and detects "unfilled template stub" vs "real prior report" collisions using first-KPI + size-ratio fingerprints. See AB#3683194 for the rolling-window rationale. | | [`scripts/visual-smoke.ps1`](assets/scripts/visual-smoke.ps1) | Optional Playwright-based layout smoke test. Renders the report at 1400 px viewport, captures a full-page screenshot under `~/android-oce-reports/_visual/`, and runs DOM-based overflow + adjacent-card-gap detection. Catches the rendered-layout bugs (text bleed, cards touching) that pure HTML/CSS validation can't see. | --- ## Inputs to confirm with the user -1. **Reporting week** — **first compute the most recent complete Sun→Sat week** (Sunday bucket = the most recent Sunday strictly before today, or today itself if today is a Sunday and the week's data is at least 6h old). Default to that and proceed without asking *unless*: - - today is itself a Sat or Sun **and** the user phrasing suggests they want "this week" (e.g. "current report", "latest data"). Then ASK explicitly between the in-progress and most-recent-complete options. - - today is a Mon–Fri — just default to the most recent complete week and proceed; do not ask. +> **⚠️ Do NOT ask the user for a reporting date by default.** As of AB#3683194 (Fadi feedback, Jun 2026) the skill uses a **rolling 7-day window** ending at start-of-day UTC on the invocation day. `assets/scripts/bootstrap-report.ps1` computes and stamps the window automatically. The previous "confirm the Sunday bucket with the user" flow silently produced stale windows (e.g. a Thursday run emitted a 4-day partial window and dropped the last complete week) — that failure mode is what this section explicitly guards against. - If the user picks the in-progress week: - - Add the badge text *"Live data — current bucket may still be filling"* to the report header. - - The `bucket-trends.js` `--end` flag + the `| where week < datetime(<END>)` source filter both still apply (use the Sunday AFTER the reporting week as `<END>`); they will drop the partial-end-bucket warning. +1. **Reporting window** — do NOT ask. Run `bootstrap-report.ps1` with no arguments; it resolves the window silently and prints: + ``` + Resolved reporting window (UTC): + Last 7 days: 2026-07-02 -> 2026-07-09 (exclusive upper bound) + Baseline: 2026-06-25 -> 2026-07-02 + 60-day trend: 2026-05-10 -> 2026-07-05 (8 weekly buckets, exclusive) + ``` + These dates are stamped into the report's `<title>`, `<div class="meta">`, and Generated banner during bootstrap — you do not hand-edit them (see `assets/templates/template-readme.md` § Date fields). - Note that Kusto's `startofweek()` is **Sunday-aligned**, so a user-spoken "week of May 3 → May 9" maps to the bucket `startofweek == 2026-05-03`. Off-by-one-week is the #1 silent error — verify by printing the distinct `startofweek` buckets from your first query and confirming the label matches the user's intent. -2. **Comparison baseline** — defaults to the prior complete week. -3. **60-day window** — last 8 complete weeks (drop the partial start week when computing trend deltas). -4. **Output filename** — `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html`, where `YYYY-MM-DD` is the **Sunday `startofweek` bucket** of the reporting week (e.g. the report for the week of May 3 → May 9, 2026 is `oncall-wow-report-2026-05-03.html`). User-scoped, outside the workspace; the date matches the Kusto bucket label used throughout the report. + **Override only when the user explicitly requests a non-default window.** Signals: "the week of X", "as of last Friday", "the report from three weeks ago", "in-progress data", "just today's numbers". Then use: + ```pwsh + .\bootstrap-report.ps1 -EndDate 2026-07-02 # e.g. reproduce the report as of Jul 2 + ``` + `-EndDate` is the exclusive upper bound of the current window (`curEnd`); the script derives `curStart = curEnd - 7d` and `prevStart = curEnd - 14d` deterministically. `-EndDate` must be today or earlier — future dates are refused. -If any of these are unstated, ask once, then proceed. +2. **Comparison baseline** — auto-computed as the immediately-prior 7 days (`[curEnd - 14d, curEnd - 7d)`). No user input. + +3. **60-day trend window** — auto-computed as 8 complete Sun-Sat weeks ending `startofweek(curEnd)` exclusive. This section still uses weekly bucketing (Kusto `startofweek()` is Sunday-aligned) because the trend view needs stable weekly denominators; the primary/WoW section is the only part that switched to rolling 7d. + +4. **Output filename** — `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html` where `YYYY-MM-DD` is the resolved `curEnd`. Example: run on 2026-07-09 (default) → `oncall-wow-report-2026-07-09.html`. User-scoped, outside the workspace. The filename date always matches the meta-line "Last 7 days" end-date; `validate-report.ps1` check #11 asserts this. + +**Kusto note (60-day trend section only):** `startofweek()` is Sunday-aligned, so `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When authoring weekly-bucketed queries (60-day trend, `wow-table-sparkline-series.kql`), verify by printing the distinct `startofweek(EventInfo_Time)` values from your first query. Off-by-one-week is the #1 silent error in weekly-bucket queries. --- @@ -83,20 +93,21 @@ If any of these are unstated, ask once, then proceed. ### Step 1 — Bootstrap the new report file from the template -This skill ships with a canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html) (a real prior week's report kept as the reference layout). **Use [`assets/scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1)** to handle all the boilerplate (Sunday-date computation, `_data/<sunday>/` directory, retention-pruning, collision detection): +This skill ships with a canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html) (a real prior report kept as the reference layout). **Use [`assets/scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1)** to handle all the boilerplate (rolling-window computation, `_data/<end-date>/` directory, header stamping, retention-pruning, collision detection): ```pwsh .\.github\skills\oncall-weekly-telemetry-report\assets\scripts\bootstrap-report.ps1 -# Optional: explicit reporting Sunday + force overwrite -# .\bootstrap-report.ps1 -ReportingSunday 2026-05-31 -Force +# Optional: explicit end-date (curEnd, exclusive upper bound) + force overwrite +# .\bootstrap-report.ps1 -EndDate 2026-07-02 -Force ``` What it does: -* Computes the reporting-Sunday from the system clock (most recent complete Sun-Sat week). -* Creates `~/android-oce-reports/oncall-wow-report-<sunday>.html` from the canonical template. -* Creates `~/android-oce-reports/_data/<sunday>/` for raw KQL JSON payloads. -* Prunes `_data/<old-sunday>/` folders older than 60 days so the cache doesn't accumulate. -* **Collision detection (the v8-hardened version):** uses a multi-marker fingerprint (title + meta-line dates + first-KPI value + size ratio) to distinguish an "unfilled template stub" (silently re-bootstrap) from a "real populated report" (HARD HALT, exit 2, require `-Force` to overwrite). The earlier single-marker (title only) version misclassified populated reports as stubs and overwrote real work. +* Resolves the rolling 7-day window from the system clock in UTC (`curEnd = today`, `curStart = curEnd - 7d`, `prevStart = curEnd - 14d`) — or from `-EndDate` if passed. +* Creates `~/android-oce-reports/oncall-wow-report-<curEnd>.html` from the canonical template. +* Creates `~/android-oce-reports/_data/<curEnd>/` for raw KQL JSON payloads. +* **Stamps the resolved window into the report's `<title>`, `<div class="meta">` block, and Generated banner** — you never hand-edit header dates. This is the AB#3683194 acceptance criterion ("the resolved window is echoed in the report header for transparency"). +* Prunes `_data/<old-end-date>/` folders older than 60 days so the cache doesn't accumulate. +* **Collision detection:** first-KPI value + file-size ratio distinguish an "unfilled template stub" (silently re-bootstrap) from a "real populated report" (HARD HALT, exit 2, require `-Force` to overwrite). Title/meta-line are NOT fingerprinted because bootstrap always rewrites them on copy. Edit the bootstrapped file in place — the template ships as a real prior-week report (not a tokenized skeleton). **Walk top-to-bottom and replace every prior-week date / KPI value / table row / verdict / PR citation with current-week data.** The CSS, sparkline JS, section ordering, and attribution-card markup are canonical — do not redesign them. See [`assets/templates/template-readme.md`](assets/templates/template-readme.md) for the full guide on what to change vs leave alone, the sparkline color palette, the CSS class reference, and the two v8 layout traps. @@ -154,32 +165,32 @@ Don't pre-filter to a hand-picked top-N list — small-but-rising errors (e.g. ` #### 3a. Per-error-code trend -Use [`assets/queries/60d-trend-codes.kql`](assets/queries/60d-trend-codes.kql) (template; replace `<START>` and `<END>` tokens — `<END>` is **exclusive** and equals the Sunday AFTER the reporting week, e.g. for a 2026-05-03 report use `2026-05-10`): +Use [`assets/queries/60d-trend-codes.kql`](assets/queries/60d-trend-codes.kql) (template; replace `<TREND_START>` and `<TREND_END>` tokens — `<TREND_END>` is **exclusive** and equals `startofweek(today)`, i.e. the Sunday that OPENS the current in-progress week. `bootstrap-report.ps1` prints the resolved values): ```kql materialized_view('ErrorStatsMetrics') -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>)) | where isnotempty(error_code) and error_code != 'success' | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) by week = startofweek(EventInfo_Time), error_code -| where week < datetime(<END>) // drop partial in-progress week at the source +| where week < datetime(<TREND_END>) // drop partial in-progress week at the source | order by error_code asc, week asc ``` -**The `| where week < datetime(<END>)` line is mandatory.** Without it, if Kusto has crossed midnight UTC into the next Sunday, a tiny partial bucket lands as `last` and turns every code into a fake −99% improvement. `bucket-trends.js` will also auto-detect and warn about this, but filtering at the source is preferred. +**The `| where week < datetime(<TREND_END>)` line is mandatory.** Without it, the current in-progress week lands as `last` and turns every code into a fake −99% improvement. `bucket-trends.js` will also auto-detect and warn about this, but filtering at the source is preferred. #### 3b. Per-error-type trend (same rigor) ```kql materialized_view('ErrorStatsMetrics') | extend unified_error_type = MergeUiRequiredExceptions(error_type) -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>)) | where isnotempty(unified_error_type) | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) by week = startofweek(EventInfo_Time), unified_error_type -| where week < datetime(<END>) +| where week < datetime(<TREND_END>) | order by unified_error_type asc, week asc ``` @@ -190,16 +201,17 @@ materialized_view('ErrorStatsMetrics') `bucket-trends.js` defaults to grouping by `error_code`. For the type runs you MUST pass `--key=unified_error_type` so it picks up the right column from the type-trend JSON. ```pwsh -# Error codes — by devices, then by requests -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=2026-03-08 --end=2026-05-10 -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=2026-03-08 --end=2026-05-10 --metric=reqs +# Error codes — by devices, then by requests. TREND_START = startofweek(today) - 56d; +# TREND_END = startofweek(today). Values printed by bootstrap-report.ps1. +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_END> +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_END> --metric=reqs # Error types — by devices, then by requests (note --key) -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=2026-03-08 --end=2026-05-10 --key=unified_error_type -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=2026-03-08 --end=2026-05-10 --key=unified_error_type --metric=reqs +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_END> --key=unified_error_type +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_END> --key=unified_error_type --metric=reqs ``` -`--end` is the Sunday AFTER the reporting week (exclusive). The script also auto-detects partial end-buckets and warns, but passing `--end` explicitly is safer. +`--end` is `startofweek(today)` (exclusive) — the Sunday that opens the current in-progress week. The script also auto-detects partial end-buckets and warns, but passing `--end` explicitly is safer. Take the **union** of all four regression sets. Both `error_code` and `error_type` regressions get a spike-attribution card in Step 5. @@ -228,9 +240,9 @@ The 60d bucketer's `--peak-floor=10000` exists for good reason (otherwise the 60 To catch these, **always** run [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) **as a separate pass after the 60d bucketing**: ```kql -// inputs: <CURR_START> = reporting-week Sunday, <CURR_END> = next Sunday (excl), -// <PRIOR_START> = baseline-week Sunday -// floor: cDev>=500 OR cReq>=5000 move: |Δd|>=25% OR |Δr|>=50% OR new-this-week +// inputs: <CUR_END> = curEnd (exclusive), <CUR_START> = curEnd - 7d, +// <PREV_START> = curEnd - 14d. Printed by bootstrap-report.ps1. +// floor: cDev>=500 OR cReq>=5000 move: |Δd|>=25% OR |Δr|>=50% OR new-this-window ``` Run it **twice — once for `error_code`, once for `error_type`**. **Merge its output rows into the same 🔴 WoW regressions callout as the standard WoW table** (sorted by current-week device count descending). Tag rows that came in via this pass with `NEW` if they were absent or near-zero in the prior week. Do *not* render this as a separate "emerging" callout — the size split is implementation detail; readers prioritize naturally by absolute device count + originator chip. @@ -241,7 +253,7 @@ For each WoW mover (regardless of size), you still owe the full Code Attribution > ⚠️ **HARD RULE — Originator pre-check.** Before claiming `Originator: Broker` on any card, you MUST run [`assets/queries/error-message-and-location.kql`](assets/queries/error-message-and-location.kql) for that error code (or type) and read **(a) the throw-site stack and (b) the top 3 `error_message` strings**. Most broker error codes flow through `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult, clientExceptionFromException}` — which intentionally bridge eSTS responses into broker exceptions. **If the throw site is in any of those three methods AND the error_message starts with `AADSTS`, the originator is eSTS, not broker.** See the AADSTS reference table in [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). Cards that skip this step must be marked low-confidence, not high. > -> **Window:** use the FULL 7-day reporting window (`<CURR_START>` → `<CURR_END>`) on `PipelineInfo_IngestionTime`, NOT a narrower 3–5 day slice — low-volume types (e.g. `SSLHandshakeException`, `IntuneAppProtectionPolicyRequiredException`) routinely return zero rows in a sub-week window. If a code/type still returns nothing, fall back to the prior 14 days before declaring "no data". +> **Window:** use the FULL 7-day rolling window (`<CUR_START>` → `<CUR_END>`) on `PipelineInfo_IngestionTime`, NOT a narrower 3–5 day slice — low-volume types (e.g. `SSLHandshakeException`, `IntuneAppProtectionPolicyRequiredException`) routinely return zero rows in a sub-window slice. If a code/type still returns nothing, fall back to the prior 14 days (`<PREV_START>` → `<CUR_END>`) before declaring "no data". For every regression card, the Code Attribution block **must** populate the following fields. Shallow PR-citation only is not acceptable. Use [`assets/docs/code-attribution-template.md`](assets/docs/code-attribution-template.md) as the per-card checklist. @@ -309,7 +321,7 @@ Slice on **all 7 dimensions** for each spike. **Preferred for 2-week WoW attribu For `error_type` cards, swap `error_code in (codes)` for `unified_error_type in (types)` and aggregate by the `MergeUiRequiredExceptions(error_type)` extension — otherwise everything else is identical. -> **Low-volume fallback (extends Step 4's pre-check fallback to the 7-dim union):** when a code/type returns sparse dim rows in the 7-day reporting window — typical for sub-1k-device entries like `TimeoutCancellationException`, `JsonSyntaxException`, `kdfv2_key_derivation_error` — widen the union query to **14 days** (`<START>` = baseline-week Sunday − 7d) before declaring "broad — needs targeted slice". The added week of context usually surfaces enough rows to compute concentration percentages. If a code STILL has no concentration after 14 days, mark every dim cell as "not sliced — sub-week volume; file the bug first, slice on persistence" — do NOT fabricate "Broad" verdicts. +> **Low-volume fallback (extends Step 4's pre-check fallback to the 7-dim union):** when a code/type returns sparse dim rows in the 7-day rolling window — typical for sub-1k-device entries like `TimeoutCancellationException`, `JsonSyntaxException`, `kdfv2_key_derivation_error` — widen the union query to **14 days** (use `<PREV_START>` as the lower bound so the window becomes `[curEnd − 14d, curEnd)`) before declaring "broad — needs targeted slice". The added week of context usually surfaces enough rows to compute concentration percentages. If a code STILL has no concentration after 14 days, mark every dim cell as "not sliced — sub-window volume; file the bug first, slice on persistence" — do NOT fabricate "Broad" verdicts. | # | Dimension | Source | Cross-check | |---|-----------|--------|-------------| @@ -326,12 +338,15 @@ For `error_type` cards, swap `error_code in (codes)` for `unified_error_type in Because `error_type` is an umbrella over many `error_code` values, every `error_type` regression card MUST also include an **8th dimension: sub-code breakdown** showing the top 3–5 `error_code`s rolled up under that type, with their device counts and Δ vs prior week. This lets the reader see whether the type-level move is driven by one sub-code or many — and routes the deep Code Attribution work to the right sub-code. ```kql +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let target_types = dynamic(['ClientException', 'ServiceException']); materialized_view('ErrorStatsMetrics') | extend unified_error_type = MergeUiRequiredExceptions(error_type) -| where EventInfo_Time > ago(14d) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where unified_error_type in (target_types) -| extend wk = startofweek(EventInfo_Time) +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by wk, unified_error_type, error_code @@ -430,13 +445,17 @@ Cite the suspect PR(s) with the same confidence ratings used in Code Attribution **6c. Per-error traffic attribution (is the *error* spike traffic-driven?).** For every error code flagged in Step 5 as a regression, additionally check whether the spike is *traffic-driven* rather than *failure-rate-driven*: ```kql +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let target_code = "<error_code>"; materialized_view('ErrorStatsMetrics') -| where EventInfo_Time > ago(14d) and error_code == target_code +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd and error_code == target_code +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), calling_package_name -| order by week asc, devs desc + by wk, calling_package_name +| order by wk asc, devs desc ``` If the spike is concentrated in a single calling app whose **overall** request volume also rose that week (cross-check `AppStatsUpdated`), and the **per-request failure rate is essentially flat**, classify the spike as a **traffic-attribution case** rather than a code regression: @@ -491,8 +510,8 @@ Then: - **Never sum percentiles.** Latency is a TDigest sketch — `percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long))` only. - **Always apply `MergeAccountType` / `MergeIsSharedDevice` / `MergeUiRequiredExceptions`** so this report agrees with the dashboard. - **Confirm the week bucket label matches the user's intent** before writing the rest of the queries (Sunday-aligned). -- **Always filter the partial in-progress week at the source** with `| where week < datetime(<END>)` where `<END>` is the Sunday immediately after the reporting week. Otherwise `bucket-trends.js` will show every error as a fake −99% improvement once UTC has crossed midnight Sunday. -- **Never carry a numeric telemetry value forward between runs.** Every KPI, table cell, delta %, device/request count, sparkline point, and verdict number must be re-pulled from Kusto for *this* week — never copied from last week's report, from a checkpoint/summary, from notes, or from memory. Telemetry shifts week to week and stale numbers read as fabricated. Near-miss precedent: a `no_tokens_found` count was about to be carried as ~23.7M when the actual current-week value was ~4.86M — a ~5× error that only the re-pull caught. If a number isn't backed by a query result file in this run's `_data/<sunday>/`, it does not go in the report. +- **Always filter the partial in-progress week at the source in the 60-day trend queries** with `| where week < datetime(<TREND_END>)` where `<TREND_END>` is `startofweek(today)` (exclusive). Otherwise `bucket-trends.js` will show every error as a fake −99% improvement. Primary/WoW queries don't need this filter because their `EventInfo_Time < curEnd` half-open bound handles it inherently. +- **Never carry a numeric telemetry value forward between runs.** Every KPI, table cell, delta %, device/request count, sparkline point, and verdict number must be re-pulled from Kusto for *this* run — never copied from a previous report, from a checkpoint/summary, from notes, or from memory. Telemetry shifts between runs and stale numbers read as fabricated. Near-miss precedent: a `no_tokens_found` count was about to be carried as ~23.7M when the actual current-window value was ~4.86M — a ~5× error that only the re-pull caught. If a number isn't backed by a query result file in this run's `_data/<end-date>/`, it does not go in the report. - **Never hardcode the "Generated" date.** It is the *run* date (system clock, local), auto-stamped by `bootstrap-report.ps1`. If you rebuild the body programmatically, derive it live with a **local-date** formatter (`new Date().toLocaleDateString('en-CA')` in Node, `(Get-Date).ToString('yyyy-MM-dd')` in PowerShell) — never paste a literal, and avoid `new Date().toISOString().slice(0,10)` (UTC; stamps tomorrow's date when run in the evening in a UTC-negative timezone). The v8 "Generated 2026-06-15 on a 2026-06-18 file" bug came from a hardcoded string in the assembler. (Reporting-week / baseline / 60d window dates are author-set and verified against the user's intended Sunday bucket — see template-readme "Date fields".) - **Originator pre-check is mandatory.** A card cannot claim `Originator: Broker` without first running [`assets/queries/error-message-and-location.kql`](assets/queries/error-message-and-location.kql) and reading the throw site + top 3 `error_message` strings. If the throw site is in `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult}` AND the message starts with `AADSTS`, the originator is **eSTS, not broker** — see the AADSTS reference in [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). - **WoW-movers pass is mandatory.** The 60d bucketer's `--peak-floor` silently drops sub-10K-device codes, so [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) MUST be run as a separate pass for both `error_code` and `error_type` (per Step 3d). Its output is **merged into the single 🔴 WoW regressions callout**, sorted by current-week device count descending, with rows tagged `NEW` / `60d↑` / originator chip. Do not render a separate "emerging" callout. Skipping the pass is how the Apr 26 `Failed to parse JWT` spike (7 → 3,461 devs over 7 weeks) hid for two reports running. @@ -512,9 +531,9 @@ Then: ## Output checklist -- [ ] New `oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the reporting-week Sunday) exists at `$env:USERPROFILE\android-oce-reports\` (NOT at repo root). If a file for this Sunday already existed, the chat session explicitly stated what changed before regenerating. +- [ ] New `oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the resolved `curEnd` — the end-date of the rolling 7-day window) exists at `$env:USERPROFILE\android-oce-reports\` (NOT at repo root). If a file for this end-date already existed, the chat session explicitly stated what changed before regenerating. - [ ] All sections present and populated (incl. 🚚 Traffic Attribution — even if “None this week”) -- [ ] **60-day trend bucketing run on the full cross-product** — `{error_code, error_type} × {devices, requests}` = 4 runs — union of regressions reported. Per-request retry storms (e.g. small device pool, exploding request count) are flagged on both axes. Source KQL filtered the partial in-progress week with `| where week < datetime(<END>)`. +- [ ] **60-day trend bucketing run on the full cross-product** — `{error_code, error_type} × {devices, requests}` = 4 runs — union of regressions reported. Per-request retry storms (e.g. small device pool, exploding request count) are flagged on both axes. Source KQL filtered the partial in-progress week with `| where week < datetime(<TREND_END>)`. - [ ] **WoW-movers pass run** ([`wow-movers.kql`](assets/queries/wow-movers.kql)) for BOTH `error_code` and `error_type`. Its output rows are **merged into the single 🔴 WoW regressions callout in Section 2** (sorted by curr-week devices descending), each row tagged `NEW` / `60d↑` / originator chip. No separate "emerging" callout. Every row carries throw-site, dominant message, originator, and a next step. If the WoW callout is empty (rare), render "None this week" rather than omit. - [ ] **Both error-codes AND error-types WoW tables have `Δ requests %` and `Δ devices %` columns**, the 60d sparkline, and a status pill. Any row crossing threshold on either metric is in the regression list. - [ ] Every WoW regression AND every 60d regression — **for both `error_code` and `error_type`** — has its own spike-attribution card with all 7 dimensions sliced. Cards are built from [`assets/templates/spike-card.html`](assets/templates/spike-card.html). diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md b/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md index ebc4c9de..5f2a5948 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md @@ -38,6 +38,42 @@ Time filter on materialized views is always **`EventInfo_Time`**. Use `PipelineI --- +## 3. Rolling 7-day WoW window (PRIMARY / attribution / latency) + +**Since AB#3683194 (Fadi feedback, Jun 2026)** the report's primary window is a **rolling 7-day window** ending at start-of-day UTC on `-EndDate` (default: today), NOT a Sun→Sat calendar week. Only the 60-day trend section (§ 7 below) still uses `startofweek()` bucketing. + +Canonical template — two rows per key (`wk` in `{prevStart, curStart}`): + +```kql +let curEnd = datetime(<CUR_END>); // exclusive upper bound (e.g. 2026-07-09) +let curStart = datetime(<CUR_START>); // curEnd - 7d +let prevStart = datetime(<PREV_START>); // curEnd - 14d +materialized_view('<view>') +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) +| summarize <aggregations> + by wk, <keys> +| order by <keys> asc, wk asc +``` + +Notes: +- Use `>= ... and < ...` (half-open) NOT `between (.. .. ..)`. `between` is inclusive-inclusive in Kusto; the half-open form correctly assigns the exact `curEnd` boundary to no bucket (the query drops it because `EventInfo_Time < curEnd`). +- The window boundaries are always aligned to `00:00 UTC` — this matches Kusto's default datetime semantics and the bootstrap script's `-EndDate` interpretation. +- `wk` values are datetimes; they sort lexicographically (ISO 8601) so downstream JS helpers (`agg.js`, `summarize-attribution.js`) that pick smallest = prev, largest = cur continue to work unchanged. +- All primary/WoW `.kql` templates in `../queries/` follow this pattern. See `../queries/reliability-auth-only.kql` and `../queries/attr-union-by-dim.kql` for full examples. + +Compute the placeholder values via `bootstrap-report.ps1` (which prints them to stdout as `Resolved reporting window (UTC):`) or manually: + +| Placeholder | Formula | Example (`-EndDate 2026-07-09`) | +|---|---|---| +| `<CUR_END>` | `-EndDate` | `2026-07-09` | +| `<CUR_START>` | `<CUR_END> - 7d` | `2026-07-02` | +| `<PREV_START>` | `<CUR_END> - 14d` | `2026-06-25` | +| `<TREND_END>` | `startofweek(<CUR_END>)` — Sunday-aligned | `2026-07-05` | +| `<TREND_START>` | `<TREND_END> - 56d` | `2026-05-10` | + +--- + ## 3. THE distinct-device-count gotcha (most important rule) `countDevices` on `ErrorStats*` is a **per-row distinct count, not additive**. If you sum it across multiple rows you will double-count any device that appeared in more than one slice. **The dashboard never does this.** Every dashboard query computes devices via: @@ -101,7 +137,9 @@ materialized_view('PerfStatsUpdated') ## 7. Week alignment — Kusto `startofweek()` is **Sunday-aligned** -If a user says "the week of May 2 → May 9", Kusto buckets it as `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. **Always confirm**: print the distinct `startofweek(EventInfo_Time)` values from your first query and verify the bucket label matches the user's intent. Off-by-one-week is the #1 silent error. +> **Scope:** only the 60-day trend section (§ 3 of the report / `bucket-trends.js` pipeline) still uses `startofweek()` weekly buckets. The primary/WoW section uses a rolling 7-day window — see § 3 of this cheatsheet. + +If a user says "the week of May 2 → May 9", Kusto buckets it as `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When writing weekly-bucketed queries (60-day trend, `wow-table-sparkline-series.kql`), **always confirm**: print the distinct `startofweek(EventInfo_Time)` values from your first query and verify the bucket labels match your intended range. Off-by-one-week is the #1 silent error in weekly-bucket queries. For an 8-complete-week 60-day window ending Sat May 9, the buckets are: `2026-03-08, 03-15, 03-22, 03-29, 04-05, 04-12, 04-19, 04-26, 05-03` — that's 9 buckets, one of which (the first) was a partial start. Drop the first; keep 8 complete weeks. @@ -158,19 +196,22 @@ materialized_view('ErrorStatsMetrics') | order by error_code asc, week asc ``` -### 8c. Spike attribution — one slicing dim at a time +### 8c. Spike attribution — rolling 7-day WoW form (one slicing dim at a time) -The MCP tool can return ~50–700 KB of JSON; multi-dim cartesians blow this out. **Slice one dimension per query**, then post-process with `summarize-attribution.js`: +The MCP tool can return ~50–700 KB of JSON; multi-dim cartesians blow this out. **Slice one dimension per query**, then post-process with `summarize-attribution.js`. The rolling-window form (see § 3): ```kql +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let codes = dynamic(['no_tokens_found','unauthorized_client','Code:-6', 'unknown_crypto_error','null_pointer_error','timed_out_execution']); materialized_view('ErrorStatsMetrics') | extend unified_account_type = MergeAccountType(account_type) | extend unified_is_shared_device = MergeIsSharedDevice(is_shared_device) -| where EventInfo_Time > ago(14d) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where error_code in (codes) -| extend wk = startofweek(EventInfo_Time) +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize devs = dcount_hll(hll_merge(countDevicesHll)) by wk, error_code, span_name // <-- swap this dim per query | order by error_code asc, wk asc, devs desc @@ -206,7 +247,7 @@ materialized_view('BrokerAdoptionStatsUpdated') | [`summarize-attribution.js`](summarize-attribution.js) | Roll up 7-dim attribution slices per (error_code, week) — feeds the spike-attribution cards | | [`queries/`](queries/) | Canonical KQL templates, one per query — see [`queries/README.md`](queries/README.md) | | [`templates/`](templates/) | Copy-paste HTML snippets for cards / footer JS | -| [`report-template.html`](../templates/report-template.html) | Canonical layout. Copy to `~/android-oce-reports/oncall-wow-report-<sunday>.html` and replace `{{TOKENS}}` only — never restructure CSS | +| [`report-template.html`](../templates/report-template.html) | Canonical layout. Bootstrap copies it to `~/android-oce-reports/oncall-wow-report-<end-date>.html` and stamps the resolved window into the header. Replace prose/values only — never restructure CSS. | --- diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql index a9e03506..2cc227e1 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql @@ -1,15 +1,24 @@ // 60-day per-error-code trend. +// +// The 60-day trend section keeps Sun-Sat weekly bucketing (Kusto's +// startofweek() is Sunday-aligned). Only the primary/WoW window is rolling. +// // Inputs (replace before pasting): -// <START> = first Sunday of the 60d window (e.g. 2026-03-08) -// <END> = end of the reporting week, EXCLUSIVE = next Sunday after the -// reporting week's Sunday (e.g. for a 2026-05-03 report, use 2026-05-10) -// Output: feed to assets/scripts/bucket-trends.js with --start=<START> (no --end needed -// because we filter the partial bucket out at the source — preferred). +// <TREND_START> = first Sunday of the 60d window (e.g. 2026-05-10 for a report +// run on 2026-07-09) +// <TREND_END> = Sunday that OPENS the current in-progress week, EXCLUSIVE. +// Compute as startofweek(today). For a run on 2026-07-09 (Thu), +// use 2026-07-05. The final `where week < datetime(<TREND_END>)` +// drops the partial in-progress bucket at the source. +// +// Output: feed to assets/scripts/bucket-trends.js with --start=<TREND_START> +// (no --end needed because we filter the partial bucket out at the source -- +// preferred). materialized_view('ErrorStatsMetrics') -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>)) | where isnotempty(error_code) and error_code != 'success' | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) by week = startofweek(EventInfo_Time), error_code -| where week < datetime(<END>) // drop partial end-week +| where week < datetime(<TREND_END>) // drop partial in-progress week | order by error_code asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql index 951e840f..a9be87c5 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql @@ -1,10 +1,17 @@ // 60-day per-error-type trend (with MergeUiRequiredExceptions to collapse variants). +// +// The 60-day trend section keeps Sun-Sat weekly bucketing (Kusto's +// startofweek() is Sunday-aligned). Only the primary/WoW window is rolling. +// +// Inputs (see 60d-trend-codes.kql for detailed semantics): +// <TREND_START> first Sunday of the 60d window +// <TREND_END> startofweek(today), exclusive materialized_view('ErrorStatsMetrics') | extend unified_error_type = MergeUiRequiredExceptions(error_type) -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>)) | where isnotempty(unified_error_type) | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) by week = startofweek(EventInfo_Time), unified_error_type -| where week < datetime(<END>) +| where week < datetime(<TREND_END>) | order by unified_error_type asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md b/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md index 12351dae..b33ed975 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md @@ -1,34 +1,45 @@ # `assets/queries/` — canonical KQL templates Each `.kql` here is a paste-and-replace template for one of the queries the OCE -weekly report needs. Token convention: +report needs. The reporting window is a **rolling 7-day window** ending at +start-of-day UTC on `-EndDate` (default: today). See +[`../scripts/bootstrap-report.ps1`](../scripts/bootstrap-report.ps1) for the +canonical window computation, and SKILL.md § "Inputs to confirm" for the +rationale (AB#3683194). + +## Placeholder convention | Token | Meaning | |---|---| -| `<START>` | Sunday of the earliest week in the window, ISO date e.g. `2026-03-08` | -| `<END>` | Sunday immediately AFTER the reporting week (EXCLUSIVE upper bound). For a 2026-05-03 report use `2026-05-10`. | -| `<PREV_WEEK>` | Sunday of the prior week (the WoW baseline). | +| `<CUR_END>` | Exclusive upper bound of the current 7-day window (e.g. `2026-07-09` on a run at any local time on 2026-07-09 UTC). | +| `<CUR_START>` | `<CUR_END> - 7d` (e.g. `2026-07-02`). Inclusive lower bound of the current window. | +| `<PREV_START>` | `<CUR_END> - 14d` (e.g. `2026-06-25`). Inclusive lower bound of the prior 7-day baseline window. The baseline window is `[PREV_START, CUR_START)`. | +| `<TREND_START>` | First Sunday of the 60-day trend, i.e. `startofweek(CUR_END) - 56d`. **Sunday-aligned.** | +| `<TREND_END>` | Sunday that OPENS the current in-progress week, exclusive: `startofweek(CUR_END)`. On a `2026-07-09` (Thu) run, that's `2026-07-05`. | | `<CODES_LIST>` | Comma-separated KQL string list, e.g. `'invalid_resource', 'null_pointer_error'` | | `<TYPES_LIST>` | Same shape but for `unified_error_type`. | -| `<DIM>` | A single column name, replace per dimension run. | +| `<DIM>` | A single column name, replaced per dimension run. | + +**The primary/WoW queries emit two rows per key (bucket = `prevStart` or `curStart`).** The JS helpers (`agg.js`, `summarize-attribution.js`) sort the bucket label lexicographically and treat the smaller value as "prev" and the larger as "cur" — so any pair of sortable datetimes works. -**The `<END>` filter is mandatory.** Always include `| where week < datetime(<END>)` after the `summarize` so the partial in-progress week is dropped at the source. Otherwise `bucket-trends.js` will see a fake −99% improvement on every code (the partial bucket will look like a fleet-wide collapse). +**The 60-day trend queries still emit Sun-Sat weekly buckets.** `startofweek()` is Sunday-aligned in Kusto. The `| where week < datetime(<TREND_END>)` filter drops the partial in-progress week at the source, which is required — otherwise `bucket-trends.js` will see a fake −99% improvement on every code. ## File index | File | Purpose | Section it feeds | |---|---|---| -| [`reliability-auth-only.kql`](reliability-auth-only.kql) | Per-week auth-only requests/devices | Top-line health, denominator caveat | -| [`broker-version-share.kql`](broker-version-share.kql) | Per-week per-version share — **evidence for denominator caveat** | Denominator caveat callout, broker adoption | +| [`reliability-auth-only.kql`](reliability-auth-only.kql) | Auth-only requests/devices for the current + prior 7-day windows | Top-line health, denominator caveat | +| [`broker-version-share.kql`](broker-version-share.kql) | Per-version share for the WoW window — **evidence for denominator caveat** | Denominator caveat callout, broker adoption | | [`broker-version-share-wow.kql`](broker-version-share-wow.kql) | Single WoW snapshot of version share — fastest evidence for cohort transitions | Denominator caveat callout | -| [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes | 60-day trend analysis | -| [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types | 60-day trend analysis | -| [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply this week (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the single 🔴 WoW regressions callout** alongside the standard WoW table; tag rows that were absent or near-zero last week with `NEW`. Do not render a separate "emerging" callout. | 🔴 WoW regressions callout (Section 2) | -| [`attr-union-by-dim.kql`](attr-union-by-dim.kql) | **PREFERRED for 2-week WoW.** All 7 dims for N codes (or types) in ONE round-trip; pipe through `summarize-attribution.js --union`. | Spike attribution cards | -| [`attr-codes-by-dim.kql`](attr-codes-by-dim.kql) | Per-dim form (run 7 times). Fall back to this only when the union exceeds payload size or the time window is wider than 2 weeks. | Spike attribution cards | +| [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes (Sun-Sat weekly buckets) | 60-day trend analysis | +| [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types (Sun-Sat weekly buckets) | 60-day trend analysis | +| [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply in the current window (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the single 🔴 WoW regressions callout** alongside the standard WoW table; tag rows that were absent or near-zero in the prior window with `NEW`. Do not render a separate "emerging" callout. | 🔴 WoW regressions callout (Section 2) | +| [`attr-union-by-dim.kql`](attr-union-by-dim.kql) | **PREFERRED for WoW.** All 7 dims for N codes (or types) in ONE round-trip; pipe through `summarize-attribution.js --union`. | Spike attribution cards | +| [`attr-codes-by-dim.kql`](attr-codes-by-dim.kql) | Per-dim form (run 7 times). Fall back to this only when the union exceeds payload size. | Spike attribution cards | | [`attr-types-by-dim.kql`](attr-types-by-dim.kql) | Per-dim form for type regressions | Spike attribution cards | | [`type-subcode-decomposition.kql`](type-subcode-decomposition.kql) | 8th dim for type cards | Type spike-attribution cards | -| [`error-message-and-location.kql`](error-message-and-location.kql) | **MANDATORY** for every broker-tagged regression. Now accepts BOTH `<CODES_LIST>` and `<TYPES_LIST>` so codes + types can be sliced in one round-trip. | Code attribution block | +| [`error-message-and-location.kql`](error-message-and-location.kql) | **MANDATORY** for every broker-tagged regression. Accepts BOTH `<CODES_LIST>` and `<TYPES_LIST>` so codes + types can be sliced in one round-trip. | Code attribution block | | [`os-version-slice.kql`](os-version-slice.kql) | OS / OEM concentration (raw `android_spans`). **On-demand only** per Step 5 — don't slice every card. | OS-version dim in attribution cards (when applicable) | -| [`latency.kql`](latency.kql) | p50/p95/p99 by hot span | Latency section | -| [`app-share.kql`](app-share.kql) | Top calling apps by week | Traffic analysis | +| [`latency.kql`](latency.kql) | p50/p95/p99 by hot span for the WoW window | Latency section | +| [`app-share.kql`](app-share.kql) | Top calling apps for the WoW window | Traffic analysis | +| [`wow-table-sparkline-series.kql`](wow-table-sparkline-series.kql) | 8-week per-code/per-type sparkline series (Sun-Sat weekly buckets) for the WoW table `data-trend` arrays. **MANDATORY** — sparkline arrays must come from real data, never fabricated. | Section 6/7 tables | diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/app-share.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/app-share.kql index 4e138833..0d34f320 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/app-share.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/app-share.kql @@ -1,11 +1,17 @@ -// Top calling apps share for last N weeks (typically 3). +// Top calling apps share for the WoW window. +// +// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1 for the canonical +// window computation. Two rows per app (wk = prevStart or curStart). +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); materialized_view('AppStatsUpdated') -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize req = sum(countRequests), dev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), calling_package_name -| where week < datetime(<END>) -| order by week asc, req desc + by wk, calling_package_name +| order by wk asc, req desc | summarize topApps = make_list(pack('app', calling_package_name, 'req', req, 'dev', dev), 25) - by week -| order by week asc + by wk +| order by wk asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-codes-by-dim.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-codes-by-dim.kql index fa31ff56..0195923d 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-codes-by-dim.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-codes-by-dim.kql @@ -2,15 +2,20 @@ // Run 7 times with <DIM> set to each of: // span_name | calling_package_name | active_broker_package_name | // broker_version | unified_account_type | unified_is_shared_device | client_sku -// (Plus android_spans-based for OS version — see os-version-slice.kql.) +// (Plus android_spans-based for OS version -- see os-version-slice.kql.) +// +// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1 for the +// canonical window computation. Two wk buckets per (code, dim value). +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let codes = dynamic([<CODES_LIST>]); materialized_view('ErrorStatsMetrics') | extend unified_account_type = MergeAccountType(account_type) | extend unified_is_shared_device = MergeIsSharedDevice(is_shared_device) -| where EventInfo_Time between (datetime(<PREV_WEEK>) .. datetime(<END>)) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where error_code in (codes) -| extend wk = startofweek(EventInfo_Time) -| where wk < datetime(<END>) +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by wk, error_code, <DIM> diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-types-by-dim.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-types-by-dim.kql index d9b06736..a9ad01d0 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-types-by-dim.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-types-by-dim.kql @@ -1,14 +1,18 @@ // Spike attribution: types by ONE dimension at a time. // Same usage as attr-codes-by-dim.kql but for error_type regressions. +// +// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1. +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let types = dynamic([<TYPES_LIST>]); materialized_view('ErrorStatsMetrics') | extend unified_error_type = MergeUiRequiredExceptions(error_type) | extend unified_account_type = MergeAccountType(account_type) | extend unified_is_shared_device = MergeIsSharedDevice(is_shared_device) -| where EventInfo_Time between (datetime(<PREV_WEEK>) .. datetime(<END>)) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where unified_error_type in (types) -| extend wk = startofweek(EventInfo_Time) -| where wk < datetime(<END>) +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by wk, unified_error_type, <DIM> diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-union-by-dim.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-union-by-dim.kql index 00afd33f..d5ac0bda 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-union-by-dim.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/attr-union-by-dim.kql @@ -1,42 +1,49 @@ -// Spike-attribution union — all 7 dims for N codes (or N types) in ONE query. +// Spike-attribution union -- all 7 dims for N codes (or N types) in ONE query. // -// Recommended for the standard 2-week WoW attribution pass (Step 5 of SKILL.md). -// 1 round-trip vs 7. ~800 KB payload for 8 codes; well under the MCP limit. +// Recommended for the standard WoW attribution pass (SKILL.md Step 5). 1 +// round-trip vs 7. ~800 KB payload for 8 codes; well under the MCP limit. // Falls back to per-dim files (assets/queries/attr-codes-by-dim.kql) if you // need a wider time window or you exceed payload size. // +// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1 for the +// canonical window computation. +// // Inputs: -// <CODES> e.g. dynamic(['no_tokens_found','timed_out_execution', ...]) -// <START> inclusive (e.g. datetime(2026-04-26)) -// <END> EXCLUSIVE Sunday after the reporting week (e.g. datetime(2026-05-10)) -// <KEY> either `error_code` or `unified_error_type` (the latter for type cards) +// <CODES> e.g. dynamic(['no_tokens_found','timed_out_execution', ...]) +// <CUR_END> e.g. 2026-07-09 (exclusive) +// <CUR_START> curEnd - 7d, e.g. 2026-07-02 +// <PREV_START> curEnd - 14d, e.g. 2026-06-25 +// <KEY> either `error_code` or `unified_error_type` (the latter for type cards) // // Output schema (consumed by `summarize-attribution.js --union`): // dim string short label per dimension -// wk datetime reporting week +// wk datetime bucket start (prevStart or curStart) // <KEY> string error_code or unified_error_type // val_string string dim value (cast via tostring() in every union leg) // devs long dcount_hll merged device count // errs long sum of countOverall (request count) // -// For type cards, swap the first line and key: +// For type cards, swap the first `base = ...` line and the key column: // let base = materialized_view('ErrorStatsMetrics') // | extend unified_error_type = MergeUiRequiredExceptions(error_type) -// | where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +// | where EventInfo_Time >= prevStart and EventInfo_Time < curEnd // | where unified_error_type in (<TYPES>) -// | extend wk = startofweek(EventInfo_Time); - -// IMPORTANT — column-aliasing gotcha: every union branch MUST emit `val_string` +// | extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart); +// +// IMPORTANT -- column-aliasing gotcha: every union branch MUST emit `val_string` // as a real `string` (never `bool(null)`), or Kusto will rename the columns // `val_string_string` and `val_string_bool` in the result schema, which then // breaks `summarize-attribution.js` (it now accepts both names as a fallback, // but emitting one consistent `string` column is cleaner). Use `tostring()` on // non-string dims (e.g. shared_dev) so every leg has a string-typed column. +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let codes = dynamic([<CODES>]); let base = materialized_view('ErrorStatsMetrics') - | where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) + | where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where error_code in (codes) - | extend wk = startofweek(EventInfo_Time); + | extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart); (base | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by dim='span', wk, error_code, val_string=tostring(span_name)) @@ -60,5 +67,4 @@ let base = materialized_view('ErrorStatsMetrics') | union (base | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by dim='client_sku', wk, error_code, val_string=tostring(client_sku)) -| where wk < datetime(<END>) | order by error_code asc, dim asc, wk asc, devs desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share-wow.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share-wow.kql index 0fe05c58..46dd009e 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share-wow.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share-wow.kql @@ -1,24 +1,30 @@ -// WoW broker-version share comparison \u2014 the canonical evidence for the +// WoW broker-version share comparison -- the canonical evidence for the // "denominator caveat" callout when an entire version cohort retires. Use this // instead of the time-series `broker-version-share.kql` when you need a single // WoW snapshot showing which versions gained/lost share. Modeled on // `wow-movers.kql`. // +// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1 for the canonical +// window computation. +// // Inputs: -// <CURR_START> Sunday of the reporting week (e.g. 2026-05-03) -// <CURR_END> Sunday after (exclusive, e.g. 2026-05-10) -// <PRIOR_START> Sunday of the baseline week (e.g. 2026-04-26) +// <CUR_END> end of the current window, EXCLUSIVE (e.g. 2026-07-09) +// <CUR_START> curEnd - 7d (e.g. 2026-07-02) +// <PREV_START> curEnd - 14d (e.g. 2026-06-25) // -// Floor: only versions with >100M reqs in either week (filters long-tail). -// Output sorted by current-week req count descending. +// Floor: only versions with >100M reqs in either window (filters long-tail). +// Output sorted by current-window req count descending. +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let curr = materialized_view('BrokerAdoptionStatsUpdated') - | where EventInfo_Time between (datetime(<CURR_START>) .. datetime(<CURR_END>)) + | where EventInfo_Time >= curStart and EventInfo_Time < curEnd | summarize cReq = sum(countRequests), cDev = dcount_hll(hll_merge(countDevicesHll)) by broker_version; let prior = materialized_view('BrokerAdoptionStatsUpdated') - | where EventInfo_Time between (datetime(<PRIOR_START>) .. datetime(<CURR_START>)) + | where EventInfo_Time >= prevStart and EventInfo_Time < curStart | summarize pReq = sum(countRequests), pDev = dcount_hll(hll_merge(countDevicesHll)) by broker_version; diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share.kql index bdfc5a41..1c2bd0db 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/broker-version-share.kql @@ -1,10 +1,17 @@ -// Per-broker-version request and device share — the canonical evidence for -// the "denominator caveat" callout. If the all-spans device count moved >20% -// WoW, this query tells you WHICH version cohort drove it. +// Per-broker-version request and device share for the WoW window -- the +// canonical evidence for the "denominator caveat" callout. If the all-spans +// device count moved >20% WoW, this query tells you WHICH version cohort drove +// it. +// +// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1 for the canonical +// window computation. Two rows per broker_version (wk = prevStart or curStart). +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); materialized_view('BrokerAdoptionStatsUpdated') -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize req = sum(countRequests), dev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), broker_version -| where week < datetime(<END>) -| order by week asc, req desc + by wk, broker_version +| order by wk asc, req desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/error-message-and-location.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/error-message-and-location.kql index 0272daaa..bd1c0500 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/error-message-and-location.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/error-message-and-location.kql @@ -11,30 +11,34 @@ // THIS TEMPLATE COVERS BOTH error_code AND error_type IN ONE ROUND-TRIP. // Pass an empty list for the side you don't want to slice. // +// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1. +// // Inputs: // <CODES_LIST> e.g. 'invalid_resource', 'null_pointer_error' (or empty) // <TYPES_LIST> e.g. 'IntuneAppProtectionPolicyRequiredException' (or empty) -// <START> datetime — should be the reporting-week Sunday (e.g. 2026-05-03). -// Use the FULL 7-day reporting window, NOT a narrower 3-5 day slice -// (low-volume types like SSLHandshakeException / Intune* may return -// zero rows in a sub-week window). -// <END> datetime of next Sunday (exclusive) +// <CUR_START> datetime -- start of the current 7-day window (e.g. 2026-07-02) +// Use the FULL 7-day window, NOT a narrower 3-5 day slice +// (low-volume types like SSLHandshakeException / Intune* may +// return zero rows in a sub-window slice). +// <CUR_END> datetime -- exclusive upper bound (e.g. 2026-07-09) // -// Tip: if the reporting window returns no rows for a low-volume code/type, fall -// back to the prior 14-day window (`<START> - 7d .. <END>`) before giving up. +// Tip: if the current 7-day window returns no rows for a low-volume code/type, +// fall back to the prior 14 days (<PREV_START> .. <CUR_END>) before giving up. // // Output column 'loc' is a JSON blob {"ClassName":"...","MethodName":"...","LineNumber":N} -// — this is normal. Read it as text. To project the method name only, use +// -- this is normal. Read it as text. To project the method name only, use // parse_json(loc).MethodName // // HARD RULE (per SKILL.md Step 4): if the throw site is in // ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult} // AND the message starts with "AADSTS", the originator is eSTS, not broker. +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); let codes = dynamic([<CODES_LIST>]); let types = dynamic([<TYPES_LIST>]); android_spans -| where PipelineInfo_IngestionTime between (datetime(<START>) .. datetime(<END>)) +| where PipelineInfo_IngestionTime >= curStart and PipelineInfo_IngestionTime < curEnd | where (array_length(codes) > 0 and error_code in (codes)) or (array_length(types) > 0 and error_type in (types)) | extend loc = tostring(error_location), diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/latency.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/latency.kql index 471fd2ca..8c04c2b0 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/latency.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/latency.kql @@ -1,13 +1,19 @@ // p50 / p95 / p99 latency on the hot spans. Always merge TDigest before percentile. +// +// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1 for the +// canonical window computation. Two wk buckets per span. +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); materialized_view('PerfStatsUpdated') -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where span_name in ('AcquireTokenSilent','AcquireTokenInteractive', 'GetAccounts','RemoveAccount','ProcessWebsiteRequest') | where span_status == 'OK' +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize p50 = percentile_tdigest(tdigest_merge(responseTimeTDigest), 50, typeof(long)), p95 = percentile_tdigest(tdigest_merge(responseTimeTDigest), 95, typeof(long)), p99 = percentile_tdigest(tdigest_merge(responseTimeTDigest), 99, typeof(long)), reqs = sum(countRequests) - by week = startofweek(EventInfo_Time), span_name -| where week < datetime(<END>) -| order by span_name asc, week asc + by wk, span_name +| order by span_name asc, wk asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/os-version-slice.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/os-version-slice.kql index dcb93df6..e8bb6581 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/os-version-slice.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/os-version-slice.kql @@ -1,8 +1,17 @@ // OS version slice (8th attribution dim). Requires raw android_spans because // ErrorStatsMetrics doesn't carry DeviceInfo_OsVersion. Keep the time window // tight (<= 7 days) to stay under the MCP 240s timeout. +// +// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1. +// +// Inputs: +// <CUR_START> e.g. 2026-07-02 +// <CUR_END> exclusive upper bound, e.g. 2026-07-09 +// <CODES_LIST> e.g. 'invalid_resource', 'null_pointer_error' +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); android_spans -| where PipelineInfo_IngestionTime between (datetime(<START>) .. datetime(<END>)) +| where PipelineInfo_IngestionTime >= curStart and PipelineInfo_IngestionTime < curEnd | where error_code in (<CODES_LIST>) | summarize devs = dcount(DeviceInfo_Id), cnt = count() diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/reliability-auth-only.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/reliability-auth-only.kql index e9758b21..7160a1d0 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/reliability-auth-only.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/reliability-auth-only.kql @@ -1,14 +1,29 @@ -// Auth-only denominator and reliability per week. +// Auth-only denominator and reliability for the WoW window. +// +// Rolling 7-day window. curEnd is EXCLUSIVE (00:00 UTC of the invocation day); +// curStart = curEnd - 7d, prevStart = curEnd - 14d. See SKILL.md and +// assets/scripts/bootstrap-report.ps1 for the canonical window computation. +// +// Inputs: +// <CUR_END> e.g. 2026-07-09 (exclusive) +// <CUR_START> curEnd - 7d, e.g. 2026-07-02 +// <PREV_START> curEnd - 14d, e.g. 2026-06-25 +// +// Output shape: two rows per bucket, wk in { prevStart, curStart }. +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let s = materialized_view('SilentAuthStatsAllRequestsMetrics') - | where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) + | where EventInfo_Time >= prevStart and EventInfo_Time < curEnd + | extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize req = sum(countRequests), devHll = hll_merge(countDevicesHll) - by week = startofweek(EventInfo_Time); + by wk; let i = materialized_view('InteractiveAuthStatsAllRequestsMetrics') - | where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) + | where EventInfo_Time >= prevStart and EventInfo_Time < curEnd + | extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize req = sum(countRequests), devHll = hll_merge(countDevicesHll) - by week = startofweek(EventInfo_Time); + by wk; union s, i | summarize authReq = sum(req), authDev = dcount_hll(hll_merge(devHll)) - by week -| where week < datetime(<END>) -| order by week asc + by wk +| order by wk asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/type-subcode-decomposition.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/type-subcode-decomposition.kql index 4454c79f..2ab5bcf0 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/type-subcode-decomposition.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/type-subcode-decomposition.kql @@ -1,12 +1,16 @@ // Sub-code decomposition for an error_type regression card (the "8th dim"). // Shows top error_codes that roll up under each unified_error_type, with WoW devices. +// +// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1. +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let types = dynamic([<TYPES_LIST>]); materialized_view('ErrorStatsMetrics') | extend unified_error_type = MergeUiRequiredExceptions(error_type) -| where EventInfo_Time between (datetime(<PREV_WEEK>) .. datetime(<END>)) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd | where unified_error_type in (types) -| extend wk = startofweek(EventInfo_Time) -| where wk < datetime(<END>) +| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by wk, unified_error_type, error_code diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-movers.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-movers.kql index 1a53a8f9..b4d3aa8a 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-movers.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-movers.kql @@ -1,4 +1,4 @@ -// WoW movers — codes (or types) that moved sharply this week regardless of 60d shape. +// WoW movers -- codes (or types) that moved sharply this window regardless of 60d shape. // // MANDATORY pass alongside `60d-trend-codes.kql` / `60d-trend-types.kql` (per // SKILL.md Step 3b). The 60d bucketer's --peak-floor=10000 EXCLUDES errors @@ -7,10 +7,13 @@ // weeks, or `Code:-11` 937 -> 2,490 devs WoW). Without this pass those spikes // are silently dropped from the report. // +// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1 for the canonical +// window computation. +// // Inputs: -// <CURR_START> Sunday of the reporting week (e.g. 2026-05-03) -// <CURR_END> Sunday after (exclusive, e.g. 2026-05-10) -// <PRIOR_START> Sunday of the baseline week (e.g. 2026-04-26) +// <CUR_END> end of the current window, EXCLUSIVE (e.g. 2026-07-09) +// <CUR_START> curEnd - 7d (e.g. 2026-07-02) +// <PREV_START> curEnd - 14d (e.g. 2026-06-25) // // To run for error_type instead of error_code, copy this query and replace: // - error_code -> MergeUiRequiredExceptions(error_type) (alias as `t`) @@ -19,16 +22,19 @@ // Thresholds (tuneable): // floor: cDev>=500 OR cReq>=5000 (small enough to catch sub-bucketer-floor codes) // move: |dDev%|>=25 OR |dReq%|>=50 (real spike, not noise) -// new-this-wk: pDev==0 OR pReq==0 (never seen before this week) +// new-this-wk: pDev==0 OR pReq==0 (never seen before this window) +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); let curr = materialized_view('ErrorStatsMetrics') - | where EventInfo_Time between (datetime(<CURR_START>) .. datetime(<CURR_END>)) + | where EventInfo_Time >= curStart and EventInfo_Time < curEnd | where isnotempty(error_code) and error_code != 'success' | summarize cDev = dcount_hll(hll_merge(countDevicesHll)), cReq = sum(countOverall) by error_code; let prior = materialized_view('ErrorStatsMetrics') - | where EventInfo_Time between (datetime(<PRIOR_START>) .. datetime(<CURR_START>)) + | where EventInfo_Time >= prevStart and EventInfo_Time < curStart | where isnotempty(error_code) and error_code != 'success' | summarize pDev = dcount_hll(hll_merge(countDevicesHll)), pReq = sum(countOverall) diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql index 7ef5ce46..92fbba63 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql @@ -2,33 +2,43 @@ // // MANDATORY (per SKILL.md Output checklist, v8): the `data-trend` arrays in // the Section 6 (error_code) and Section 7 (error_type) WoW tables must come -// from real data — not be fabricated from a "roughly increasing" pattern. +// from real data -- not be fabricated from a "roughly increasing" pattern. // Past failure mode: small-volume codes (Broker request cancelled, // kdfv2_key_derivation_error, TimeoutCancellationException) were filtered out // by the 60d bucketer's peak-floor, then their sparklines were invented inline // in the WoW table HTML. That's data dishonesty even when the array looks plausible. // // This query returns 8 weekly buckets for every code/type that appears in -// either the WoW movers list OR the 60d trend output. Run it twice — once with -// the codes filter, once with the types filter — and feed the result into the +// either the WoW movers list OR the 60d trend output. Run it twice -- once with +// the codes filter, once with the types filter -- and feed the result into the // WoW-table generator so every row has a real-data trend. // +// NOTE ON WEEK ALIGNMENT: even though the primary WoW section now uses a +// rolling 7-day window (see SKILL.md), the sparkline still uses Sun-Sat +// weekly buckets. Rolling windows produce jagged sparkline endpoints because +// the last "week" only covers the calendar days that happen to fall in the +// current window; weekly-bucketed sparklines are cleaner and align with the +// 60-day trend section. +// // Inputs: -// <START> Sunday of week-0 (e.g. 2026-04-12 for an 8-week window ending 2026-06-06) -// <END> Sunday after the reporting week, EXCLUSIVE (e.g. 2026-06-07) -// <CODES> Dynamic list of error_code values whose sparklines we need. -// Build this from the union of: -// * wow-movers-codes.json results -// * 60d-codes regression/spike/improvement bucket members -// For the type variant, swap to `unified_error_type in (<TYPES>)` -// and the MergeUiRequiredExceptions extension. +// <TREND_START> Sunday of week-0 (e.g. 2026-05-10 for an 8-week window ending 2026-07-05) +// <TREND_END> Sunday that OPENS the current in-progress week, EXCLUSIVE. +// Compute as startofweek(today). For a report run on 2026-07-09 +// (a Thu), use 2026-07-05. The `where week < datetime(<TREND_END>)` +// drops the partial in-progress bucket at the source. +// <CODES> Dynamic list of error_code values whose sparklines we need. +// Build this from the union of: +// * wow-movers-codes.json results +// * 60d-codes regression/spike/improvement bucket members +// For the type variant, swap to `unified_error_type in (<TYPES>)` +// and the MergeUiRequiredExceptions extension. let codes = dynamic([<CODES>]); materialized_view('ErrorStatsMetrics') -| where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) +| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>)) | where error_code in (codes) | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) by week = startofweek(EventInfo_Time), error_code -| where week < datetime(<END>) +| where week < datetime(<TREND_END>) | order by error_code asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/agg.js b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/agg.js index 349e8d2d..dca04c16 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/agg.js +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/agg.js @@ -15,17 +15,24 @@ * * Input shape: a Kusto MCP JSON file produced by: * + * let curEnd = datetime(<CUR_END>); + * let curStart = datetime(<CUR_START>); + * let prevStart = datetime(<PREV_START>); * let codes = dynamic([...]); * materialized_view('ErrorStatsMetrics') - * | where EventInfo_Time between (datetime(<prev_week>) .. datetime(<this_week_end>)) + * | where EventInfo_Time >= prevStart and EventInfo_Time < curEnd * | where error_code in (codes) // or unified_error_type in (types) - * | extend wk = startofweek(EventInfo_Time) - * | where wk < datetime(<reporting_week_end_sunday>) // drop partial end! + * | extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart) * | summarize devs = dcount_hll(hll_merge(countDevicesHll)), * errs = sum(countOverall) * by wk, error_code, <ONE_DIMENSION> * | order by error_code asc, wk asc, devs desc * + * (`wk` is now the START datetime of the 7-day window -- one of two values: + * prevStart or curStart. The script sorts wks lexicographically and treats + * the smallest as "prev" and the largest as "cur", so any pair of sortable + * bucket labels works.) + * * Usage: * node agg.js <input.json> <error_key> <dim_col> [<dim_col2> ...] [--top=N] [--metric=devs|reqs] * diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 index 703aa653..6541c1fa 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 @@ -4,57 +4,86 @@ .DESCRIPTION Implements SKILL.md Step 1 as a script so the workflow doesn't drift across - runs: - 1. Computes the reporting-week Sunday from the current date (most recent - complete Sun-Sat week unless -ReportingSunday is passed explicitly). - 2. Creates ~/android-oce-reports/_data/<sunday>/ for raw query payloads. - 3. Decides what to do if the target report file already exists: - - If the existing file is an UNFILLED template stub (header dates - still match the canonical template's reference week), silently - re-bootstrap from the template — there's nothing to preserve. - - If the existing file contains real per-week content (the dates - inside differ from the template's reference week), HALT and - require the caller to explicitly delete or rename the file first. - This is the "filename collision rule" from SKILL.md. - 4. Prunes _data/<sunday>/ folders older than -DataRetentionDays (default 60) - so the directory doesn't accumulate stale payloads indefinitely. - -.PARAMETER ReportingSunday - Sunday of the reporting week (yyyy-MM-dd). If omitted, defaults to the most - recent complete Sun-Sat week relative to the system clock. + runs. The reporting window is a ROLLING 7-DAY window ending at start-of-day + (UTC) on -EndDate (defaults to today): + + curStart = EndDate - 7d (inclusive) + curEnd = EndDate (exclusive, == 00:00 UTC of the end date) + prevStart = EndDate - 14d + prevEnd = curStart + + Historical rationale (AB#3683194, Fadi feedback Jun 2026): the prior + implementation aligned the reporting window on Sun-Sat calendar weeks and + defaulted to "the Sunday of the currently in-progress week". Run on a + Thursday it emitted a 4-day partial window and dropped the last complete + week -- the exact "missed the last 6 days" the developer reported. The + rolling-window model always covers the 7 fully-elapsed days immediately + before the invocation with no user prompting. + + The script also: + 1. Creates ~/android-oce-reports/_data/<end-date>/ for raw query payloads. + 2. Copies the canonical template into + ~/android-oce-reports/oncall-wow-report-<end-date>.html. + 3. Stamps the resolved window into the <title>, the <div class="meta"> + block, and the "Generated <strong>...</strong>" banner so the header + can never drift from what was actually queried (this is the ADO + acceptance criterion "resolved window is echoed in the report header + for transparency"). + 4. Decides what to do if the target report file already exists: + - If the existing file is an UNFILLED template stub (multiple + fingerprint markers still match the canonical template), silently + re-bootstrap -- nothing to preserve. + - Otherwise HALT and require -Force. + 5. Prunes _data/<old-end-date>/ folders older than -DataRetentionDays + (default 60). + +.PARAMETER EndDate + End of the reporting window (yyyy-MM-dd, UTC). Exclusive upper bound: data + is queried up to but not including 00:00 UTC on this date. Defaults to + today (UTC). Example: -EndDate 2026-07-09 on a run at any local time on + 2026-07-09 UTC produces the same window [2026-07-02 00:00, 2026-07-09 00:00). .PARAMETER Force Skip the collision check and overwrite any existing file. .PARAMETER DataRetentionDays - How many days of _data/<sunday>/ folders to keep before pruning. Default 60. + How many days of _data/<end-date>/ folders to keep before pruning. Default 60. .PARAMETER SkillRoot Path to the skill folder. Defaults to the location of this script's parent. .EXAMPLE .\bootstrap-report.ps1 - # Default: latest complete week, halt on collision + # Default: rolling 7 days ending today (UTC), halt on collision. .EXAMPLE - .\bootstrap-report.ps1 -ReportingSunday 2026-05-31 -Force + .\bootstrap-report.ps1 -EndDate 2026-07-09 -Force + # Reproduce the report for a specific window end. .OUTPUTS Prints the absolute path of the newly created report file. + +.NOTES + Fixes AB#3683194. Breaking change vs pre-fix versions: + * -ReportingSunday parameter has been REMOVED; use -EndDate instead. + (The old name encoded the exact semantic mismatch that caused the bug.) + * Filename is now oncall-wow-report-<end-date>.html (was <sunday>.html). #> [CmdletBinding()] param( - [string]$ReportingSunday, + [string]$EndDate, [switch]$Force, [int]$DataRetentionDays = 60, [string]$SkillRoot ) $ErrorActionPreference = 'Stop' +# --------------------------------------------------------------------------- # Locate the skill folder + canonical template +# --------------------------------------------------------------------------- if (-not $SkillRoot) { - # This script lives at <skill>/assets/scripts/bootstrap-report.ps1, so go up 2 levels - # to reach <skill>/assets/. Templates live at <skill>/assets/templates/. + # This script lives at <skill>/assets/scripts/bootstrap-report.ps1, so go up + # 2 levels to reach <skill>/assets/. Templates live at <skill>/assets/templates/. $SkillRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) } $template = Join-Path $SkillRoot 'templates\report-template.html' @@ -62,61 +91,93 @@ if (-not (Test-Path $template)) { throw "Canonical template not found at $template. Pass -SkillRoot if running outside the skill folder." } -# Compute the reporting Sunday -if (-not $ReportingSunday) { - $today = [datetime]::Today - # Most recent Sunday strictly before today, OR today if today is Sunday - $offset = ($today.DayOfWeek.value__ + 7) % 7 # 0..6 days back to the previous Sunday - $sunday = $today.AddDays(-$offset) - # If today is Sunday but it's still early in the day, prefer the prior complete week - if ($today.DayOfWeek -eq [DayOfWeek]::Sunday -and (Get-Date).Hour -lt 6) { - $sunday = $sunday.AddDays(-7) - } - $ReportingSunday = $sunday.ToString('yyyy-MM-dd') +# --------------------------------------------------------------------------- +# Resolve the rolling window +# --------------------------------------------------------------------------- +if (-not $EndDate) { + # UTC "today" so the window boundaries are stable across time zones and + # match Kusto's default datetime semantics (EventInfo_Time is UTC). + $EndDate = [datetime]::UtcNow.Date.ToString('yyyy-MM-dd') } -[void][datetime]::Parse($ReportingSunday) # validate format +try { + $curEnd = [datetime]::ParseExact($EndDate, 'yyyy-MM-dd', + [System.Globalization.CultureInfo]::InvariantCulture, + [System.Globalization.DateTimeStyles]::AssumeUniversal -bor + [System.Globalization.DateTimeStyles]::AdjustToUniversal) +} catch { + throw "Invalid -EndDate '$EndDate'. Use yyyy-MM-dd." +} +$curStart = $curEnd.AddDays(-7) +$prevStart = $curEnd.AddDays(-14) +$prevEnd = $curStart + +# 60-day trend keeps weekly (Sun-Sat) bucketing -- the trend view is a +# longer-horizon signal and needs stable weekly denominators. Its exclusive +# upper bound is startofweek(curEnd) so the current in-progress week is +# always dropped from the trend. +$curEndDow = [int]$curEnd.DayOfWeek # Sun=0 .. Sat=6 +$sixtyDayEnd = $curEnd.AddDays(-$curEndDow) # Kusto startofweek() equivalent +$sixtyDayStart = $sixtyDayEnd.AddDays(-56) # 8 complete weeks + +# Sanity check: curEnd must not be in the future (a common footgun if the +# user passes a stale -EndDate that hasn't happened yet in UTC). +$nowUtc = [datetime]::UtcNow +if ($curEnd -gt $nowUtc.AddHours(1)) { + throw "Resolved curEnd $($curEnd.ToString('yyyy-MM-dd HH:mm')) UTC is in the future (now UTC = $($nowUtc.ToString('yyyy-MM-dd HH:mm'))). Refusing to bootstrap a report with a future window." +} + +$curEndStr = $curEnd.ToString('yyyy-MM-dd') +$curStartStr = $curStart.ToString('yyyy-MM-dd') +$prevStartStr = $prevStart.ToString('yyyy-MM-dd') +$prevEndStr = $prevEnd.ToString('yyyy-MM-dd') +Write-Host "Resolved reporting window (UTC):" +Write-Host " Last 7 days: $curStartStr -> $curEndStr (exclusive upper bound)" +Write-Host " Baseline: $prevStartStr -> $prevEndStr" +Write-Host " 60-day trend: $($sixtyDayStart.ToString('yyyy-MM-dd')) -> $($sixtyDayEnd.ToString('yyyy-MM-dd')) (8 weekly buckets, exclusive)" +# NOTE: Console output uses ASCII '->'; the HTML stamp below uses U+2192 arrows +# and U+00B7 middle-dots to match the template's canonical visual style. This +# is safe because $outText is written via [System.Text.UTF8Encoding]::new($false) +# which preserves multi-byte code points end-to-end (per the UTF-8 trap in +# template-readme.md, only the '@...@' heredoc-to-Set-Content path strips them). + +# --------------------------------------------------------------------------- # Paths +# --------------------------------------------------------------------------- $reportDir = Join-Path $env:USERPROFILE 'android-oce-reports' -$dataDir = Join-Path $reportDir "_data\$ReportingSunday" -$out = Join-Path $reportDir "oncall-wow-report-$ReportingSunday.html" +$dataDir = Join-Path $reportDir "_data\$curEndStr" +$out = Join-Path $reportDir "oncall-wow-report-$curEndStr.html" New-Item -ItemType Directory -Force $reportDir | Out-Null New-Item -ItemType Directory -Force $dataDir | Out-Null -# Read the template's reference dates so we can detect "unfilled stub" collisions. -# A reliable signal of "this file is the template stub": MULTIPLE markers all -# still match the template. We check title, the meta-line dates, AND the first -# KPI value — any divergence means real content has been written. +# --------------------------------------------------------------------------- +# Collision detection -- "unfilled stub" vs "real report" +# --------------------------------------------------------------------------- $templateText = [IO.File]::ReadAllText($template) function Get-FingerprintMarkers([string]$text) { + # Only fingerprint fields we do NOT stamp on write. The <title> and + # <div class="meta"> are rewritten unconditionally during bootstrap, so they + # always diverge from the template's copy after any bootstrap run -- + # including a fresh re-bootstrap of an unfilled stub. We rely on the first + # KPI value + file size to distinguish "unfilled stub" from "author has + # populated real data". $m = @{} - if ($text -match '<title>([^<]+?)') { $m['title'] = $Matches[1].Trim() } - if ($text -match '
\s*([^<]+)') { $m['metaDate'] = $Matches[1].Trim() } - # NOTE: the "Generated" date is intentionally NOT a fingerprint marker — bootstrap - # stamps it to the actual run date below, so it never matches the template after copy. - # First KPI tile's value (e.g. "10.58 B"). Differs week-to-week. if ($text -match '
\s*
[^<]+
\s*
([^<]+?)
') { $m['firstKpi'] = $Matches[1].Trim() } return $m } - $templateMarkers = Get-FingerprintMarkers $templateText -# Collision check if ((Test-Path $out) -and -not $Force) { $existingText = [IO.File]::ReadAllText($out) $existingMarkers = Get-FingerprintMarkers $existingText - # "Unfilled stub" requires ALL markers to match the template AND the file size - # to be within 5% of the template's. ANY divergence (a single value updated, - # a single KPI populated, sections added) means real content exists. $allMatch = $true foreach ($k in $templateMarkers.Keys) { if ($existingMarkers[$k] -ne $templateMarkers[$k]) { $allMatch = $false; break } } $sizeRatio = (Get-Item $out).Length / [Math]::Max(1, (Get-Item $template).Length) $sizeClose = ($sizeRatio -ge 0.95) -and ($sizeRatio -le 1.05) - $isUnfilledStub = $allMatch -and $sizeClose if ($isUnfilledStub) { @@ -132,7 +193,7 @@ if ((Test-Path $out) -and -not $Force) { $divergence += " size: template=$((Get-Item $template).Length) bytes existing=$((Get-Item $out).Length) bytes ratio=$([Math]::Round($sizeRatio,2))x" } Write-Error @" -A populated report already exists for the same Sunday bucket: +A populated report already exists for the same end-date bucket: $out Divergence from the template (which is why this is NOT classified as an unfilled stub): @@ -147,30 +208,81 @@ Per the SKILL.md filename-collision rule, do NOT silently overwrite. Either: } } -# Bootstrap +# --------------------------------------------------------------------------- +# Bootstrap: copy the template +# --------------------------------------------------------------------------- Copy-Item $template $out -Force Write-Host "Bootstrapped $out from $template" Write-Host "Data folder: $dataDir" -# Stamp the actual run date into the "Generated ..." banner so the -# report never carries a stale template date (the v8 bug where it read 2026-06-15 -# on a file produced 2026-06-18). This is purely mechanical — today's clock date — -# and has zero off-by-one risk. The reporting-week / baseline / 60-day meta dates -# are still AUTHOR-set (see template-readme.md "Date fields"); bootstrap does not -# touch them because they must be verified against the user's intended week bucket. -# Use UTF8-no-BOM read/write so the report's emojis/arrows survive (the UTF-8 trap). -$today = (Get-Date).ToString('yyyy-MM-dd') +# --------------------------------------------------------------------------- +# Stamp the resolved window into the report header +# +# The template ships with hard-coded dates from a real prior week. We rewrite +# them mechanically so the header always matches what the queries actually +# targeted. This removes the last hand-authored date field and closes the +# window-drift class of bugs entirely (AB#3683194 acceptance criterion: +# "resolved window is echoed in the report header for transparency"). +# +# Uses UTF8-no-BOM read/write so the report's emojis/arrows survive (the +# UTF-8 trap called out in template-readme.md). +# --------------------------------------------------------------------------- +function Format-DateHuman([datetime]$d, [switch]$IncludeYear) { + # e.g. "Thu Jul 2, 2026" or "Thu Jul 2" + $dow = $d.ToString('ddd', [System.Globalization.CultureInfo]::InvariantCulture) + $mon = $d.ToString('MMM', [System.Globalization.CultureInfo]::InvariantCulture) + $day = $d.Day + if ($IncludeYear) { return "$dow $mon $day, $($d.Year)" } + return "$dow $mon $day" +} +# Display dates in the header. For a half-open [curStart, curEnd) window the +# calendar dates covered are [curStart, curEnd - 1 day], but the user-facing +# convention (matching the ADO issue "Jul 2 - Jul 9 when run on Jul 9") shows +# the interval endpoints -- so we render curStart -> curEnd literally. +# U+2192 RIGHTWARDS ARROW and U+00B7 MIDDLE DOT, matching the template's style. +$arrow = [char]0x2192 +$dot = [char]0x00B7 +$curLabel = "$(Format-DateHuman $curStart -IncludeYear:$false) $arrow $(Format-DateHuman $curEnd -IncludeYear:$true)" +$prevLabel = "$(Format-DateHuman $prevStart -IncludeYear:$false) $arrow $(Format-DateHuman $prevEnd -IncludeYear:$false)" +# 60-day trend: display the last COMPLETE week's Saturday (sixtyDayEnd - 1) as the +# reader-facing upper bound; sixtyDayEnd itself is the exclusive Sunday cutoff. +$sixtyDaySatLabel = "$(Format-DateHuman $sixtyDayStart -IncludeYear:$false) $arrow $(Format-DateHuman ($sixtyDayEnd.AddDays(-1)) -IncludeYear:$false)" +$todayStr = (Get-Date).ToUniversalTime().ToString('yyyy-MM-dd') + $outText = [IO.File]::ReadAllText($out) -$outText = [regex]::Replace($outText, 'Generated\s+[^<]*', "Generated $today") + +# 1) ... +$newTitle = "Android Broker $dot On-Call Report $([char]0x2014) Last 7 days ending $curEndStr" +$outText = [regex]::Replace($outText, '[^<]*', "$newTitle") + +# 2) The
block up through the closing
immediately +# before the badge. We use a single-line non-greedy match on the full block. +$newMeta = @" +
+ Last 7 days: $curLabel  vs  $prevLabel  $dot  + 60-day trend: $sixtyDaySatLabel (8 complete weeks)  $dot  + Source: android_spans materialized views  $dot  + Generated $todayStr +
+"@ +# The template's meta div is multi-line; use single-line mode with .*? non-greedy. +$outText = [regex]::Replace($outText, '(?s)
.*?
', $newMeta) + +# 3) Belt-and-suspenders: if for some reason the meta-div rewrite above didn't +# hit (e.g. the template was restructured), still update the Generated banner +# so the file never carries a stale template date. +$outText = [regex]::Replace($outText, 'Generated\s+[^<]*', "Generated $todayStr") + [IO.File]::WriteAllText($out, $outText, [System.Text.UTF8Encoding]::new($false)) -Write-Host "Stamped Generated date: $today" +Write-Host "Stamped resolved window into and meta block. Generated=$todayStr." +# --------------------------------------------------------------------------- # Prune old _data folders +# --------------------------------------------------------------------------- $dataRoot = Join-Path $reportDir '_data' if (Test-Path $dataRoot) { $cutoff = (Get-Date).AddDays(-$DataRetentionDays) $oldFolders = Get-ChildItem $dataRoot -Directory | Where-Object { - # Folder name should look like a date; skip the current run's folder $_.FullName -ne $dataDir -and $_.LastWriteTime -lt $cutoff } diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js index c602fba3..1043bfec 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js @@ -1,19 +1,30 @@ #!/usr/bin/env node /** - * bucket-trends.js — Bucket every error code into 60-day trend categories. + * bucket-trends.js -- Bucket every error code into 60-day trend categories. + * + * This tool operates ONLY on the 60-day trend section, which still uses Sun-Sat + * weekly buckets (Kusto startofweek()-aligned). The primary/WoW section uses a + * rolling 7-day window and does NOT go through this script -- it consumes the + * two-bucket outputs of reliability-auth-only.kql / wow-movers.kql / etc. + * directly. * * Input: a Kusto MCP JSON result file from a query of the form: * * materialized_view('ErrorStatsMetrics') - * | where EventInfo_Time between (datetime(<start>) .. datetime(<end_exclusive>)) + * | where EventInfo_Time between (datetime(<trend_start>) .. datetime(<trend_end>)) * | where isnotempty(error_code) and error_code != 'success' * | summarize errs=sum(countOverall), * devs=dcount_hll(hll_merge(countDevicesHll)) * by week=startofweek(EventInfo_Time), error_code - * | where week < datetime(<reporting_week_end_sunday>) // drop partial end-week! + * | where week < datetime(<trend_end>) // drop partial in-progress week! * | order by error_code asc, week asc * -// (Use dcount_hll on countDevicesHll, NOT sum(countDevices) — see ../docs/kusto-cheatsheet.md.) + * (Use dcount_hll on countDevicesHll, NOT sum(countDevices) -- see ../docs/kusto-cheatsheet.md.) + * + * <trend_end> convention: startofweek(today) -- i.e. the Sunday that OPENS the + * currently in-progress week. Every complete week strictly before that Sunday + * is kept; the in-progress week is dropped. See assets/scripts/bootstrap-report.ps1 + * for how this is computed (`$sixtyDayEnd`). * * Usage: * node bucket-trends.js <mcp-output.json> @@ -23,7 +34,7 @@ * --start defaults to the second-earliest week in the data (drops partial start week). * --end defaults to the most recent week, but the script will WARN-AND-DROP any week * where (latest EventInfo_Time in the bucket - week-start) < 6 days, because that - * is a partial end-week and will turn every error into a fake -99% improvement. + * is a partial in-progress week and will turn every error into a fake -99% improvement. * * --metric=devs (default) buckets on weekly device counts (catches errors hitting more users) * --metric=reqs buckets on weekly request counts (catches per-device retry storms) diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/summarize-attribution.js b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/summarize-attribution.js index 9d86a75c..404aa4cc 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/summarize-attribution.js +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/summarize-attribution.js @@ -19,13 +19,16 @@ * Per-file schema: row[0] must include `error_code`, `wk`/`week`, `devs`/`countDevices`, * and exactly one trailing string column (the dimension value). * - * 2) Union mode (NEW, recommended for 2-week WoW attribution — one query covers all dims): + * 2) Union mode (recommended for WoW attribution -- one query covers all dims): * * node summarize-attribution.js --union <attribution-union.json> * * Expected schema (any column order): * dim string -- short label e.g. 'span', 'calling_app', 'broker_ver' - * wk | week datetime + * wk | week datetime -- for the rolling-window queries this is the + * bucket START datetime (prevStart or curStart); + * the script sorts them lexicographically and + * treats the smallest as prev, largest as cur. * error_code string (or `error_type` — use --key=error_type to switch) * val_string string } EITHER `val_string`+`val_bool` (Kusto union of * val_bool bool } mixed-type slice columns) ... @@ -33,7 +36,7 @@ * devs long (use `dcount_hll(hll_merge(countDevicesHll))` upstream) * errs long (optional — request count, used for retry-storm detection) * - * The union form is what Step 5 of SKILL.md now recommends — 1 round-trip vs 7. + * The union form is what Step 5 of SKILL.md recommends — 1 round-trip vs 7. * See assets/queries/attr-union-by-dim.kql. * * Output: per error_code, per dimension, the top-5 values for each week (prior + curr), diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 index e13494b3..39b78136 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 @@ -24,6 +24,11 @@ 9c. .dim-pct content guard — every dim-pct cell must be a short bare percentage (no "(...)" annotations, <= 9 chars). A long pct value starves the flex name column and collapses real labels to "1...". + 10. Rolling-window header integrity (AB#3683194): the meta-line "Last 7 + days: <curStart> → <curEnd>" dates must match the filename's end-date + (curEnd = end-date, curStart = end-date - 7d). Prevents a stale + template stub from being published as if it were fresh, and catches + any hand-edit that broke the auto-stamp. Exits with non-zero status if any HARD check fails (stale tokens, devs/reqs leak, U+FFFD, unbalanced div depth, missing layout-guard CSS). @@ -34,7 +39,7 @@ .EXAMPLE .\validate-report.ps1 - .\validate-report.ps1 -Path C:\path\to\oncall-wow-report-2026-05-03.html + .\validate-report.ps1 -Path C:\path\to\oncall-wow-report-2026-07-09.html #> [CmdletBinding()] param( @@ -316,6 +321,46 @@ if ($suspectCount -gt 0) { Pass "No suspicious low-peak data-trend arrays detected" } +# ---- 11. Rolling-window header integrity (AB#3683194) ---- +# The meta line must read "Last 7 days: <weekday> <Mon> <D>[, YYYY] -> <weekday> +# <Mon> <D>, YYYY", and those dates must be self-consistent with the filename's +# end-date (curEnd = filename date, curStart = curEnd - 7d). +$filename = Split-Path $Path -Leaf +if ($filename -match '^oncall-wow-report-(\d{4}-\d{2}-\d{2})\.html$') { + $fnEnd = [datetime]::ParseExact($Matches[1], 'yyyy-MM-dd', + [System.Globalization.CultureInfo]::InvariantCulture) + $fnStart = $fnEnd.AddDays(-7) + + # Match: "Last 7 days: <weekday> <Mon> <D>[, YYYY]? -> <weekday> <Mon> <D>, YYYY" + # -> may be U+2192 arrow or ASCII "->". + $metaRe = 'Last\s+7\s+days:\s*(?:<[^>]+>)?\s*(\w{3})\s+(\w{3})\s+(\d{1,2})(?:,\s*(\d{4}))?\s*(?:[\u2192]|->|->)\s*(\w{3})\s+(\w{3})\s+(\d{1,2}),\s*(\d{4})' + if ($content -match $metaRe) { + $s = "$($Matches[2]) $($Matches[3]), $($Matches[8])" # e.g. "Jul 2, 2026" (year from end side) + $e = "$($Matches[6]) $($Matches[7]), $($Matches[8])" + try { + $mStart = [datetime]::ParseExact($s, 'MMM d, yyyy', + [System.Globalization.CultureInfo]::InvariantCulture) + $mEnd = [datetime]::ParseExact($e, 'MMM d, yyyy', + [System.Globalization.CultureInfo]::InvariantCulture) + # If the start ended up after the end (year boundary), roll the start back a year. + if ($mStart -gt $mEnd) { $mStart = $mStart.AddYears(-1) } + if ($mStart -ne $fnStart) { + Add-Fail "Meta-line 'Last 7 days' start = $($mStart.ToString('yyyy-MM-dd')) but filename implies $($fnStart.ToString('yyyy-MM-dd')) (filename end $($fnEnd.ToString('yyyy-MM-dd')) - 7d). Header is out of sync with filename -- did you edit dates by hand? Re-run bootstrap-report.ps1." + } elseif ($mEnd -ne $fnEnd) { + Add-Fail "Meta-line 'Last 7 days' end = $($mEnd.ToString('yyyy-MM-dd')) but filename says $($fnEnd.ToString('yyyy-MM-dd'))." + } else { + Pass "Meta-line rolling-window dates match filename ($($mStart.ToString('yyyy-MM-dd')) -> $($mEnd.ToString('yyyy-MM-dd')))" + } + } catch { + Add-Warn "Meta-line dates matched pattern but failed to parse: '$s' / '$e'. Skipping integrity check." + } + } else { + Add-Warn "Meta-line 'Last 7 days: ... -> ...' pattern not found. Either the report predates AB#3683194 (rolling-window rewrite) or the header was hand-edited. Re-run bootstrap-report.ps1 to restore the auto-stamped meta line." + } +} else { + Add-Warn "Filename '$filename' does not match 'oncall-wow-report-YYYY-MM-DD.html'; skipping meta-line date consistency check." +} + Write-Host "" if ($failures.Count -eq 0) { Write-Host "All hard checks passed." -ForegroundColor Green diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/visual-smoke.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/visual-smoke.ps1 index c4ca9272..dc8076c7 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/visual-smoke.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/visual-smoke.ps1 @@ -30,7 +30,7 @@ .EXAMPLE .\visual-smoke.ps1 - # checks the latest report + writes ~/android-oce-reports/_visual/oncall-wow-report-<sunday>.png + # checks the latest report + writes ~/android-oce-reports/_visual/oncall-wow-report-<end-date>.png .EXAMPLE .\visual-smoke.ps1 -Path C:\path\to\report.html diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html index 20a1654e..c7571694 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html @@ -1,14 +1,19 @@ <!DOCTYPE html> <!-- TEMPLATE - see assets/templates/template-readme.md for usage. This file IS a real prior-week - report kept as a structural reference. To produce a new week's report, copy this - file to ~/android-oce-reports/oncall-wow-report-<sunday>.html and edit IN PLACE, - replacing prior-week dates, KPI values, table rows, attribution-card prose, and - PR citations with the current week's data. The CSS in <head> is canonical - do - not restyle per week. Authors mark unfinished sections with the literal sentinel string the validator greps for (see assets/templates/template-readme.md). --> + report kept as a structural reference. To produce a new report, run + assets/scripts/bootstrap-report.ps1 which copies this file to + ~/android-oce-reports/oncall-wow-report-<end-date>.html and STAMPS the + resolved rolling-7-day window into the <title>, meta line, and Generated + banner. You then edit IN PLACE, replacing prior-window KPI values, table + rows, attribution-card prose, and PR citations with the current window's + data. The CSS in <head> is canonical - do not restyle per run. Header + dates (title, meta line, Generated) are stamped by bootstrap - do not + hand-edit them. Authors mark unfinished sections with the literal sentinel + string the validator greps for (see assets/templates/template-readme.md). --> <html lang="en"> <head> <meta charset="UTF-8"> -<title>Android Broker · On-Call Weekly Report — Week of May 3, 2026 +Android Broker · Weekly On-Call Report