Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 68 additions & 49 deletions .github/skills/oncall-weekly-telemetry-report/SKILL.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,42 @@ Time filter on materialized views is always **`EventInfo_Time`**. Use `PipelineI

---

## 3. Rolling 7-day WoW window (PRIMARY / attribution / latency)

**Since AB#3683194 (Fadi feedback, Jun 2026)** the report's primary window is a **rolling 7-day window** ending at start-of-day UTC on `-EndDate` (default: today), NOT a Sun→Sat calendar week. Only the 60-day trend section (§ 7 below) still uses `startofweek()` bucketing.

Canonical template — two rows per key (`wk` in `{prevStart, curStart}`):

```kql
let curEnd = datetime(<CUR_END>); // exclusive upper bound (e.g. 2026-07-09)
let curStart = datetime(<CUR_START>); // curEnd - 7d
let prevStart = datetime(<PREV_START>); // curEnd - 14d
materialized_view('<view>')
| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd
| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart)
| summarize <aggregations>
by wk, <keys>
| order by <keys> asc, wk asc
```

Notes:
- Use `>= ... and < ...` (half-open) NOT `between (.. .. ..)`. `between` is inclusive-inclusive in Kusto; the half-open form correctly assigns the exact `curEnd` boundary to no bucket (the query drops it because `EventInfo_Time < curEnd`).
- The window boundaries are always aligned to `00:00 UTC` — this matches Kusto's default datetime semantics and the bootstrap script's `-EndDate` interpretation.
- `wk` values are datetimes; they sort lexicographically (ISO 8601) so downstream JS helpers (`agg.js`, `summarize-attribution.js`) that pick smallest = prev, largest = cur continue to work unchanged.
- All primary/WoW `.kql` templates in `../queries/` follow this pattern. See `../queries/reliability-auth-only.kql` and `../queries/attr-union-by-dim.kql` for full examples.

Compute the placeholder values via `bootstrap-report.ps1` (which prints them to stdout as `Resolved reporting window (UTC):`) or manually:

| Placeholder | Formula | Example (`-EndDate 2026-07-09`) |
|---|---|---|
| `<CUR_END>` | `-EndDate` | `2026-07-09` |
| `<CUR_START>` | `<CUR_END> - 7d` | `2026-07-02` |
| `<PREV_START>` | `<CUR_END> - 14d` | `2026-06-25` |
| `<TREND_END>` | `startofweek(<CUR_END>)` — Sunday-aligned | `2026-07-05` |
| `<TREND_START>` | `<TREND_END> - 56d` | `2026-05-10` |

---

## 3. THE distinct-device-count gotcha (most important rule)

`countDevices` on `ErrorStats*` is a **per-row distinct count, not additive**. If you sum it across multiple rows you will double-count any device that appeared in more than one slice. **The dashboard never does this.** Every dashboard query computes devices via:
Expand Down Expand Up @@ -101,7 +137,9 @@ materialized_view('PerfStatsUpdated')

## 7. Week alignment — Kusto `startofweek()` is **Sunday-aligned**

If a user says "the week of May 2 → May 9", Kusto buckets it as `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. **Always confirm**: print the distinct `startofweek(EventInfo_Time)` values from your first query and verify the bucket label matches the user's intent. Off-by-one-week is the #1 silent error.
> **Scope:** only the 60-day trend section (§ 3 of the report / `bucket-trends.js` pipeline) still uses `startofweek()` weekly buckets. The primary/WoW section uses a rolling 7-day window — see § 3 of this cheatsheet.

If a user says "the week of May 2 → May 9", Kusto buckets it as `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When writing weekly-bucketed queries (60-day trend, `wow-table-sparkline-series.kql`), **always confirm**: print the distinct `startofweek(EventInfo_Time)` values from your first query and verify the bucket labels match your intended range. Off-by-one-week is the #1 silent error in weekly-bucket queries.

For an 8-complete-week 60-day window ending Sat May 9, the buckets are:
`2026-03-08, 03-15, 03-22, 03-29, 04-05, 04-12, 04-19, 04-26, 05-03` — that's 9 buckets, one of which (the first) was a partial start. Drop the first; keep 8 complete weeks.
Expand Down Expand Up @@ -158,19 +196,22 @@ materialized_view('ErrorStatsMetrics')
| order by error_code asc, week asc
```

### 8c. Spike attribution — one slicing dim at a time
### 8c. Spike attribution — rolling 7-day WoW form (one slicing dim at a time)

The MCP tool can return ~50–700 KB of JSON; multi-dim cartesians blow this out. **Slice one dimension per query**, then post-process with `summarize-attribution.js`:
The MCP tool can return ~50–700 KB of JSON; multi-dim cartesians blow this out. **Slice one dimension per query**, then post-process with `summarize-attribution.js`. The rolling-window form (see § 3):

```kql
let curEnd = datetime(<CUR_END>);
let curStart = datetime(<CUR_START>);
let prevStart = datetime(<PREV_START>);
let codes = dynamic(['no_tokens_found','unauthorized_client','Code:-6',
'unknown_crypto_error','null_pointer_error','timed_out_execution']);
materialized_view('ErrorStatsMetrics')
| extend unified_account_type = MergeAccountType(account_type)
| extend unified_is_shared_device = MergeIsSharedDevice(is_shared_device)
| where EventInfo_Time > ago(14d)
| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd
| where error_code in (codes)
| extend wk = startofweek(EventInfo_Time)
| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart)
| summarize devs = dcount_hll(hll_merge(countDevicesHll))
by wk, error_code, span_name // <-- swap this dim per query
| order by error_code asc, wk asc, devs desc
Expand Down Expand Up @@ -206,7 +247,7 @@ materialized_view('BrokerAdoptionStatsUpdated')
| [`summarize-attribution.js`](summarize-attribution.js) | Roll up 7-dim attribution slices per (error_code, week) — feeds the spike-attribution cards |
| [`queries/`](queries/) | Canonical KQL templates, one per query — see [`queries/README.md`](queries/README.md) |
| [`templates/`](templates/) | Copy-paste HTML snippets for cards / footer JS |
| [`report-template.html`](../templates/report-template.html) | Canonical layout. Copy to `~/android-oce-reports/oncall-wow-report-<sunday>.html` and replace `{{TOKENS}}` only — never restructure CSS |
| [`report-template.html`](../templates/report-template.html) | Canonical layout. Bootstrap copies it to `~/android-oce-reports/oncall-wow-report-<end-date>.html` and stamps the resolved window into the header. Replace prose/values only — never restructure CSS. |

---

Expand Down
Original file line number Diff line number Diff line change
@@ -1,15 +1,24 @@
// 60-day per-error-code trend.
//
// The 60-day trend section keeps Sun-Sat weekly bucketing (Kusto's
// startofweek() is Sunday-aligned). Only the primary/WoW window is rolling.
//
// Inputs (replace before pasting):
// <START> = first Sunday of the 60d window (e.g. 2026-03-08)
// <END> = end of the reporting week, EXCLUSIVE = next Sunday after the
// reporting week's Sunday (e.g. for a 2026-05-03 report, use 2026-05-10)
// Output: feed to assets/scripts/bucket-trends.js with --start=<START> (no --end needed
// because we filter the partial bucket out at the source — preferred).
// <TREND_START> = first Sunday of the 60d window (e.g. 2026-05-10 for a report
// run on 2026-07-09)
// <TREND_END> = Sunday that OPENS the current in-progress week, EXCLUSIVE.
// Compute as startofweek(today). For a run on 2026-07-09 (Thu),
// use 2026-07-05. The final `where week < datetime(<TREND_END>)`
// drops the partial in-progress bucket at the source.
//
// Output: feed to assets/scripts/bucket-trends.js with --start=<TREND_START>
// (no --end needed because we filter the partial bucket out at the source --
// preferred).
materialized_view('ErrorStatsMetrics')
| where EventInfo_Time between (datetime(<START>) .. datetime(<END>))
| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>))
| where isnotempty(error_code) and error_code != 'success'
| summarize errs = sum(countOverall),
devs = dcount_hll(hll_merge(countDevicesHll))
by week = startofweek(EventInfo_Time), error_code
| where week < datetime(<END>) // drop partial end-week
| where week < datetime(<TREND_END>) // drop partial in-progress week
| order by error_code asc, week asc
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
// 60-day per-error-type trend (with MergeUiRequiredExceptions to collapse variants).
//
// The 60-day trend section keeps Sun-Sat weekly bucketing (Kusto's
// startofweek() is Sunday-aligned). Only the primary/WoW window is rolling.
//
// Inputs (see 60d-trend-codes.kql for detailed semantics):
// <TREND_START> first Sunday of the 60d window
// <TREND_END> startofweek(today), exclusive
materialized_view('ErrorStatsMetrics')
| extend unified_error_type = MergeUiRequiredExceptions(error_type)
| where EventInfo_Time between (datetime(<START>) .. datetime(<END>))
| where EventInfo_Time between (datetime(<TREND_START>) .. datetime(<TREND_END>))
| where isnotempty(unified_error_type)
| summarize errs = sum(countOverall),
devs = dcount_hll(hll_merge(countDevicesHll))
by week = startofweek(EventInfo_Time), unified_error_type
| where week < datetime(<END>)
| where week < datetime(<TREND_END>)
| order by unified_error_type asc, week asc
Original file line number Diff line number Diff line change
@@ -1,34 +1,45 @@
# `assets/queries/` — canonical KQL templates

Each `.kql` here is a paste-and-replace template for one of the queries the OCE
weekly report needs. Token convention:
report needs. The reporting window is a **rolling 7-day window** ending at
start-of-day UTC on `-EndDate` (default: today). See
[`../scripts/bootstrap-report.ps1`](../scripts/bootstrap-report.ps1) for the
canonical window computation, and SKILL.md § "Inputs to confirm" for the
rationale (AB#3683194).

## Placeholder convention

| Token | Meaning |
|---|---|
| `<START>` | Sunday of the earliest week in the window, ISO date e.g. `2026-03-08` |
| `<END>` | Sunday immediately AFTER the reporting week (EXCLUSIVE upper bound). For a 2026-05-03 report use `2026-05-10`. |
| `<PREV_WEEK>` | Sunday of the prior week (the WoW baseline). |
| `<CUR_END>` | Exclusive upper bound of the current 7-day window (e.g. `2026-07-09` on a run at any local time on 2026-07-09 UTC). |
| `<CUR_START>` | `<CUR_END> - 7d` (e.g. `2026-07-02`). Inclusive lower bound of the current window. |
| `<PREV_START>` | `<CUR_END> - 14d` (e.g. `2026-06-25`). Inclusive lower bound of the prior 7-day baseline window. The baseline window is `[PREV_START, CUR_START)`. |
| `<TREND_START>` | First Sunday of the 60-day trend, i.e. `startofweek(CUR_END) - 56d`. **Sunday-aligned.** |
| `<TREND_END>` | Sunday that OPENS the current in-progress week, exclusive: `startofweek(CUR_END)`. On a `2026-07-09` (Thu) run, that's `2026-07-05`. |
| `<CODES_LIST>` | Comma-separated KQL string list, e.g. `'invalid_resource', 'null_pointer_error'` |
| `<TYPES_LIST>` | Same shape but for `unified_error_type`. |
| `<DIM>` | A single column name, replace per dimension run. |
| `<DIM>` | A single column name, replaced per dimension run. |

**The primary/WoW queries emit two rows per key (bucket = `prevStart` or `curStart`).** The JS helpers (`agg.js`, `summarize-attribution.js`) sort the bucket label lexicographically and treat the smaller value as "prev" and the larger as "cur" — so any pair of sortable datetimes works.

**The `<END>` filter is mandatory.** Always include `| where week < datetime(<END>)` after the `summarize` so the partial in-progress week is dropped at the source. Otherwise `bucket-trends.js` will see a fake −99% improvement on every code (the partial bucket will look like a fleet-wide collapse).
**The 60-day trend queries still emit Sun-Sat weekly buckets.** `startofweek()` is Sunday-aligned in Kusto. The `| where week < datetime(<TREND_END>)` filter drops the partial in-progress week at the source, which is required — otherwise `bucket-trends.js` will see a fake −99% improvement on every code.

## File index

| File | Purpose | Section it feeds |
|---|---|---|
| [`reliability-auth-only.kql`](reliability-auth-only.kql) | Per-week auth-only requests/devices | Top-line health, denominator caveat |
| [`broker-version-share.kql`](broker-version-share.kql) | Per-week per-version share — **evidence for denominator caveat** | Denominator caveat callout, broker adoption |
| [`reliability-auth-only.kql`](reliability-auth-only.kql) | Auth-only requests/devices for the current + prior 7-day windows | Top-line health, denominator caveat |
| [`broker-version-share.kql`](broker-version-share.kql) | Per-version share for the WoW window — **evidence for denominator caveat** | Denominator caveat callout, broker adoption |
| [`broker-version-share-wow.kql`](broker-version-share-wow.kql) | Single WoW snapshot of version share — fastest evidence for cohort transitions | Denominator caveat callout |
| [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes | 60-day trend analysis |
| [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types | 60-day trend analysis |
| [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply this week (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the single 🔴 WoW regressions callout** alongside the standard WoW table; tag rows that were absent or near-zero last week with `NEW`. Do not render a separate "emerging" callout. | 🔴 WoW regressions callout (Section 2) |
| [`attr-union-by-dim.kql`](attr-union-by-dim.kql) | **PREFERRED for 2-week WoW.** All 7 dims for N codes (or types) in ONE round-trip; pipe through `summarize-attribution.js --union`. | Spike attribution cards |
| [`attr-codes-by-dim.kql`](attr-codes-by-dim.kql) | Per-dim form (run 7 times). Fall back to this only when the union exceeds payload size or the time window is wider than 2 weeks. | Spike attribution cards |
| [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes (Sun-Sat weekly buckets) | 60-day trend analysis |
| [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types (Sun-Sat weekly buckets) | 60-day trend analysis |
| [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply in the current window (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the single 🔴 WoW regressions callout** alongside the standard WoW table; tag rows that were absent or near-zero in the prior window with `NEW`. Do not render a separate "emerging" callout. | 🔴 WoW regressions callout (Section 2) |
| [`attr-union-by-dim.kql`](attr-union-by-dim.kql) | **PREFERRED for WoW.** All 7 dims for N codes (or types) in ONE round-trip; pipe through `summarize-attribution.js --union`. | Spike attribution cards |
| [`attr-codes-by-dim.kql`](attr-codes-by-dim.kql) | Per-dim form (run 7 times). Fall back to this only when the union exceeds payload size. | Spike attribution cards |
| [`attr-types-by-dim.kql`](attr-types-by-dim.kql) | Per-dim form for type regressions | Spike attribution cards |
| [`type-subcode-decomposition.kql`](type-subcode-decomposition.kql) | 8th dim for type cards | Type spike-attribution cards |
| [`error-message-and-location.kql`](error-message-and-location.kql) | **MANDATORY** for every broker-tagged regression. Now accepts BOTH `<CODES_LIST>` and `<TYPES_LIST>` so codes + types can be sliced in one round-trip. | Code attribution block |
| [`error-message-and-location.kql`](error-message-and-location.kql) | **MANDATORY** for every broker-tagged regression. Accepts BOTH `<CODES_LIST>` and `<TYPES_LIST>` so codes + types can be sliced in one round-trip. | Code attribution block |
| [`os-version-slice.kql`](os-version-slice.kql) | OS / OEM concentration (raw `android_spans`). **On-demand only** per Step 5 — don't slice every card. | OS-version dim in attribution cards (when applicable) |
| [`latency.kql`](latency.kql) | p50/p95/p99 by hot span | Latency section |
| [`app-share.kql`](app-share.kql) | Top calling apps by week | Traffic analysis |
| [`latency.kql`](latency.kql) | p50/p95/p99 by hot span for the WoW window | Latency section |
| [`app-share.kql`](app-share.kql) | Top calling apps for the WoW window | Traffic analysis |
| [`wow-table-sparkline-series.kql`](wow-table-sparkline-series.kql) | 8-week per-code/per-type sparkline series (Sun-Sat weekly buckets) for the WoW table `data-trend` arrays. **MANDATORY** — sparkline arrays must come from real data, never fabricated. | Section 6/7 tables |
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
// Top calling apps share for last N weeks (typically 3).
// Top calling apps share for the WoW window.
//
// Rolling 7-day window. See SKILL.md and bootstrap-report.ps1 for the canonical
// window computation. Two rows per app (wk = prevStart or curStart).
let curEnd = datetime(<CUR_END>);
let curStart = datetime(<CUR_START>);
let prevStart = datetime(<PREV_START>);
materialized_view('AppStatsUpdated')
| where EventInfo_Time between (datetime(<START>) .. datetime(<END>))
| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd
| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart)
| summarize req = sum(countRequests),
dev = dcount_hll(hll_merge(countDevicesHll))
by week = startofweek(EventInfo_Time), calling_package_name
| where week < datetime(<END>)
| order by week asc, req desc
by wk, calling_package_name
| order by wk asc, req desc
| summarize topApps = make_list(pack('app', calling_package_name, 'req', req, 'dev', dev), 25)
by week
| order by week asc
by wk
| order by wk asc
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,20 @@
// Run 7 times with <DIM> set to each of:
// span_name | calling_package_name | active_broker_package_name |
// broker_version | unified_account_type | unified_is_shared_device | client_sku
// (Plus android_spans-based for OS version — see os-version-slice.kql.)
// (Plus android_spans-based for OS version -- see os-version-slice.kql.)
//
// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1 for the
// canonical window computation. Two wk buckets per (code, dim value).
let curEnd = datetime(<CUR_END>);
let curStart = datetime(<CUR_START>);
let prevStart = datetime(<PREV_START>);
let codes = dynamic([<CODES_LIST>]);
materialized_view('ErrorStatsMetrics')
| extend unified_account_type = MergeAccountType(account_type)
| extend unified_is_shared_device = MergeIsSharedDevice(is_shared_device)
| where EventInfo_Time between (datetime(<PREV_WEEK>) .. datetime(<END>))
| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd
| where error_code in (codes)
| extend wk = startofweek(EventInfo_Time)
| where wk < datetime(<END>)
| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart)
| summarize devs = dcount_hll(hll_merge(countDevicesHll)),
errs = sum(countOverall)
by wk, error_code, <DIM>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
// Spike attribution: types by ONE dimension at a time.
// Same usage as attr-codes-by-dim.kql but for error_type regressions.
//
// Rolling 7-day WoW window. See SKILL.md and bootstrap-report.ps1.
let curEnd = datetime(<CUR_END>);
let curStart = datetime(<CUR_START>);
let prevStart = datetime(<PREV_START>);
let types = dynamic([<TYPES_LIST>]);
materialized_view('ErrorStatsMetrics')
| extend unified_error_type = MergeUiRequiredExceptions(error_type)
| extend unified_account_type = MergeAccountType(account_type)
| extend unified_is_shared_device = MergeIsSharedDevice(is_shared_device)
| where EventInfo_Time between (datetime(<PREV_WEEK>) .. datetime(<END>))
| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd
| where unified_error_type in (types)
| extend wk = startofweek(EventInfo_Time)
| where wk < datetime(<END>)
| extend wk = iff(EventInfo_Time >= curStart, curStart, prevStart)
| summarize devs = dcount_hll(hll_merge(countDevicesHll)),
errs = sum(countOverall)
by wk, unified_error_type, <DIM>
Expand Down
Loading
Loading