feat: add scheduled usage metrics report job for UX team (OPS-4148)

[weimiao67]

What changed

Adds the first scheduled job in the repo: a weekly Azure Container App Job that aggregates existing ops_event activity into a usage report and uploads it to Azure Blob Storage for the UX team. The report is delivered as a single two-sheet .xlsx each run:

• an "Aggregate" sheet — per-day × division × role counts (logins, logouts, active users, agreements viewed/edited, BLIs created, etc.), and
• a "Per-user" sheet — each named user who signed in during the reporting window (name, email, division, roles, sign_in_count, last_sign_in_utc).

Privacy note: the per-user sheet names individual users — an explicit, approved requirement change (#4148) that reverses the original counts-only posture. IP addresses are excluded from both sheets. Read access to the reports/ prefix should be granted with the awareness that the .xlsx contains named user data.

• backend/data_tools/src/usage_metrics/utils.py — aggregation module. Reads ops_event rows within a lookback window (SUCCESS-only), attributes each event to a division/role, and buckets counts. aggregate_user_sign_ins builds the per-user view: sign-ins are counted on the resolved actor id only (not fanned out per role) from SUCCESS LOGIN_ATTEMPT events over the same lookback window, falling back to the event_details snapshot for users no longer in the live table. build_workbook renders both sheets with openpyxl directly (no pandas). Uploads the workbook as dated + stable -latest blobs (two blobs), or writes it locally when no storage account is configured.
• backend/data_tools/src/azure_utils/utils.py — adds upload_blob helper (managed identity / RBAC) mirroring the existing download path, with an optional content_type so the .xlsx is uploaded with the spreadsheet MIME type (clean browser download via SAS link).
• backend/data_tools/environment/*.py + types.py — adds usage_metrics_* config properties to DataToolsConfig and each environment (Azure reads env vars; local/dev/test use defaults).
• backend/data_tools/scripts/azure/create_usage_metrics_job.sh — one-time script to create the scheduled job. Configured to match how the real staging jobs run (public ghcr.io image, DB password as a container-app secret, shared user-assigned MI for Blob write). Validates required env vars up front so a missing password fails immediately rather than at cron time.
• backend/data_tools/scripts/usage_metrics.sh — wrapper the job executes.
• .github/workflows/stg_be_build_and_deploy.yml — updates the usage-metrics-job image on each merge to main (guarded so it's a no-op until the job exists).
• backend/data_tools/Pipfile / Pipfile.lock — adds openpyxl (relocked).
• backend/data_tools/tests/usage_metrics/test_utils.py — unit tests for aggregation, attribution, per-user sign-in counting (incl. multi-role no-double-count, failed/out-of-window exclusion, deleted-user fallback), workbook shape, and xlsx-only upload.
• backend/data_tools/CLAUDE.md — "Enable on staging" runbook with verified staging values; documents the .xlsx output and the privacy/semantics notes.

Verified on staging

The job was created and test-fired against staging (opre-ops-stg-app-rg): it aggregated ~4,200 ops_event rows and uploaded the workbook to data/reports/. Two fixes came out of that verification:

• CSV dropped in favor of xlsx-only. The .xlsx "Aggregate" sheet already carries the same per-day × division × role counts the CSV did, so the CSV was redundant. build_csv/generate_report_csv and the CSV uploads are removed; each run now produces two blobs (usage-metrics-latest.xlsx + dated) instead of four.
• create_usage_metrics_job.sh --args fix. The script passed the command as one comma-quoted string ("/bin/ash, -c, ./…"), which az stored as a single argument — the container exited 128 (StartError) with no logs. Now passed as separate tokens (/bin/sh + script path), matching the working data-tools jobs.

Issue

#4148

How to test

Run the unit tests:

cd backend/data_tools
pipenv run pytest tests/usage_metrics/test_utils.py -v

Enabling on staging (see backend/data_tools/CLAUDE.md → "Scheduled Usage Metrics Report"):

export PGUSER=ops PGHOST=opre-ops-stg-db-pg-server.postgres.database.azure.com PGPORT=5432 PGDATABASE=postgres
export PGPASSWORD='<ops DB password — the pgpassword secret on the other staging jobs>'
export USAGE_METRICS_STORAGE_ACCOUNT_URL="https://opreopsstgappsa.blob.core.windows.net"
./scripts/azure/create_usage_metrics_job.sh opre-ops-stg-app-rg storageAccountUser opre-ops-stg-app-cae

# test-fire without waiting for the Monday cron:
az containerapp job start -n usage-metrics-job -g opre-ops-stg-app-rg

Then confirm the report appears at data/reports/usage-metrics-latest.xlsx (the two-sheet workbook with the Aggregate and Per-user sheets).

A11y impact

• No accessibility-impacting changes in this PR

Definition of Done Checklist

• Automated unit tests updated and passed
• Automated integration tests updated and passed
• Test coverage maintained/improved
• Security tests passed
• Form validations updated

[josbell]

Review findings

🔴 Critical — create_usage_metrics_job.sh:76: Bash # comment truncates the az command

The multi-line comment block (lines 76–81) is embedded inside the \-continued az containerapp job create invocation. In bash, when a \<newline> continuation is consumed, # on the next physical line starts a comment that ends at the physical EOL — and since line 76 has no trailing \, the continuation chain stops there. The az command is submitted after --cron-expression "50 4 * * 1" only, missing --args, --environment, --mi-user-assigned, --secrets, and --env-vars. Azure CLI requires --environment and will fail; with set -euo pipefail the script exits immediately and the job is never created. This means the script cannot currently be used to provision the job on any environment.

Fix: Move the comment block above the az containerapp job create call (before line 68), then remove lines 76–81 from inside the continued command.

---

🟠 High — utils.py:197: is_deactivating_update silently excludes all automated bulk deactivations

is_deactivating_update detects a deactivation by checking event_details.get("request.json") for a status field. This only matches manual UI-driven deactivations (where the HTTP request payload is captured under request.json). The automated batch deactivation path in disable_users.py creates UPDATE_USER events with event_details={"user_id": ..., "message": "User deactivated via automated process."} — no request.json key. details.get("request.json") returns None, and the function returns False for every automated deactivation. In a production environment where the weekly disable_users job is the primary deactivation mechanism, the deactivated_users metric will systematically undercount with no error or warning.

Fix: Also check for the automated deactivation format, e.g. look for the message key containing "deactivated", or check event_details.get("user_id") combined with absence of request.json.

---

🟡 Medium — utils.py:366: parse_lookback_days accepts 0, silently producing an empty report

parse_lookback_days validates that the configured value is a parseable integer but does not check > 0. With lookback_days=0, the cutoff is datetime.now() and OpsEvent.created_on >= cutoff matches no historical rows. The job succeeds and uploads a valid .xlsx with column headers but no data rows — no error or warning is raised. An operator who accidentally sets USAGE_METRICS_LOOKBACK_DAYS=0 would receive a blank report from the UX team with no indication the configuration was wrong.

Fix: Add if days <= 0: raise ValueError(f"usage_metrics_lookback_days must be > 0, got {days}") after the int() conversion in parse_lookback_days.

[weimiao67]

Thanks for the thorough review — all three findings were valid and are now fixed in c94cc06.

🔴 #1 — bash comment truncating the az command: Confirmed (reproduced the truncation in a scratch script). Moved the --args explanatory comment block above the az containerapp job create call and removed it from inside the \-continued invocation, so all flags (--args, --environment, --mi-user-assigned, --secrets, --env-vars) are now passed. bash -n passes.

🟠 #2 — automated deactivations undercounted: Fixed at both ends. The disable_users job now emits an explicit top-level "status": UserStatus.INACTIVE.name on its UPDATE_USER event (symmetric with the manual UI path's request.json.status), and is_deactivating_update now reads status from either event_details['request.json'] (manual) or top-level event_details['status'] (automated). Chose the explicit-status approach over string-matching "message" to avoid brittleness. Note: automated events already written to the DB won't be counted until they age out of the lookback window — only events emitted after this deploys carry the new status.

🟡 #3 — parse_lookback_days accepting 0: Added a days <= 0 guard raising ValueError (also catches negatives), so a misconfigured window fails fast instead of silently producing an empty report.

Added unit tests for the automated-format detection and the non-positive lookback rejection. pytest tests/usage_metrics/ tests/disable_users/ → 29 passed.

[jonnalley]

@josbell can you please re-review so we can avoid stale PRs? Thank you!