Skip to content

Establish the seasonal live-client Workshop validation and structured-diff workflow #21

Description

@Teakowa

Parent: #10
Depends on: #18, #19

Goal

Establish a repeatable seasonal validation workflow that takes the canonical Workshop census through the live Overwatch client, records provenance-rich captures, and reports structured semantic/catalog drift against prior evidence.

Current status

The repository-owned/offline portion is implemented and merged via #30: provenance-rich LiveCapture, canonical census/catalog identity linkage, structured diff classification, CLI support, and the documented maintainer procedure are in main.

This issue remains open for the real-client acceptance boundary only. The remaining work is to run the procedure against an actual current Overwatch client, record an en-US baseline and zh-CN locale sweep with pinned provenance, review the resulting feature-attributed structured diff, and preserve those captures as real evidence. No fabricated or synthetic capture should be used to close this issue.

Context

Workshop changes are partly black-box and may not appear in patch notes. Offline catalog/parser tests therefore cannot by themselves establish that the current client still accepts and exports the declared surface. #19 provides deterministic census shards; this issue turns them into repeatable live-client evidence without making live validation the runtime dependency of workshop-rs.

Scope

  • Define the manual/semi-automated workflow for assembling census shards into client-importable probes, importing them into the current Overwatch Workshop client, exporting/copying the resulting Workshop text, and recording the capture.
  • Record capture provenance including game/client version or season identifier where observable, capture date, client locale, catalog identity, census identity/version, and relevant environment notes.
  • Use en-US as the primary semantic baseline and define an explicit zh-CN locale sweep for localization/catalog drift; keep additional locales optional/evidence-driven.
  • Compare new captures with prior evidence using stable Define the Workshop conformance result contract and stable feature identities #18 feature identities and semantic/catalog-aware classifications rather than only whole-text diffs.
  • Classify observed changes such as lexical/locale/content/schema/signature/serialization drift where the evidence supports that distinction; keep runtime-behavior claims separate unless a reproducible runtime experiment exists.
  • Preserve raw client captures as evidence while generating machine-readable structured diff reports suitable for review and later catalog updates.
  • Document the review path from observed drift to a focused canonical catalog/semantic issue or data update; do not silently rewrite expected output to match a new capture.
  • Keep the workflow usable without requiring a network service or runtime online catalog lookup.

Non-goals

  • Automating Overwatch client startup, login, locale switching, or gameplay through unsupported mechanisms.
  • Claiming exhaustive runtime/gameplay correctness from import/export acceptance.
  • Making seasonal client validation mandatory for every normal code PR.
  • Automatically accepting client drift as the new canonical truth without review.
  • Expanding all client locales in the first implementation.

Acceptance criteria

  • A maintainer can follow the documented procedure from Build the canonical sharded Workshop feature census #19 census artifacts to a provenance-recorded live-client capture and structured comparison report.
  • en-US baseline and zh-CN locale sweep produce feature-attributed evidence rather than only raw text snapshots.
  • Prior/new capture differences are classified without conflating locale drift, catalog/content drift, semantic/schema drift, and runtime-only uncertainty.
  • Raw captures and machine-readable reports are reproducible/pinned to census/catalog identities.
  • Observed drift leads to explicit reviewed follow-up work rather than silent expectation updates.
  • The normal workshop-rs runtime remains fully offline with bundled reviewed catalog data.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions