diff --git a/.agents/skills/benchmark-report/SKILL.md b/.agents/skills/benchmark-report/SKILL.md index 55c2ff231..c9861c70a 100644 --- a/.agents/skills/benchmark-report/SKILL.md +++ b/.agents/skills/benchmark-report/SKILL.md @@ -15,7 +15,7 @@ An explicitly requested diagnostic report may be delivered separately from a cer ## Working contract -Repository reports default to Chinese-only output and checks. English generation (including translated illustrations) requires an explicit manual request. Bind generated languages in the report manifest; English navigation opens a not-generated notice when absent. Do not publish stale English pages. Run English-specific validation only for an explicitly generated English edition, retaining full parity checks when both editions exist. +Public report CLIs default to English where supported and expose `--language en|zh`. Preserve the selected language on rebuild. English builds retain the matching Chinese edition; keep measured material in its source language. Historical Chinese-only renderers must disclose that limit and reject unsupported requests. Bind generated languages and the selected entry in report metadata. English navigation opens a not-generated notice when absent; never publish stale English pages. Validate each generated edition and retain route, value and evidence-link parity checks when both exist. 1. Establish the decision being supported, the measured unit, cohort, versions, execution location, freshness policy, and intended deliverable. Inspect the actual dataset, runner, outputs, scorer, prior review evidence, and current report. Do not infer them from screenshots or prose. 2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. For source-dependent tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release. diff --git a/.agents/skills/benchmark-report/references/report-and-release.md b/.agents/skills/benchmark-report/references/report-and-release.md index 2ae72fc91..0501b5a2a 100644 --- a/.agents/skills/benchmark-report/references/report-and-release.md +++ b/.agents/skills/benchmark-report/references/report-and-release.md @@ -20,9 +20,9 @@ A diagnostic instance profile can bind `diagnostic_input.path` relative to the d ## Language generation -All tracks generate and validate Chinese by default. English requires an explicit manual trigger; do not automatically translate prose, generate English illustrations, render English routes or run English-specific checks. A manifest records which languages were generated. When English is absent, retain only a clear not-generated notice with a return link, and test that control. Remove stale English output and unused translated assets from the new bundle while preserving source translations. Parity and English localization checks apply when the English edition is explicitly generated; Chinese-only releases retain all Chinese evidence, semantics, links and visual gates. +Public CLIs default to English where the current renderer supports it and accept `--language en|zh`. Rebuilds preserve the saved choice. English builds retain a matching Chinese edition; language selection never translates measured evidence or changes scores. Historical Chinese-only reports must disclose their limit and reject an English request. Record generated editions and the primary entry in metadata. If English is absent, keep only its not-generated notice with a return link and remove stale English pages and unused assets. Validate each generated edition; bilingual reports require matching routes, values, section order and evidence links. -For the current trajectory report, `make trajectory-replay-report EVIDENCE=/absolute/path/results.json` builds Chinese only. Explicit `ENGLISH=1` (or the build script's `--include-english`) enables English. Report, visual and packaging checks use the generated manifest, so they do not regenerate an absent edition. +For the current trajectory report, `make trajectory-replay-report EVIDENCE=/absolute/path/results.json` defaults to English. Set `REPORT_LANGUAGE=zh` (or the build script's `--language zh`) for Chinese-only output. Report, visual and packaging checks use the generated manifest, so they do not regenerate an absent edition. ## Reader-facing structure diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a3ef67585..5a6de6306 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: with: version: "0.7.8" - run: uv sync --frozen --python "${{ matrix.python }}" --extra test - - run: uv run --frozen --python "${{ matrix.python }}" --no-sync python -m pytest tests/test_portable_cli.py tests/test_cli_scope.py tests/test_web_fetch_quickstart.py -q + - run: uv run --frozen --python "${{ matrix.python }}" --no-sync python -m pytest tests/test_portable_cli.py tests/test_cli_scope.py tests/test_web_fetch_quickstart.py tests/test_public_report_language.py -q test: runs-on: ubuntu-latest steps: diff --git a/AGENTS.md b/AGENTS.md index e120a3869..c1312079e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ These requirements apply to **all** report tracks; browser-specific rules below - A reference, scoring or acceptance correction must fix the shared cause, not patch stored verdicts or one provider. Add paired executable controls to `evaluation/scoring-regressions.json` where both error directions are plausible, run them through the public `score_attempt` and task predicate, and bind the root cause plus regression IDs in every new reference/scoring revision receipt. A revision without this provenance fails before evidence is changed. - A heterogeneous same-window latency cohort keeps every predeclared method-task call. Report the common-denominator all-call distribution separately from each method's conditional successful-call distribution; an all-method common-pass intersection is diagnostic only and must never drive post-outcome task selection or a ranking. - Evaluation credentials come only from the repository `.env`. Artifact Site publishing is the sole user-scoped exception: read explicit `ARTIFACT_SITE_TOKEN`, then `~/.config/artifact-site/tokens/`; the previous `ARTIFACT_HUB_TOKEN` and `~/.config/artifact-hub/` locations are read-only migration fallbacks. Never load `.env` for publishing. Save new authorizations per target host without overwriting another deployment's token. Keep `.env.example` current and publish only account requirements, official acquisition links and variable names. -- All report tracks default to Chinese-only generation and checks. English prose, illustrations, rendering and validation require an explicit manual request. When English is absent, its navigation control must open a not-generated notice with a return link; do not ship stale English pages or unused English-only assets. Preserve translation sources for manual use. Record generated languages in the report manifest. Run release/visual checks for those languages and the not-generated notice. When both editions are generated, their HTML route inventories, section order, tables, values, filters and evidence links must match one-to-one. Keep measured evidence unchanged. +- Public report CLIs default to English where the owning renderer supports it; expose `--language en|zh` and preserve the saved primary language on rebuild. English builds retain the matching Chinese edition. Chinese-only historical renderers must state their limitation and reject an English request rather than relabel untranslated pages. Never infer language from host locale or translate measured evidence. Record generated editions and the selected entry in report metadata. When English is absent, its navigation opens a not-generated notice with a return link; never ship stale English pages. Check the generated languages and notice, and require matching route inventories, section order, tables, values, filters and evidence links when both editions exist. - Web, dynamic, popular-site and document text use shared `score_attempt` orchestration and one task-result predicate. Task-specific requirements live in case configuration, not provider- or track-specific scoring branches. Binary acceptance and text diagnostics remain separate; missing reference coverage is unavailable, not zero or perfect. - Dynamic-content reference capture follows `harness/web_fetch/README.md`: retain paired initial HTTP bytes and independently rendered visible text, bound completion and the full declared scope. Distinguish embedded state from follow-up fetching; never require JavaScript as the only valid route. - Popular-site stable text must be independently observed and task-relevant. Do not freeze volatile ranks, prices, member counts or current list membership. Title diagnostics join returned entities to independent names by detail URL and never substitute for body quality, recall, freshness or task acceptance. diff --git a/Makefile b/Makefile index eb4c300c5..b8f4e7cc2 100644 --- a/Makefile +++ b/Makefile @@ -3,6 +3,7 @@ PY := $(UV) run --frozen python TEST_PY := $(UV) run --frozen --all-extras python WEBMAINBENCH_SAMPLE ?= evaluation/public-webmainbench-200.json WEBMAINBENCH_PROFILE ?= evaluation/current-report.json +REPORT_LANGUAGE ?= DIAGNOSTIC_PROFILE ?= evaluation/live-webfetch/current-report.json .PHONY: setup setup-test import-env runtime-check naming-check test test-python test-ui fixture-smoke ci report report-check popular-sites-report popular-sites-report-check wcxb-release-check resource-image public-benchmark-report public-benchmark-sample webmainbench-run webmainbench-score webmainbench-review webmainbench-assemble browser-use-review browser-use-report browser-use-report-check browser-use-visual-check browser-use-release-check trajectory-replay-report trajectory-replay-report-check trajectory-replay-visual-check trajectory-replay-release-check trajectory-replay-package trajectory-replay-publish webarena-catalog-report webarena-catalog-release-check public-webarena-report public-webarena-report-audit public-webarena-report-release @@ -26,7 +27,7 @@ report-template: runtime-check webfetch-diagnostic-report: runtime-check @test -n "$(OUTPUT)" - $(PY) scripts/build_report.py --profile "$(DIAGNOSTIC_PROFILE)" $(if $(EVIDENCE),--evidence "$(EVIDENCE)") --output "$(OUTPUT)" + $(PY) scripts/build_report.py --profile "$(DIAGNOSTIC_PROFILE)" $(if $(REPORT_LANGUAGE),--language "$(REPORT_LANGUAGE)",) $(if $(EVIDENCE),--evidence "$(EVIDENCE)") --output "$(OUTPUT)" $(PY) scripts/check_diagnostic_report.py "$(OUTPUT)" --profile "$(DIAGNOSTIC_PROFILE)" naming-check: runtime-check @@ -84,7 +85,7 @@ package-check: runtime-check report: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) - $(PY) scripts/build_report.py --evidence "$(EVIDENCE)" --output reports/webfetch $(if $(filter 1,$(ENGLISH)),--include-english,) + $(PY) scripts/build_report.py --evidence "$(EVIDENCE)" --output reports/webfetch $(if $(REPORT_LANGUAGE),--language "$(REPORT_LANGUAGE)",) $(if $(filter 1,$(ENGLISH)),--include-english,) report-check: runtime-check $(PY) scripts/check_report.py reports/webfetch @@ -93,7 +94,7 @@ report-check: runtime-check popular-sites-report: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) - $(PY) scripts/build_report.py --profile evaluation/popular-sites/current-report.json --evidence "$(EVIDENCE)" --output reports/popular-sites $(if $(filter 1,$(ENGLISH)),--include-english,) + $(PY) scripts/build_report.py --profile evaluation/popular-sites/current-report.json --evidence "$(EVIDENCE)" --output reports/popular-sites $(if $(REPORT_LANGUAGE),--language "$(REPORT_LANGUAGE)",) $(if $(filter 1,$(ENGLISH)),--include-english,) popular-sites-report-check: runtime-check $(PY) scripts/check_report.py reports/popular-sites --profile evaluation/popular-sites/current-report.json @@ -107,7 +108,7 @@ wcxb-release-check: runtime-check browser-use-report: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE="/absolute/path [/absolute/path...]" is required' >&2; exit 2) - $(PY) scripts/build_browser_use_report.py $(foreach path,$(EVIDENCE),--evidence "$(path)") --output reports/browser-use $(if $(filter 1,$(ENGLISH)),--include-english,) + $(PY) scripts/build_browser_use_report.py $(foreach path,$(EVIDENCE),--evidence "$(path)") --output reports/browser-use $(if $(REPORT_LANGUAGE),--language "$(REPORT_LANGUAGE)",) $(if $(filter 1,$(ENGLISH)),--include-english,) browser-use-review: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) @@ -126,7 +127,7 @@ browser-use-release-check: browser-use-report-check trajectory-replay-report: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) - $(PY) scripts/build_webarena_replay_report.py --results "$(EVIDENCE)" $(if $(filter 1,$(ENGLISH)),--include-english,) + $(PY) scripts/build_webarena_replay_report.py --results "$(EVIDENCE)" $(if $(REPORT_LANGUAGE),--language "$(REPORT_LANGUAGE)",) $(if $(filter 1,$(ENGLISH)),--include-english,) trajectory-replay-report-check: runtime-check $(PY) scripts/check_webarena_replay_report.py reports/trajectory-replay @@ -166,7 +167,7 @@ resource-image: public-benchmark-report: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path/to/reviewed-public-benchmark-evidence is required' >&2; exit 2) - $(PY) scripts/build_report.py --evidence "$(EVIDENCE)" --output reports/webfetch-public-benchmarks $(if $(filter 1,$(ENGLISH)),--include-english,) + $(PY) scripts/build_report.py --evidence "$(EVIDENCE)" --output reports/webfetch-public-benchmarks $(if $(REPORT_LANGUAGE),--language "$(REPORT_LANGUAGE)",) $(if $(filter 1,$(ENGLISH)),--include-english,) $(PY) scripts/check_report.py reports/webfetch-public-benchmarks $(PY) scripts/check_report_links.py reports/webfetch-public-benchmarks diff --git a/README.md b/README.md index 4fa0303e0..79bf4354d 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,8 @@ Install an extra with `uv sync --frozen --extra NAME`. Browser binaries, Docker, | Output | What you will find | | --- | --- | -| `report/index.html` | Comparisons, task results and evidence links | +| `report/en/index.html` | English comparisons, task results and evidence links | +| `report/index.html` | Report entry; opens the selected language | | `results.jsonl`, `summary.json` | Scores and aggregates for scripts or further analysis | | `manifest.json`, `dataset.json` | Methods, versions, settings and tasks used | | `artifacts/` | Saved responses and other evidence | @@ -98,7 +99,14 @@ Rebuild a report from saved evidence without repeating browser or API calls: uv run --frozen browser-harness-eval report /absolute/path/to/run ``` -Reports default to Chinese. Supported builders can generate English on request; an absent English edition shows a notice instead of stale pages. +The Web Fetch CLI opens English by default, with a switchable Chinese edition. Choose Chinese when starting a run, or change the language of a saved report: + +```bash +uv run --frozen browser-harness-eval run --delay 0 --language zh +uv run --frozen browser-harness-eval report /absolute/path/to/run --language en +``` + +Rebuilding without `--language` keeps the previous choice. Language changes do not repeat calls or change scores; task wording, responses and logs retain their original language. English (`en`) and Chinese (`zh`) are supported; other language codes are rejected. See [report language support](docs/report-hosting.md#report-languages) for the standalone builders and historical reports. To publish online: diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index d5d728e86..1d9bcb08d 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -85,7 +85,8 @@ uv run --frozen browser-harness-eval run --delay 0 | 输出 | 内容 | | --- | --- | -| `report/index.html` | 比较、任务结果和证据入口 | +| `report/en/index.html` | 英文比较、任务结果和证据入口 | +| `report/index.html` | 报告入口，打开所选语言 | | `results.jsonl`、`summary.json` | 供程序分析的评分和汇总 | | `manifest.json`、`dataset.json` | 使用的方法、版本、设置和任务 | | `artifacts/` | 保存的响应及其他证据 | @@ -98,7 +99,14 @@ uv run --frozen browser-harness-eval run --delay 0 uv run --frozen browser-harness-eval report /absolute/path/to/run ``` -报告默认中文。支持英文的构建器可按需生成英文；未生成时，英文入口显示说明，不会展示旧页面。 +Web Fetch 命令默认打开英文报告，并保留可切换的中文版。开始评测时可以选中文，也可以更改已有报告的语言： + +```bash +uv run --frozen browser-harness-eval run --delay 0 --language zh +uv run --frozen browser-harness-eval report /absolute/path/to/run --language en +``` + +重建时不传 `--language`，会沿用上次的选择。切换语言不重新调用浏览器或服务，也不改变评分；任务原文、响应和日志保留原来的语言。目前支持英文（`en`）和中文（`zh`），其他语言代码会报错。独立构建器和历史报告的支持范围见[报告语言说明](report-hosting.md#report-languages)。 在线发布： diff --git a/docs/report-hosting.md b/docs/report-hosting.md index f41bd9d7e..c90a9dbc9 100644 --- a/docs/report-hosting.md +++ b/docs/report-hosting.md @@ -2,6 +2,27 @@ Publish the generated `report/` directory. Keep the rest of the run directory locally so you can rebuild and audit it. +## Report languages + +The Web Fetch CLI defaults to English. `run --language zh` generates Chinese only; `report --language en` rebuilds both editions and opens English. A rebuild without `--language` keeps the saved choice, including Chinese-only reports created by older versions. + +| Builder | Language selection | +| --- | --- | +| `browser-harness-eval run`, `report`, `rescore` | `--language en` or `--language zh`; new runs default to English | +| `scripts/build_report.py`: standard Web Fetch, local audit, WCXB, WebMainBench sample, research companion | `--language en` or `--language zh`; default English | +| `scripts/build_browser_use_report.py` | `--language en` or `--language zh`; default English | +| `scripts/build_webarena_replay_report.py` | `--language en` or `--language zh`; default English | +| Complete 545-case WebMainBench and diagnostic-cohort reports | Chinese-only content; omitted language keeps the historical Chinese default; explicit `en` is rejected | +| Curated replay example, public WebArena recording report and catalog | Chinese-only renderers; no language option | + +Builders keep the saved language when rebuilding without `--language`. Make follows the same rule; set an explicit override through `REPORT_LANGUAGE=en` or `REPORT_LANGUAGE=zh`. + +English builds retain a Chinese edition so readers can switch languages. Chinese-only builds show a notice at the English entry. These choices affect report text and navigation, not scores, timings or saved evidence. Task names, source quotations, responses and logs stay in their original language. + +Upload the entire report directory, including `en/`, assets and evidence. The report root opens the selected language; Chinese links remain usable when English is the default. Serve the directory over HTTP for evidence that loads on demand. + +Only `en` and `zh` are supported. Unsupported codes fail before a run starts or an existing report is replaced. Browser or operating-system language does not silently change a report. To add a language, extend the owning renderer's translation catalog and route, value and evidence-link checks; see [contributing](../CONTRIBUTING.md). + ## 1. Choose a site [Artifact Site](https://github.com/lexmount/artifact-site) hosts multi-page HTML reports and keeps version history. Sign in or create an account on one of these deployments: diff --git a/harness/browser_use/report.py b/harness/browser_use/report.py index 87fb7f37d..ebcdfbeab 100644 --- a/harness/browser_use/report.py +++ b/harness/browser_use/report.py @@ -1422,7 +1422,7 @@ def _page(*, title: str, active: str, body: str, depth: int = 0) -> str: + f">{_e(label)}" for href, label, key in NAV ) - return f'{_e(title)} · Browser Use 评测
{body}

结论只适用于本轮预注册样本、Agent、模型与运行配置；展示调整不改变测量结果。

' + return f'{_e(title)} · Browser Use 评测
{body}

结论只适用于本轮预注册样本、Agent、模型与运行配置；展示调整不改变测量结果。

' def _evidence_viewer_page() -> str: @@ -1468,7 +1468,7 @@ def _english_page(body: str) -> str: ) ) return ( - '' + '' '' 'English edition · Browser Use evaluation' '
', + lambda header: re.sub(r'(]*?\bhref=")([^"]*)(")', chinese_home, header[0]), + text, flags=re.S, + ) + if page == root / "index.html": + redirect = ( + '' + ) + if "" not in text: + raise ValueError("Report homepage has no head element") + text = text.replace("", redirect + "", 1) + page.write_text(text, encoding="utf-8") + path = language_metadata_path(root) + contract = json.loads(path.read_text()) + contract.update(primary_language=language, entrypoint=entry) + path.write_text(json.dumps(contract) + "\n", encoding="utf-8") + return root / entry + + +def generated_languages(root: Path) -> tuple[str, ...]: + path = language_metadata_path(root) if not path.exists(): return ("zh", "en") value = json.loads(path.read_text(encoding="utf-8")) - languages = value.get("languages") + # Older trajectory manifests predate explicit language metadata and are bilingual. + legacy = path.name == "report-manifest.json" and "primary_language" not in value + languages = value.get("languages", ["zh", "en"] if legacy else None) if languages not in (["zh"], ["zh", "en"]): raise ValueError("unsupported generated report languages") return tuple(languages) @@ -50,7 +154,22 @@ def write_language_contract(root: Path, *, include_english: bool) -> None: ) +def check_primary_report_language(root: Path) -> None: + path = language_metadata_path(root) + contract = json.loads(path.read_text()) if path.is_file() else {} + if "primary_language" in contract: + language = primary_report_language(root) + entry = "en/index.html" if language == "en" else "index.html" + if (language not in generated_languages(root) + or contract.get("entrypoint") != entry or not (root / entry).is_file()): + raise ValueError("Primary report language and entrypoint differ from generated output") + redirected = 'data-report-language-entry' in (root / "index.html").read_text() + if redirected != (language == "en"): + raise ValueError("Report root does not open the selected language") + + def check_language_contract(root: Path) -> None: + check_primary_report_language(root) if "en" in generated_languages(root): return files = {p.relative_to(root / "en").as_posix() for p in (root / "en").rglob("*") if p.is_file()} diff --git a/harness/trajectory_replay/report_languages.py b/harness/trajectory_replay/report_languages.py index cc3b28219..97a3c6d91 100644 --- a/harness/trajectory_replay/report_languages.py +++ b/harness/trajectory_replay/report_languages.py @@ -1,16 +1,13 @@ """Languages materialized in a report, independent of available translations.""" -import json from pathlib import Path def report_languages(root: Path) -> tuple[str, ...]: - path = root / "report-manifest.json" - manifest = json.loads(path.read_text()) if path.is_file() else {} - languages = manifest.get("languages", ["zh", "en"]) - if languages not in (["zh"], ["zh", "en"]): - raise ValueError("unsupported report languages") - return tuple(languages) + from harness.report_languages import check_primary_report_language, generated_languages + + check_primary_report_language(root) + return generated_languages(root) def language_roots(root: Path) -> tuple[Path, ...]: diff --git a/harness/trajectory_replay/webarena_report.py b/harness/trajectory_replay/webarena_report.py index 9b387e9e2..31925b615 100644 --- a/harness/trajectory_replay/webarena_report.py +++ b/harness/trajectory_replay/webarena_report.py @@ -564,7 +564,7 @@ def _shell(lang: str, active: str, title: str, body: str, *, depth: int = 0, cur css.href=css.dataset.src; })();''' body = body.replace('{_e(title)}{preview_bootstrap}{_nav(lang, active, depth=depth, current_route=current_route)}
{body}
{footer}
''' + return f'''{_e(title)}{preview_bootstrap}{_nav(lang, active, depth=depth, current_route=current_route)}
{body}
{footer}
''' def _winner(value: str, won: bool, *, lang: str) -> str: @@ -1396,7 +1396,13 @@ def build_webarena_replay_report( interpretation_path: str | Path | None = None, interpretation_audit_path: str | Path | None = None, include_english: bool = False, + language: str | None = None, ) -> dict[str, Any]: + from harness.report_languages import ( + includes_english, + select_report_language, + ) + include_english = includes_english(language, include_english) languages = ("zh", "en") if include_english else ("zh",) profile_source = Path(profile_path) profile = json.loads(profile_source.read_text(encoding="utf-8")) @@ -2091,7 +2097,7 @@ def build_webarena_replay_report( english_root = output / "en" english_root.mkdir() (english_root / "index.html").write_text( - '' + '' '' '' @@ -2130,6 +2136,9 @@ def build_webarena_replay_report( if profile.get("measurement_notes") else {}, } _write_json(output / "report-manifest.json", manifest) + if language is not None: + entry = select_report_language(output, language) + manifest.update(primary_language=language, entrypoint=entry.relative_to(output).as_posix()) return manifest @@ -2189,7 +2198,7 @@ def _catalog_shell(title: str, body: str, participant: str) -> str: '方法与边界' ) return ( - '' + '' '' f"{html.escape(title)}{nav}{body}" f"
WebArena-Verified 全量轨迹重放 · {html.escape(participant)}
" @@ -2628,6 +2637,8 @@ def build_main(argv=None) -> None: action="store_true", help="Explicitly generate the English report and its assets", ) + from harness.report_languages import add_language_argument, primary_report_language + add_language_argument(parser) args = parser.parse_args(argv) profile = json.loads(args.profile.read_text(encoding="utf-8")) dataset_path = args.dataset or bound_dataset_path(profile) @@ -2648,6 +2659,7 @@ def build_main(argv=None) -> None: profile_path=args.profile, output_dir=args.output, include_english=args.include_english, + language=args.language or primary_report_language(Path(args.output)), interpretation_path=content / "trajectory-replay-interpretation.json", interpretation_audit_path=( args.interpretation_audit diff --git a/harness/web_fetch/README.md b/harness/web_fetch/README.md index d1e20e5fd..f75241140 100644 --- a/harness/web_fetch/README.md +++ b/harness/web_fetch/README.md @@ -709,11 +709,11 @@ new scores or timings; local fixture validation is not a benchmark rerun. ## WCXB paired timing and publication -WCXB builds generate Chinese pages by default. Their English navigation opens a -not-generated notice with a return link. Pass `--include-english` to -`scripts/build_report.py --wcxb` to generate both editions; the release checks -then require matching route inventories and values. English repair content is -required and packaged only for that explicit bilingual build. +The WCXB command defaults to English, with a matching Chinese edition. Use +`--language zh` for Chinese-only output; its English entry shows a not-generated +notice. Bilingual release checks require matching routes and values. English +repair content must be present in the bound asset set for an English build. +See [report languages](../../docs/report-hosting.md#report-languages) for all builders. `wcxb_latency.py` owns paired-call collection, receipt validation and certification. The collector uses the profile's complete qualified population, timeout and binary diff --git a/harness/web_fetch/cli.py b/harness/web_fetch/cli.py index f5ae69d67..acd6be5ed 100644 --- a/harness/web_fetch/cli.py +++ b/harness/web_fetch/cli.py @@ -12,6 +12,7 @@ from .report_build import build_report from .runner import rescore_evaluation, run_evaluation from .storage import data_home +from harness.report_languages import add_language_argument, primary_report_language EXAMPLES = Path(__file__).parent / "data" @@ -50,7 +51,8 @@ def command_run(args: argparse.Namespace) -> int: provider_concurrency=args.provider_concurrency, max_failure_retries=args.max_failure_retries, ) - report_path = build_report(args.output, args.output / 'report', diagnostic_preview=True) + report_path = build_report(args.output, args.output / 'report', diagnostic_preview=True, + language=args.language) print(f"Report: {report_path.resolve()}") kinds={case.id:case.task_kind for case in cases} failures = [r for r in records if not task_passed(r['ok'],r['metrics'],kinds[r['case_id']])] @@ -58,7 +60,9 @@ def command_run(args: argparse.Namespace) -> int: def command_report(args: argparse.Namespace) -> int: - report_path = build_report(args.run_dir, args.run_dir / 'report', diagnostic_preview=True) + language = args.language or primary_report_language(args.run_dir / 'report') + report_path = build_report(args.run_dir, args.run_dir / 'report', diagnostic_preview=True, + language=language) print(f"Report: {report_path.resolve()}") return 0 @@ -66,7 +70,9 @@ def command_report(args: argparse.Namespace) -> int: def command_rescore(args: argparse.Namespace) -> int: dataset, cases = load_dataset(args.dataset) records, _ = rescore_evaluation(run_dir=args.run_dir, dataset_meta=dataset, dataset_path=args.dataset, cases=cases) - report_path = build_report(args.run_dir, args.run_dir / 'report', diagnostic_preview=True) + language = args.language or primary_report_language(args.run_dir / 'report') + report_path = build_report(args.run_dir, args.run_dir / 'report', diagnostic_preview=True, + language=language) print(f"Rescored {len(records)} persisted attempts; report: {report_path.resolve()}") return 0 @@ -105,6 +111,7 @@ def build_parser() -> argparse.ArgumentParser: sub = parser.add_subparsers(dest="command", required=True) run = sub.add_parser("run", help="Run providers and generate a multi-page report") + add_language_argument(run, default="en") run.add_argument("--dataset", type=Path, default=EXAMPLES / "fixture-dataset.json") run.add_argument("--provider-config", type=Path, default=EXAMPLES / "fixture-providers.json") run.add_argument("--providers", default="fixture", help="Comma-separated configured provider names") @@ -131,11 +138,13 @@ def build_parser() -> argparse.ArgumentParser: report = sub.add_parser("report", help="Rebuild HTML from an existing run directory") report.add_argument("run_dir", type=Path) + add_language_argument(report) report.set_defaults(handler=command_report) rescore = sub.add_parser("rescore", help="Recompute metrics/report from persisted artifacts and task rules") rescore.add_argument("run_dir", type=Path) rescore.add_argument("--dataset", type=Path, required=True) + add_language_argument(rescore) rescore.set_defaults(handler=command_rescore) validate = sub.add_parser("validate", help="Validate dataset and provider credentials without network calls") diff --git a/harness/web_fetch/report.py b/harness/web_fetch/report.py index 005a3c7e2..de5cd981f 100644 --- a/harness/web_fetch/report.py +++ b/harness/web_fetch/report.py @@ -123,7 +123,7 @@ def page(lang,path,title,body): root_prefix='../'*route.count('/') nav=''.join(f'{esc(x[k])}' for k,p in [('home','index.html'),('dataset','datasets/index.html'),('method','methodology/index.html'),('results','analysis/index.html'),('review','review/index.html'),('guide','guide/index.html'),('adoption','adoption/index.html'),('repro','reproduce/index.html'),('changes','changes/index.html'),('accounts','accounts/index.html')]) other=('en/'+path if include_english else 'en/index.html') if lang=='zh' else path - head=f'{esc(title)} · WCXB
Web Fetch{"English" if lang=="zh" else "中文"}
{body}
WCXB v1.0 · {case_count} {"页评测样本" if lang=="zh" else "evaluated test pages"} · CC BY 4.0 · {esc(x["source"])}
' + head=f'{esc(title)} · WCXB
Web Fetch{"English" if lang=="zh" else "中文"}
{body}
WCXB v1.0 · {case_count} {"页评测样本" if lang=="zh" else "evaluated test pages"} · CC BY 4.0 · {esc(x["source"])}
' target=out/('en' if lang=='en' else '')/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(head,encoding='utf-8') def fmt(v,digits=1):return f'{v*100:.{digits}f}%' diag_labels={ @@ -727,7 +727,7 @@ def write(out,path,title,body,*,root_links=False): counterpart=(base+'en/'+path if _ENGLISH_GENERATED else base+'en/index.html') language=f'English' footer='所有分数由本轮调用记录计算。网页与服务可能变化；结论仅适用于本轮样本和配置。' - page=f'{document_base}{esc(title)} · Web Fetch
{_RELIABILITY_BANNER}{body}

{footer}

' + page=f'{document_base}{esc(title)} · Web Fetch
{_RELIABILITY_BANNER}{body}

{footer}

' target=contained_path(out,path);target.parent.mkdir(parents=True,exist_ok=True);target.write_text(page,encoding='utf-8') _RESEARCH_ROUTES=[ @@ -916,7 +916,7 @@ def _research_write(out,path,title,body,english=False): language=f'{"中文" if english else "English"}' status=('Research edition · 0 measured results' if english else '调研版 · 0 条实测结果') footer=('Research was verified from public papers, repositories, datasets and scorers. No benchmark result is claimed in this edition.' if english else '调研依据为公开论文、仓库、数据和评分器；本版不声称任何实测成绩。') - page=f'{esc(title)} · Web Fetch
{status}
{body}

{footer}

' + page=f'{esc(title)} · Web Fetch
{status}
{body}

{footer}

' target=contained_path(out,local_path);target.parent.mkdir(parents=True,exist_ok=True);target.write_text(page,encoding='utf-8') def generate_public_benchmark_research(out:Path,research:dict,*,include_english:bool=False): @@ -2598,7 +2598,7 @@ def _wmb_write(out,path,title,body,english=False): counterpart=base+path if english else base+'en/'+path language=f'{"中文" if english else "English"}' footer=('Scores apply to the frozen inputs and methods declared in this report. They do not represent the live web or the official WebMainBench leaderboards.' if english else '分数适用于本报告声明的冻结输入和方法，不代表实时网页或官方 WebMainBench 榜单。') - page=f'{esc(title)} · Web Fetch
{body}

{footer}

' + page=f'{esc(title)} · Web Fetch
{body}

{footer}

' page=page.replace(f'href="{base}assets/style.css"',f'href="{base}assets/style.css?v={hashlib.sha256((STYLE+EXTRA).encode()).hexdigest()[:16]}"').replace(f'src="{base}assets/report.js"',f'src="{base}assets/report.js?v={hashlib.sha256(JS.encode()).hexdigest()[:16]}"') target=contained_path(out,local);target.parent.mkdir(parents=True,exist_ok=True);target.write_text(page,encoding='utf-8') @@ -3320,7 +3320,7 @@ def table(headers, rows, sort_types=None, table_id=None): def page(route, title, body): prefix = "../" * route.count("/") links = ''.join(anchor(path, label) for label, path in navigation) - markup = ('' + esc(title) + ' · WebMainBench 545
' + anchor("index.html", "WebMainBench · 545") + '' + anchor("en/index.html", "English") + '
' + body + diff --git a/harness/web_fetch/report_build.py b/harness/web_fetch/report_build.py index ff9286246..bb543e4da 100644 --- a/harness/web_fetch/report_build.py +++ b/harness/web_fetch/report_build.py @@ -139,8 +139,11 @@ def copy_methodology_assets(output, assets=None): -def build_public_benchmark_research(source: Path, output: Path, *, include_english: bool = False) -> Path: +def build_public_benchmark_research(source: Path, output: Path, *, include_english: bool = False, + language: str | None = None) -> Path: """Build the research-only companion without measured evidence.""" + from harness.report_languages import includes_english, select_report_language + include_english = includes_english(language, include_english) source, output = source.resolve(), output.resolve() marker = '.generated-report' if output.exists() and not (output / marker).is_file(): @@ -166,6 +169,8 @@ def build_public_benchmark_research(source: Path, output: Path, *, include_engli generate_public_benchmark_research(staging, research, include_english=include_english) if sample is not None: shutil.copy2(sample_path, staging / 'sample-manifest.json') + if language is not None: + select_report_language(staging, language) (staging / marker).write_text('generated\n', encoding='utf-8') backup = output.with_name(output.name + '.previous') if backup.exists(): @@ -178,7 +183,7 @@ def build_public_benchmark_research(source: Path, output: Path, *, include_engli except Exception: shutil.rmtree(staging, ignore_errors=True) raise - return output / 'index.html' + return output / ('en/index.html' if language == 'en' else 'index.html') def _install_report_tree(staging: Path, output: Path): @@ -198,7 +203,9 @@ def _install_report_tree(staging: Path, output: Path): def build_report(source: Path, output: Path, *, include_english: bool = False, local_audit: bool = False, authored_interpretation: bool = False, - diagnostic_preview: bool = False) -> Path: + diagnostic_preview: bool = False, language: str | None = None) -> Path: + from harness.report_languages import includes_english, select_report_language + include_english = includes_english(language, include_english) source, output = source.resolve(), output.resolve() if source == output or output in source.parents: raise ValueError('Report output must not contain its evidence source') @@ -212,7 +219,8 @@ def build_report(source: Path, output: Path, *, include_english: bool = False, from .reference_lifecycle import require_certifiable_evidence require_certifiable_evidence(dataset_probe) if dataset_probe.get('comparison', {}).get('kind') == 'webmainbench_frozen_html_to_markdown': - return build_webmainbench_report(source, output, include_english=include_english) + return build_webmainbench_report(source, output, include_english=include_english, + language=language) if not local_audit: require_selection_history(source) dataset = json.loads(contained_path(source,'dataset.json').read_text()) @@ -495,12 +503,14 @@ def build_report(source: Path, output: Path, *, include_english: bool = False, }, ensure_ascii=False, indent=2)) generate(staging,dataset,records,manifests,include_english=include_english, authored_interpretation=authored_interpretation) + if language is not None: + select_report_language(staging, language) (staging/marker).write_text('Generated from persisted evidence. Rebuild; do not edit.\n') # Replace only an explicitly marked generated directory; never merge stale pages. _install_report_tree(staging, output) finally: if staging.exists(): shutil.rmtree(staging) - return output/'index.html' + return output/('en/index.html' if language == 'en' else 'index.html') def validate_webmainbench_correction(receipt: dict, dataset: dict) -> None: @@ -577,8 +587,11 @@ def validate_webmainbench_anomaly_review(receipt: dict, source: Path, raise ValueError('Diagnostic review must not mutate score-bearing evidence') -def build_webmainbench_report(source: Path, output: Path, *, include_english: bool = False) -> Path: +def build_webmainbench_report(source: Path, output: Path, *, include_english: bool = False, + language: str | None = None) -> Path: """Build the current public-benchmark report without legacy task predicates.""" + from harness.report_languages import includes_english, select_report_language + include_english = includes_english(language, include_english) from harness.reporting.assets import copy_bound_assets, load_bound_content authority = Path(__file__).resolve().parents[2] @@ -677,6 +690,8 @@ def build_webmainbench_report(source: Path, output: Path, *, include_english: bo content_binding['relative_path']: content_binding['sha256']}) shutil.copy2(content_binding_path, staging / 'report-content-binding.json') generate_webmainbench(staging, dataset, records, summaries, manifests, review, research, corrections, adoption, fixture_correction, teds_revision, anomaly_review, latency_diagnostic, include_english=include_english, report_content=report_content) + if language is not None: + select_report_language(staging, language) (staging / '.generated-report').write_text('Generated from persisted evidence. Rebuild; do not edit.\n') if output.exists(): backup = Path(tempfile.mkdtemp(prefix='.report-swap-', dir=output.parent)) / 'report' @@ -692,7 +707,7 @@ def build_webmainbench_report(source: Path, output: Path, *, include_english: bo finally: if staging.exists(): shutil.rmtree(staging) - return output / 'index.html' + return output / ('en/index.html' if language == 'en' else 'index.html') def public_webmainbench_scoring_projection(records: list[dict]) -> list[dict]: @@ -927,8 +942,10 @@ def build_webmainbench_full_report(source: Path, output: Path) -> Path: def build_wcxb_report(source: Path, corpus: Path, sample_path: Path, profile: dict, output: Path, *, content_binding: dict, authority_root: Path, - include_english: bool = False) -> Path: + include_english: bool = False, language: str | None = None) -> Path: """Build the one current report from a fully certified WCXB matrix.""" + from harness.report_languages import includes_english, select_report_language + include_english = includes_english(language, include_english) from harness.web_fetch.wcxb_dataset import validate_manifest from harness.web_fetch.webmainbench_runner import WCXB_PROVIDERS, load_wcxb_source source, output = source.resolve(), output.resolve() @@ -1128,6 +1145,8 @@ def build_wcxb_report(source: Path, corpus: Path, sample_path: Path, profile: di for folder in ('artifacts', 'references', 'inputs'): shutil.rmtree(staging / folder, ignore_errors=True) (staging / '.generated-report').write_text('generated\n') + if language is not None: + select_report_language(staging, language) from .wcxb_publication import write_inventory write_inventory(staging, profile) if output.exists(): @@ -1143,7 +1162,7 @@ def build_wcxb_report(source: Path, corpus: Path, sample_path: Path, profile: di except BaseException: shutil.rmtree(staging, ignore_errors=True) raise - return output / 'index.html' + return output / ('en/index.html' if language == 'en' else 'index.html') def validate_local_presentation(output: Path, binding_path: Path): diff --git a/harness/web_fetch/report_i18n.py b/harness/web_fetch/report_i18n.py index ba0acf0de..63ccbef7c 100644 --- a/harness/web_fetch/report_i18n.py +++ b/harness/web_fetch/report_i18n.py @@ -37,6 +37,29 @@ # Exact editorial translations take priority. Dynamic evidence (task names, # references and provider output) is deliberately absent from this catalog. TEXT = { + "查看本报告的任务范围、结果数量与验收规则。": "See the tasks, result counts and acceptance rules used in this report.", + "内容选择": "Content selection", + "典型处理流程": "Typical processing flow", + "失败环节": "Failure stage", + "失败类别": "Failure category", + "搜索": "Search", + "搜索案例、方法或原因": "Search tasks, methods or causes", + "类别": "Category", + "示例说明 · Web Fetch": "Example notes · Web Fetch", + "未开启 debug": "Debug logging disabled", + "未设置": "Not configured", + "云端 API 后端未公开版本": "Cloud API backend version not disclosed", + "使用本轮任务配置与方法配置运行，或从保存的结果重新构建报告：": "Run with this task and method configuration, or rebuild the report from saved results:", + "重新运行会调用配置中的方法；重建报告只读取已有结果。本轮参数、版本与可执行文件哈希见上表。": "Rerunning calls the configured methods. Rebuilding reads saved results only. Parameters, versions and executable hashes are listed above.", + "至少需要两种方法才能比较任务交集。": "At least two methods are needed to compare shared tasks.", + "· 未保存源页快照": "· No source-page snapshot saved", + "本轮方法": "Methods in this run", + "运行前准备配置对应的程序与账户。报告保留方法名称、版本和执行证据，不包含凭据值。实际参数与版本见": "Install the configured programs and set up any required accounts before running. The report saves method names, versions and execution evidence, without credential values. See parameters and versions in", + "方法 SHA-256": "Method SHA-256", + "这次物理调用没有保存可展示的 stderr / debug 日志；不会补造不存在的日志。": "No displayable stderr or debug log was saved for this physical call.", + "下载最终物理调用记录": "Download the final physical-call record", + "。运行记录保留请求状态、耗时和证据路径；原始输出保持调用后保存的字节。": ". The record retains request status, duration and evidence paths. Raw output preserves the saved response bytes.", + "请求成功，并同时满足本例最低字符数、全部必需核对点、显著标记及结构要求；不得命中非参考正文自身包含的错误页标记。完整度、纯净度和 F1 另算，不作为通过门槛。": "A successful request must satisfy the task's minimum length, all required text checks, salient markers and structure requirements. Error-page markers absent from the reference must not appear. Recall, precision and F1 are calculated separately and do not determine acceptance.", "已保存的诊断日志": "Saved diagnostic logs", "已保存文件截断；原生日志范围见说明": "Saved file truncated; see native log coverage notes", "已保存文件未截断；原生日志范围见说明": "Saved file not truncated; see native log coverage notes", @@ -2929,10 +2952,9 @@ "。托管服务按官方文档；内置工具按固定源码。此处说明可选能力，": ". Managed services follow official documentation, while built-in tools follow pinned source. These are available capabilities, ", }) -def _register_release_translations() -> None: - """Keep release prose with the shared history instead of an exact-string code map.""" - path = Path(__file__).parents[2] / 'research/report-changes.json' - history = json.loads(path.read_text(encoding='utf-8')) +def release_translations(history: dict) -> dict[str, str]: + """Localize only the structured release records included in this report.""" + translations = {} for release in history['releases']: if any(kind in release.get('comparison_kinds', []) for kind in ( 'webmainbench_frozen_html_to_markdown', 'public_benchmark_research', @@ -2944,14 +2966,13 @@ def _register_release_translations() -> None: or len(release.get('changes', [])) != len(english.get('changes', []))): raise ValueError('Web Fetch release requires complete English text: ' + release['id']) for key in ('title', 'summary'): - TEXT[release[key]] = english[key] + translations[release[key]] = english[key] for source_row, target_row in zip(release['changes'], english['changes'], strict=True): if len(source_row) != 4 or len(target_row) != 4: raise ValueError('Web Fetch release change rows must have four columns: ' + release['id']) - TEXT.update(zip(source_row, target_row, strict=True)) + translations.update(zip(source_row, target_row, strict=True)) - -_register_release_translations() + return translations REGEX.insert(0, ( @@ -3390,13 +3411,21 @@ def _register_release_translations() -> None: (re.compile(r"^每种方法均使用同一组 (\d+) 个任务，包含通过、未通过、错误和超时。耗时是单次调用从输入到返回的时间，不包含调用之间的统一冷却间隔。各方法串行执行，仍受本地与远程环境、网络和产品内部限制影响。$"), r"Each method uses the same \1 tasks, including passes, failures, errors, and timeouts. Latency runs from input to return for a single call, excluding shared cooldown intervals. Each method runs serially and remains subject to local and remote environments, networks, and product limits."), ] -def translate_text(value: str) -> str: +REGEX.extend([ + (re.compile(r"^本报告纳入 ([\d,]+) 个逻辑结果。运行参数见下表；保留的物理调用与输出见各结果详情。$"), + r"This report includes \1 logical results. Run parameters are listed below; each result detail contains its retained physical calls and outputs."), + (re.compile(r"^· 请求：(成功|失败) · 耗时：([\d,.]+) ms · 实际输出：([\d,]+) 字符$"), + lambda m: f"· Request: {'success' if m[1] == '成功' else 'failure'} · Duration: {m[2]} ms · Actual output: {m[3]} characters"), +]) + + +def translate_text(value: str, translations=None) -> str: leading = value[:len(value) - len(value.lstrip())] trailing = value[len(value.rstrip()):] core = value.strip() if not core: return value - translated = TEXT.get(core) + translated = (translations or {}).get(core, TEXT.get(core)) if translated is None: for marker in (" ↕", " ↑", " ↓"): if core.endswith(marker) and core[:-len(marker)] in TEXT: @@ -3413,6 +3442,10 @@ def translate_text(value: str) -> str: r"Complete-reference coverage: \1/\2. A missing reference affects only full-text metrics, not task acceptance. ", core, ) + translated = re.sub( + r"文字核对范围：各方法均为 ([\d,]+/[\d,]+) 例。", + r"Text-check coverage: \1 cases for every method. ", translated, + ) for source, target in sorted(PHRASES.items(), key=lambda item: len(item[0]), reverse=True): translated = translated.replace(source, target) if translated == core: @@ -3421,11 +3454,12 @@ def translate_text(value: str) -> str: class Localizer(HTMLParser): - def __init__(self, source: PurePosixPath | None = None) -> None: + def __init__(self, source: PurePosixPath | None = None, translations=None) -> None: super().__init__(convert_charrefs=False) self.parts: list[str] = [] self.stack: list[str] = [] self.source = source + self.translations = translations def _start_tag(self, attrs) -> str: original = self.get_starttag_text() @@ -3433,8 +3467,6 @@ def _start_tag(self, attrs) -> str: return original values = dict(attrs) article = self.source in {PurePosixPath("guide/index.html"), PurePosixPath("guide/article.html")} - if not article and "data-i18n-attrs" not in values: - return original def replace_attribute(markup, name, value): pattern = re.compile(r'''(? None: @@ -3478,7 +3510,7 @@ def handle_data(self, data: str) -> None: if any(tag in self.stack for tag in ("pre", "code", "script", "style")): self.parts.append(data) else: - self.parts.append(translate_text(data)) + self.parts.append(translate_text(data, self.translations)) def handle_entityref(self, name: str) -> None: self.parts.append(f"&{name};") @@ -3584,11 +3616,11 @@ def _header(source: PurePosixPath, english: PurePosixPath) -> str: '') -def localize_html(page: str, source: PurePosixPath) -> str: +def localize_html(page: str, source: PurePosixPath, *, translations=None) -> str: english = PurePosixPath("en") / source page = page.replace('', '', 1) page = re.sub(r'
.*?
', _header(source, english), page, count=1, flags=re.S) - parser = Localizer(source) + parser = Localizer(source, translations) parser.feed(page) parser.close() return _rewrite_links("".join(parser.parts), source, english) @@ -3596,6 +3628,8 @@ def localize_html(page: str, source: PurePosixPath) -> str: def write_english_report(out: Path) -> int: """Mirror every Chinese HTML route and return the mirrored route count.""" + history_path = out / "report-changes.json" + translations = release_translations(json.loads(history_path.read_text())) if history_path.is_file() else {} english_root = out / "en" if english_root.exists(): shutil.rmtree(english_root) @@ -3611,7 +3645,7 @@ def write_english_report(out: Path) -> int: source = PurePosixPath(source_path.relative_to(out).as_posix()) target = out / "en" / Path(str(source)) target.parent.mkdir(parents=True, exist_ok=True) - localized = localize_html(source_path.read_text(encoding="utf-8"), source) + localized = localize_html(source_path.read_text(encoding="utf-8"), source, translations=translations) if source in {PurePosixPath("guide/index.html"), PurePosixPath("guide/article.html")}: untranslated = untranslated_english_narrative(localized, allowed_source_text) if untranslated: diff --git a/scripts/build_report.py b/scripts/build_report.py index f5d4552a1..484c365b7 100644 --- a/scripts/build_report.py +++ b/scripts/build_report.py @@ -3,6 +3,7 @@ from pathlib import Path ROOT=Path(__file__).resolve().parents[1] sys.path.insert(0,str(ROOT)) +from harness.report_languages import add_language_argument, resolve_report_language, SUPPORTED_REPORT_LANGUAGES from harness.web_fetch.report_build import build_report, build_public_benchmark_research, build_wcxb_report, build_webmainbench_full_report, build_diagnostic_cohorts, refresh_local_presentation from harness.web_fetch.storage import ( current_evidence_dir, @@ -18,8 +19,9 @@ help='current-report profile under evaluation/; requires an explicit output unless the profile declares report_output') parser.add_argument('--public-benchmark-research',type=Path) parser.add_argument('--output',type=Path) + add_language_argument(parser) parser.add_argument('--include-english',action='store_true', - help='also generate the English report; Chinese-only is the default') + help='generate both editions; --language selects the homepage') parser.add_argument('--refresh-presentation', type=Path, help='refresh an existing local report using a hash-bound current-local-results.json; preserve measured evidence') parser.add_argument('--local-audit',action='store_true', @@ -34,10 +36,16 @@ from harness.reporting.templates import resolve_template resolve_template('webfetch') if args.refresh_presentation: - if not args.output or args.evidence or args.profile or args.local_audit or args.include_english or args.diagnostic_cohorts or args.wcxb or args.webmainbench_full or args.public_benchmark_research: + if not args.output or args.evidence or args.profile or args.local_audit or args.include_english or args.language or args.diagnostic_cohorts or args.wcxb or args.webmainbench_full or args.public_benchmark_research: parser.error('--refresh-presentation requires only an existing --output and its result binding') print(refresh_local_presentation(args.output, args.refresh_presentation)) raise SystemExit(0) + def requested_language(output, supported=SUPPORTED_REPORT_LANGUAGES): + try: + return resolve_report_language(output, args.language, supported=supported) + except ValueError as error: + parser.error(str(error)) + if args.profile: args.profile=(ROOT/args.profile).resolve() if not args.profile.is_absolute() else args.profile.resolve() if not args.profile.is_relative_to((ROOT/'evaluation').resolve()): @@ -49,6 +57,7 @@ parser.error('diagnostic build requires --output and evidence or a bound profile') if args.local_audit or args.wcxb or args.webmainbench_full or args.public_benchmark_research or args.include_english: parser.error('--diagnostic-cohorts requires its own diagnostic evidence model and Chinese output') + requested_language(args.output, supported=('zh',)) print(build_diagnostic_cohorts(args.evidence,args.output,profile=args.profile)) raise SystemExit(0) if args.local_audit: @@ -58,15 +67,17 @@ parser.error('--local-audit cannot be combined with a release profile, WCXB, WebMainBench, or public benchmark research') evidence=require_evidence(args.evidence) print(build_report(evidence,args.output, - include_english=args.include_english, + include_english=args.include_english, language=requested_language(args.output), local_audit=True)) raise SystemExit(0) if sum(bool(value) for value in (args.profile, args.wcxb, args.webmainbench_full)) > 1: parser.error('--profile, --wcxb and --webmainbench-full are mutually exclusive') if args.webmainbench_full: if not args.evidence or args.include_english: - parser.error('--webmainbench-full requires --evidence and generates Chinese only') - print(build_webmainbench_full_report(args.evidence,args.output or ROOT/'reports/webmainbench-full')) + parser.error('--webmainbench-full requires --evidence; this historical report has no English edition') + output=args.output or ROOT/'reports/webmainbench-full' + requested_language(output, supported=('zh',)) + print(build_webmainbench_full_report(args.evidence,output)) raise SystemExit(0) if args.public_benchmark_research: if args.evidence: @@ -75,7 +86,7 @@ parser.error('The independent public-benchmark report now contains reviewed measurements; a research-only draft requires an explicit separate --output') output=args.output or ROOT/'reports/webfetch-public-benchmarks' print(build_public_benchmark_research( - args.public_benchmark_research,output,include_english=args.include_english)) + args.public_benchmark_research,output,include_english=args.include_english, language=requested_language(output))) raise SystemExit(0) if args.profile: profile_path=args.profile @@ -91,7 +102,8 @@ sample=args.wcxb_sample or ROOT/profile['sample_manifest'] print(build_wcxb_report(args.evidence,corpus,sample,profile,args.output or ROOT/'reports/wcxb', content_binding=json.loads(profile_path.with_name('report-content-binding.json').read_text()), - authority_root=ROOT, include_english=args.include_english)) + authority_root=ROOT, include_english=args.include_english, + language=requested_language(args.output or ROOT/'reports/wcxb'))) raise SystemExit(0) evidence=require_evidence(args.evidence or current_evidence_dir()) if profile.get('comparison_kind')!='webmainbench_frozen_html_to_markdown': @@ -104,5 +116,5 @@ parser.error('a custom --profile requires --output or report_output in the profile') output=(args.output or (ROOT/declared_output if declared_output else ROOT/'reports/webfetch')) print(build_report(evidence,output, - include_english=args.include_english, + include_english=args.include_english, language=requested_language(output), authored_interpretation=True)) diff --git a/tests/browser_use/test_report.py b/tests/browser_use/test_report.py index 1d218f6ea..87cc58ef0 100644 --- a/tests/browser_use/test_report.py +++ b/tests/browser_use/test_report.py @@ -982,8 +982,9 @@ def read_json(path: Path) -> dict: ) +@pytest.mark.parametrize("language", [None, "en", "zh"]) def test_report_publishes_one_final_logical_result_and_aggregates_cost( - tmp_path: Path, monkeypatch + tmp_path: Path, monkeypatch, language ) -> None: _use_single_model_profile(monkeypatch) evidence = tmp_path / "evidence" @@ -1066,8 +1067,21 @@ def test_report_publishes_one_final_logical_result_and_aggregates_cost( evidence=evidence, output=output, selection_path=Path("evaluation/browser-use/webarena-verified-pilot.json"), + language=language, ) - text = report.read_text(encoding="utf-8") + if language is not None: + contract = json.loads((output / "report-languages.json").read_text()) + assert contract["primary_language"] == language + assert contract["languages"] == (["zh", "en"] if language == "en" else ["zh"]) + assert report == output / ("en/index.html" if language == "en" else "index.html") + if language == "en": + from harness.web_fetch.report_i18n import untranslated_english_narrative + selection = load_pilot(Path("evaluation/browser-use/webarena-verified-pilot.json")) + source_text = {"中文", *(task.rationale for task in selection.tasks)} + for page in (output / "en").rglob("*.html"): + assert not untranslated_english_narrative(page.read_text(), source_text), page + + text = (output / "index.html").read_text(encoding="utf-8") assert "1/1" in text data = json.loads((output / "report-data.json").read_text(encoding="utf-8")) assert data["summary"]["gpt-5.6-sol"]["chromium"]["score"] == 1 @@ -1135,9 +1149,11 @@ def test_report_publishes_one_final_logical_result_and_aggregates_cost( assert 'id="evidence-viewer-content"' in viewer assert "execution-evidence[.]json" in viewer assert "fetch(new URL(source,document.baseURI)" in viewer - assert not (output / "en/evidence-viewer.html").exists() - assert "英文报告未生成" in (output / "en/index.html").read_text() - assert json.loads((output / "report-languages.json").read_text())["languages"] == ["zh"] + assert (output / "en/evidence-viewer.html").exists() is (language == "en") + assert ("英文报告未生成" in (output / "en/index.html").read_text()) is (language != "en") + assert json.loads((output / "report-languages.json").read_text())["languages"] == ( + ["zh", "en"] if language == "en" else ["zh"] + ) public_results = [ json.loads(line) for line in (output / "data/results.jsonl").read_text(encoding="utf-8").splitlines() diff --git a/tests/test_public_report_language.py b/tests/test_public_report_language.py new file mode 100644 index 000000000..45b5a286f --- /dev/null +++ b/tests/test_public_report_language.py @@ -0,0 +1,190 @@ +"""Language choice at the public run/rebuild boundary, independent of scoring.""" +import json +import os +import runpy +import shlex +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[1] + + +@pytest.mark.parametrize("mode", ["--diagnostic-cohorts", "--webmainbench-full"]) +@pytest.mark.parametrize("language", [None, "zh", "en"]) +@pytest.mark.parametrize("saved", [False, True]) +def test_chinese_only_commands_preserve_omitted_language(tmp_path, monkeypatch, mode, language, saved): + from harness.web_fetch import report_build + output = tmp_path / "report" + if saved: + output.mkdir() + (output / "report-languages.json").write_text(json.dumps({"languages": ["zh"], "primary_language": "zh"})) + calls = [] + builder = "build_diagnostic_cohorts" if mode == "--diagnostic-cohorts" else "build_webmainbench_full_report" + monkeypatch.setattr(report_build, builder, lambda *args, **kwargs: calls.append((args, kwargs)) or output / "index.html") + script = str(ROOT / "scripts/build_report.py") + argv = [script, mode, "--evidence", str(tmp_path / "evidence"), "--output", str(output)] + if language: + argv += ["--language", language] + monkeypatch.setattr(sys, "argv", argv) + with pytest.raises(SystemExit) as result: + runpy.run_path(script, run_name="__main__") + assert result.value.code == (2 if language == "en" else 0) + assert len(calls) == (0 if language == "en" else 1) + + +@pytest.mark.parametrize("language", [None, "zh"]) +def test_historical_diagnostic_command_builds_actual_report(tmp_path, language): + from tests.test_report_templates import diagnostic_example + evidence = tmp_path / "evidence" + diagnostic_example(evidence) + output = tmp_path / "report" + command = [sys.executable, str(ROOT / "scripts/build_report.py"), "--diagnostic-cohorts", + "--evidence", str(evidence), "--output", str(output)] + if language: + command += ["--language", language] + result = subprocess.run(command, cwd=ROOT, capture_output=True, text=True) + assert result.returncode == 0, result.stderr + assert json.loads((output / "report-manifest.json").read_text())["languages"] == ["zh"] + original = (output / "index.html").read_bytes() + english = subprocess.run(command + ["--language", "en"], cwd=ROOT, capture_output=True, text=True) + assert english.returncode == 2 + assert "supports only zh" in english.stderr + assert (output / "index.html").read_bytes() == original + # Historical diagnostics deliberately require a new directory, never overwrite. + again = subprocess.run(command, cwd=ROOT, capture_output=True, text=True) + assert again.returncode != 0 + assert "new output" in again.stderr + assert (output / "index.html").read_bytes() == original + + +@pytest.mark.parametrize("target", ["report", "popular-sites-report", "browser-use-report", "trajectory-replay-report", "public-benchmark-report", "webfetch-diagnostic-report"]) +@pytest.mark.parametrize("language", [None, "en", "zh"]) +def test_make_language_override_is_only_explicit(tmp_path, monkeypatch, target, language): + make = shutil.which("make") + if not make or subprocess.run([make, "--version"], capture_output=True).returncode: + pytest.skip("Make is not installed") + args = [make, "--no-print-directory", "-n", "-f", str(ROOT / "Makefile"), target, "EVIDENCE=" + str(tmp_path), "OUTPUT=" + str(tmp_path / "report")] + if language: + args += ["REPORT_LANGUAGE=" + language] + env = {key: value for key, value in os.environ.items() if key not in {"REPORT_LANGUAGE", "MAKEFLAGS"}} + result = subprocess.run(args, cwd=tmp_path, capture_output=True, text=True, env=env) + assert result.returncode == 0, result.stderr + command = next(shlex.split(line) for line in result.stdout.splitlines() if "scripts/build_" in line) + if language: + assert command[command.index("--language") + 1] == language + else: + assert "--language" not in command + if target == "report": + # Exercise the expanded public command with a saved Chinese selection. + # Evidence validation/rendering are covered by the real CLI scenarios below. + from unittest.mock import patch + from harness.web_fetch import report_build, storage + output = tmp_path / "reports/webfetch" + output.mkdir(parents=True) + (output / "report-languages.json").write_text(json.dumps({"primary_language": "zh", "languages": ["zh"]})) + command = command[command.index("scripts/build_report.py"):] + command[0] = str(ROOT / command[0]) + captured = [] + monkeypatch.chdir(tmp_path) + with patch.object(sys, "argv", command), \ + patch.object(storage, "require_canonical_inputs"), patch.object(storage, "require_current_report_profile"), \ + patch.object(storage, "require_evidence", return_value=tmp_path), \ + patch.object(report_build, "build_report", side_effect=lambda *a, **kw: captured.append(kw) or output / "index.html"): + runpy.run_path(command[0], run_name="__main__") + assert captured[0]["language"] == (language or "zh") + + + +def cli(*args): + return subprocess.run([sys.executable, "-m", "harness.web_fetch", *args], + cwd=ROOT, capture_output=True, text=True, + env={**os.environ, "PYTHONPATH": str(ROOT)}) + + +@pytest.mark.parametrize("language", [None, "en", "zh"]) +def test_run_selects_language_and_rebuild_preserves_it(tmp_path, language): + run = tmp_path / "run" + options = ["--language", language] if language else [] + result = cli("run", "--delay", "0", "--output", str(run), *options) + assert result.returncode == 0, result.stdout + result.stderr + selected = language or "en" + contract = json.loads((run / "report/report-languages.json").read_text()) + assert contract["primary_language"] == selected + assert contract["languages"] == (["zh", "en"] if selected == "en" else ["zh"]) + expected = run / "report" / ("en/index.html" if selected == "en" else "index.html") + assert str(expected) in result.stdout + assert ('lang="en"' if selected == "en" else 'lang="zh-CN"') in expected.read_text() + measured = (run / "results.jsonl").read_bytes() + rebuilt = cli("report", str(run)) + assert rebuilt.returncode == 0, rebuilt.stdout + rebuilt.stderr + assert str(expected) in rebuilt.stdout + assert (run / "results.jsonl").read_bytes() == measured + other = "zh" if selected == "en" else "en" + changed = cli("report", str(run), "--language", other) + assert changed.returncode == 0, changed.stdout + changed.stderr + assert json.loads((run / "report/report-languages.json").read_text())["primary_language"] == other + assert (run / "results.jsonl").read_bytes() == measured + from harness.reporting.links import check + check(run / "report") + + +def test_unsupported_language_fails_before_calls_or_output(tmp_path): + run = tmp_path / "run" + result = cli("run", "--language", "fr", "--output", str(run)) + assert result.returncode == 2 + assert "en" in result.stderr and "zh" in result.stderr + assert not run.exists() + + +@pytest.mark.parametrize("script", ["build_report.py", "build_browser_use_report.py", "build_webarena_replay_report.py"]) +def test_report_entrypoints_reject_unknown_language_before_loading_evidence(tmp_path, script): + if script == "build_webarena_replay_report.py": + pytest.importorskip("webarena_verified") + output = tmp_path / "report" + arguments = ["--language", "fr", "--output", str(output)] + arguments += (["--results", "missing"] if "replay" in script else ["--evidence", "missing"]) + result = subprocess.run([sys.executable, str(ROOT / "scripts" / script), *arguments], + cwd=ROOT, capture_output=True, text=True) + assert result.returncode == 2 + assert "invalid choice" in result.stderr + assert not output.exists() + + +def test_default_english_has_localized_ui_and_preserved_task_text(tmp_path): + from harness.web_fetch.report_i18n import untranslated_english_narrative + run = tmp_path / "run" + result = cli("run", "--delay", "0", "--output", str(run)) + assert result.returncode == 0, result.stderr + dataset = json.loads((run / "dataset.json").read_text()) + # Task wording and participant metadata are source inputs, not report copy. + source_text = {"中文", "确定性安装测试", "固定"} + source_text.update(case["name"] for case in dataset["cases"]) + source_text.update(case["task_prompt"] for case in dataset["cases"]) + for page in (run / "report/en").rglob("*.html"): + assert not untranslated_english_narrative(page.read_text(), source_text), page + for case in dataset["cases"]: + assert case["task_prompt"] in (run / f"report/en/datasets/{case['id']}.html").read_text() + + +def test_release_translation_uses_each_reports_records_without_global_state(tmp_path): + from harness.web_fetch.report_i18n import TEXT, write_english_report + title, summary = "独立版本标题", "独立版本说明" + for edition in ("First", "Second"): + root = tmp_path / edition + root.mkdir() + (root / "index.html").write_text(f'

{title}

{summary}

') + history = {"releases": [{"id": "example", "title": title, "summary": summary, "changes": [], + "i18n": {"en": {"title": edition, "summary": edition + " release", "changes": []}}}]} + path = root / "report-changes.json" + path.write_text(json.dumps(history)) + original = path.read_bytes() + write_english_report(root) + english = (root / "en/index.html").read_text() + assert f'

{edition}

' in english + assert f'

{edition} release

' in english + assert path.read_bytes() == original + assert title not in TEXT and summary not in TEXT diff --git a/tests/test_report_changes.py b/tests/test_report_changes.py index a39a179f2..8d6b940f8 100644 --- a/tests/test_report_changes.py +++ b/tests/test_report_changes.py @@ -30,7 +30,7 @@ def test_webmainbench_history_is_localized_from_one_record(self): def test_web_fetch_release_translations_come_from_history(self): from harness.web_fetch.report_changes import ROOT - from harness.web_fetch.report_i18n import translate_text + from harness.web_fetch.report_i18n import release_translations, translate_text data = json.loads((ROOT / 'research/report-changes.json').read_text(encoding='utf-8')) from harness.web_fetch.report_changes import select_cohort_history kinds = ('generic', 'headless_markdown', 'web_fetch_methods', @@ -39,14 +39,15 @@ def test_web_fetch_release_translations_come_from_history(self): releases = {row['id']: row for kind in kinds for row in select_cohort_history(data, {'comparison': {'kind': kind}}, [])['releases']} self.assertTrue(releases) + translations = release_translations({"releases": list(releases.values())}) for row in releases.values(): english = row['i18n']['en'] for key in ('title', 'summary'): - self.assertEqual(translate_text(row[key]), english[key], row['id']) + self.assertEqual(translate_text(row[key], translations), english[key], row['id']) self.assertEqual(len(row['changes']), len(english['changes'])) for source, target in zip(row['changes'], english['changes'], strict=True): for zh, en in zip(source, target, strict=True): - self.assertEqual(translate_text(zh), en, row['id']) + self.assertEqual(translate_text(zh, translations), en, row['id']) def test_dynamic_local_audit_release_is_selected_only_for_its_cohort(self): from harness.web_fetch.report_changes import ROOT, select_cohort_history @@ -82,11 +83,12 @@ def test_risk_control_release_is_selected_only_for_its_cohort(self): def test_current_web_fetch_release_binds_score_revision_and_exact_rerun_scope(self): from harness.web_fetch.report_changes import ROOT, select_cohort_history - from harness.web_fetch.report_i18n import translate_text + from harness.web_fetch.report_i18n import release_translations, translate_text history = json.loads((ROOT/'research/report-changes.json').read_text()) selected = select_cohort_history( history, {'comparison': {'kind': 'web_fetch_methods'}}, [] ) + translations = release_translations(selected) current = selected['releases'][0] self.assertEqual(current['id'], '2026-09-15-official-route-and-call-drilldown') self.assertEqual(set(current['scoring_revision_input_sha256']), { @@ -101,7 +103,7 @@ def test_current_web_fetch_release_binds_score_revision_and_exact_rerun_scope(se self.assertNotIn('新增异常组', current['summary']) for value in [current['title'], current['summary'], *(field for row in current['changes'] for field in row)]: - self.assertNotRegex(translate_text(value), r'[\u4e00-\u9fff]') + self.assertNotRegex(translate_text(value, translations), r'[\u4e00-\u9fff]') def test_first_correction_is_identity_bound_not_latest(self): from harness.web_fetch.report_changes import first_correction diff --git a/tests/test_report_languages.py b/tests/test_report_languages.py index 3f45f6ca9..c870fd889 100644 --- a/tests/test_report_languages.py +++ b/tests/test_report_languages.py @@ -11,6 +11,22 @@ ) +@pytest.mark.parametrize("supported", [("en", "zh"), ("zh",)]) +@pytest.mark.parametrize("saved", [None, "en", "zh", "legacy"]) +@pytest.mark.parametrize("requested", [None, "en", "zh"]) +def test_language_resolution_respects_choice_and_renderer_capability(tmp_path, supported, saved, requested): + from harness.report_languages import resolve_report_language + if saved: + contract = {"languages": ["zh"]} if saved == "legacy" else {"primary_language": saved} + (tmp_path / "report-languages.json").write_text(json.dumps(contract)) + expected = requested or ("zh" if saved == "legacy" else saved) or supported[0] + if expected in supported: + assert resolve_report_language(tmp_path, requested, supported=supported) == expected + else: + with pytest.raises(ValueError, match="supports only zh"): + resolve_report_language(tmp_path, requested, supported=supported) + + def test_chinese_output_has_notice_without_translated_routes(tmp_path): write_language_contract(tmp_path, include_english=False) assert generated_languages(tmp_path) == ("zh",) @@ -64,3 +80,58 @@ def test_chinese_spacing_preserves_numeric_parity_and_detects_changes(self): ) self.assertNotEqual(visible_numbers("100题"), visible_numbers("99 tasks")) self.assertNotEqual(visible_numbers("100题"), visible_numbers("all tasks")) + + +@pytest.mark.parametrize("base_href,home_href", [(None, "../../index.html"), ("../../", "index.html"), ("../../", ""), ("../../", "./")]) +def test_primary_english_keeps_explicit_chinese_navigation_and_evidence(tmp_path, base_href, home_href): + from harness.report_languages import REPORT_PAGE_MARKER, select_report_language + base = f'' if base_href else "" + def page(content, head=""): + return f'{REPORT_PAGE_MARKER}{head}{content}' + (tmp_path / "index.html").write_text(page('
Home
')) + (tmp_path / "en").mkdir() + (tmp_path / "en/index.html").write_text(page('
中文
')) + detail = tmp_path / "results/task/index.html" + detail.parent.mkdir(parents=True) + detail.write_text(page(f'
Home
', base)) + evidence = tmp_path / "artifacts/original.html" + evidence.parent.mkdir() + evidence.write_text(page('
Original
')) + evidence_bytes = evidence.read_bytes() + write_language_contract(tmp_path, include_english=True) + assert select_report_language(tmp_path, "en") == tmp_path / "en/index.html" + assert 'href="../index.html?report-language=zh"' in (tmp_path / "en/index.html").read_text() + assert 'token=abc&report-language=zh#overview' in detail.read_text() + assert 'data-report-language-entry' in (tmp_path / "index.html").read_text() + assert evidence.read_bytes() == evidence_bytes + contract = json.loads((tmp_path / "report-languages.json").read_text()) + assert contract["entrypoint"] == "en/index.html" + + +def test_rebuild_language_recognizes_older_chinese_only_reports(tmp_path): + from harness.report_languages import primary_report_language + assert primary_report_language(tmp_path) == "en" + write_language_contract(tmp_path, include_english=False) + assert primary_report_language(tmp_path) == "zh" + write_language_contract(tmp_path, include_english=True) + assert primary_report_language(tmp_path) == "zh" + + +@pytest.mark.parametrize("damage", ["unsupported", "missing_edition", "wrong_entry", "missing_redirect"]) +def test_release_rejects_a_wrong_primary_language(tmp_path, damage): + from harness.report_languages import REPORT_PAGE_MARKER, select_report_language + (tmp_path / "index.html").write_text(f'{REPORT_PAGE_MARKER}') + (tmp_path / "en").mkdir() + (tmp_path / "en/index.html").write_text('English') + write_language_contract(tmp_path, include_english=True) + select_report_language(tmp_path, "en") + check_language_contract(tmp_path) + path = tmp_path / "report-languages.json" + contract = json.loads(path.read_text()) + if damage == "unsupported": contract["primary_language"] = "fr" + elif damage == "missing_edition": contract["languages"] = ["zh"] + elif damage == "wrong_entry": contract["entrypoint"] = "index.html" + else: (tmp_path / "index.html").write_text('Chinese root without selected-language redirect') + path.write_text(json.dumps(contract)) + with pytest.raises(ValueError, match="language|entrypoint"): + check_language_contract(tmp_path) diff --git a/tests/test_report_presentation.py b/tests/test_report_presentation.py index f2040dcbe..da7737ff7 100644 --- a/tests/test_report_presentation.py +++ b/tests/test_report_presentation.py @@ -258,8 +258,9 @@ def test_article_aria_labels_localize_without_touching_evidence_attributes(self) self.assertIn('alt="Article contents"',translated) self.assertIn('src="data:image/png;base64,YQ=="',translated) other=localize_html(markup,PurePosixPath('attempts/example.html')) - self.assertIn('aria-label="文章目录"',other) - self.assertIn('alt="文章目录"',other) + self.assertIn('aria-label="Article contents"',other) + self.assertIn('alt="Article contents"',other) + self.assertIn('data-evidence="文章目录"',other) def test_article_selects_localized_assets_before_rewriting_relative_urls(self): from pathlib import PurePosixPath from harness.web_fetch.report_i18n import localize_html diff --git a/tests/test_wcxb_report_languages.py b/tests/test_wcxb_report_languages.py index fe8a14de0..2e0f68906 100644 --- a/tests/test_wcxb_report_languages.py +++ b/tests/test_wcxb_report_languages.py @@ -44,7 +44,8 @@ def test_wcxb_command_forwards_requested_language(tmp_path, monkeypatch, include monkeypatch.setattr(report_build, 'build_wcxb_report', lambda *args, **kwargs: captured.append(kwargs) or tmp_path / 'index.html') root = Path(__file__).resolve().parents[1] - argv = [str(root / 'scripts/build_report.py'), '--wcxb', '--evidence', str(tmp_path)] + argv = [str(root / 'scripts/build_report.py'), '--wcxb', '--evidence', str(tmp_path), + '--output', str(tmp_path / 'new-report')] if include_english: argv.append('--include-english') monkeypatch.setattr('sys.argv', argv) @@ -52,7 +53,7 @@ def test_wcxb_command_forwards_requested_language(tmp_path, monkeypatch, include runpy.run_path(argv[0], run_name='__main__') assert result.value.code == 0 binding = json.loads((root / 'evaluation/wcxb/report-content-binding.json').read_text()) - assert captured == [{'include_english': include_english, 'authority_root': root, + assert captured == [{'include_english': include_english, 'language': 'en', 'authority_root': root, 'content_binding': binding}] diff --git a/tests/trajectory_replay/test_webarena_report.py b/tests/trajectory_replay/test_webarena_report.py index 3ad8e5d73..1375d871f 100644 --- a/tests/trajectory_replay/test_webarena_report.py +++ b/tests/trajectory_replay/test_webarena_report.py @@ -137,8 +137,11 @@ def test_complete_three_method_report_builds_one_unified_matrix( manifest = build_webarena_replay_report( results_path=files["results.json"], dataset_path=files["dataset.json"], review_path=files["review.json"], profile_path=files["profile.json"], - output_dir=output, include_english=True, + output_dir=output, language="en", ) + assert manifest["primary_language"] == "en" + assert manifest["entrypoint"] == "en/index.html" + assert json.loads((output / "report-manifest.json").read_text())["languages"] == ["zh", "en"] invalid_profile = tmp_path / "reset-every-task-profile.json" invalid_profile.write_text( json.dumps(