recipe(layoutlm): add document QA support for impira/layoutlm-document-qa

[DingmaomaoBJTU]

This PR adds impira/layoutlm-document-qa support to the LayoutLM export path in winml-cli, along with flat fp32 and w8a16 document-QA recipes. It ships at Outcome L1 (effort=L1-light, goal_ceiling=L2) and the supplied tester verdicts reached L2 PASS. The main caveat is that raw winml perf with random dummy inputs crashes on this checkpoint because token_type_ids can contain 1 while type_vocab_size=1; the tester therefore used the valid-input fallback path for L1 measurements.

Implementation summary: this adds LayoutLMQAIOConfig registration for (layoutlm, question-answering), a ZeroTokenTypeLayoutLMTextInputGenerator that forces all-zero token_type_ids, generic bbox ordering support in src\winml\modelkit\onnx\io.py, README indexing, fp32/w8a16 recipes, and unit coverage in tests/unit/export/test_onnx_config_overrides.py and tests/unit/export/test_pytorch_export.py.

1. Recipe path(s)

   • examples/recipes/impira_layoutlm-document-qa/question-answering_fp32_config.json
   • examples/recipes/impira_layoutlm-document-qa/question-answering_w8a16_config.json
   • Code paths in scope: src/winml/modelkit/models/hf/layoutlm.py, src/winml/modelkit/models/hf/__init__.py, src/winml/modelkit/onnx/io.py, tests/unit/export/test_onnx_config_overrides.py, tests/unit/export/test_pytorch_export.py

2. README row

   • Added row in examples/recipes/README.md: yes.

3. Build output dir

   • Refreshed isolated baseline against current origin/main f102649b63e6bf013a180523d15e6cab75906dd9.
   • Baseline output dir: temp\impira_layoutlm-document-qa\baseline_offline
   • fp32 output dir: temp\impira_layoutlm-document-qa\fp32
   • w8a16 output dir: temp\impira_layoutlm-document-qa\w8a16
   • Exact refreshed baseline command:
     C:\Users\qiowu\source\repos\winml-cli\.venv\Scripts\winml.exe build -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\baseline_offline --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
   • Refreshed baseline result: FAIL with exact evidence Error: Unrecognized configuration class <class 'transformers.models.layoutlm.configuration_layoutlm.LayoutLMConfig'> for this kind of AutoModel: AutoModelForNextSentencePrediction.
   • Engineering delta claimed by this PR: current origin/main fails at the refreshed isolated baseline above, while the branch-specific LayoutLM QA registration/input-generation changes below produce successful fp32 and w8a16 builds.

4. Build log

   • fp32 build command: ./.venv/Scripts/winml.exe build -c examples/recipes/impira_layoutlm-document-qa/question-answering_fp32_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\fp32 --ep cpu --device cpu --rebuild
     • Result: ✅ Build complete in 278.1s
     • Artifact: temp\impira_layoutlm-document-qa\fp32\model.onnx
     • I/O: inputs input_ids, bbox, attention_mask, token_type_ids as INT32 with shapes [1,512], [1,512,4], [1,512], [1,512]; outputs start_logits, end_logits as FLOAT [1,512]
     • Initializer histogram: FLOAT=206, INT64=63
     • Build quant: null
   • w8a16 build command: ./.venv/Scripts/winml.exe build -c examples/recipes/impira_layoutlm-document-qa/question-answering_w8a16_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\w8a16 --rebuild
     • Result: ✅ Quantize 82.3s / Precision: uint8/uint16 (weight/activation) / ✅ Build complete in 392.4s
     • Artifact: temp\impira_layoutlm-document-qa\w8a16\model.onnx
     • I/O: same contract as fp32
     • Initializer histogram: FLOAT=411, INT32=196, INT64=63, UINT16=225, UINT8=196
     • Build quant: mode=static, weight_type=uint8, activation_type=uint16

5. Appended findings

   • Lane A only (not staged in this PR): local skill-repo commit 26e1b7b1d83919d53c673c142854dbda16c6eb84 appended layoutlm-001, layoutlm-002, and layoutlm-003 to copilot-skills/dev_skill/adding-model-support/model_knowledge/layoutlm.json.
   • Methodology finding _meta-060 was also appended locally in the skill repo; it is declared here for traceability but intentionally excluded from this model PR diff.

6. Optimum-coverage probe

   • Vendor registrations: ['feature-extraction','fill-mask','text-classification','token-classification']
   • After WinML changes: ['feature-extraction','fill-mask','text-classification','token-classification','question-answering']
   • Added by WinML: ['question-answering']
   • Verdict: branch state adds question-answering coverage for LayoutLM beyond the vendor baseline.

7. Claimed (Effort, Goal, Outcome)

   • model_id: impira/layoutlm-document-qa
   • model_type: layoutlm
   • effort: L1-light
   • goal_ceiling: L2
   • outcome: L1
   • target_eps: cpu
   • Planner facts carried into the implementation: architectures=["LayoutLMForQuestionAnswering"], model_type="layoutlm", is_encoder_decoder=false, type_vocab_size=1, max_position_embeddings=514, max_2d_position_embeddings=1024; the critical finding was that vendor dummy generation can emit token_type_ids=1, so this checkpoint required forcing all-zero token_type_ids.
   • The claimed delta is specifically from refreshed-current-main baseline FAIL (item 3) to branch-state recipe builds/L1/L2 PASS evidence (items 4, 8, 10, 11).

8. Goal-ladder verdict table

   Catalog gate summary: refreshed-current-main baseline_build=FAIL, verdict=real-engineering, file_pr=true.

   Tier	Verdict	Evidence
   L0	PASS	The refreshed isolated baseline against current origin/main failed with Error: Unrecognized configuration class <class 'transformers.models.layoutlm.configuration_layoutlm.LayoutLMConfig'> for this kind of AutoModel: AutoModelForNextSentencePrediction. After the branch's LayoutLM QA export/input-generation changes, both recipe builds succeeded. fp32 produced temp\impira_layoutlm-document-qa\fp32\model.onnx with INT32 inputs (input_ids, bbox, attention_mask, token_type_ids) and FLOAT outputs (start_logits, end_logits), histogram FLOAT=206, INT64=63, and build_quant=null. w8a16 produced temp\impira_layoutlm-document-qa\w8a16\model.onnx with the same I/O contract, histogram FLOAT=411, INT32=196, INT64=63, UINT16=225, UINT8=196, and build_quant={mode: static, weight_type: uint8, activation_type: uint16}.
   L1	PASS	Providers: ['AzureExecutionProvider', 'CPUExecutionProvider']. Raw winml perf on both artifacts crashed with `Error: Benchmark failed: Inference failed
   L2	PASS	fp32 vs PT: start_logits cosine 0.9999999999219783, max_abs 0.0022459030151367188, mean_abs 2.549774944782257e-05; end_logits cosine 0.9999999999193658, max_abs 0.0023813247680664062, mean_abs 2.7379952371120453e-05. w8a16 vs PT: start_logits cosine 0.9522225190737116, max_abs 13.425172805786133, mean_abs 2.4731171131134033; end_logits cosine 0.9597506819201421, max_abs 12.746326446533203, mean_abs 2.3781232833862305.

9. Methodology-evolution declaration

   • _meta-060 (silent-failure) was appended locally in the Lane A skill repo: explicit winml build overrides such as --device cpu --ep cpu can silently clear config.quant, yielding a successful build of the wrong precision for a quant recipe.
   • Reviewer/tester guidance was updated locally so quant recipes are validated from the preserved quant config and artifact, not from a superficially successful overridden build.

10. Perf & eval data

Raw winml perf on both artifacts crashed on random dummy inputs with the Gather/token-type error quoted in item 8, so the tester used the valid-input fallback path for the L1 measurements below.

EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM Δ	Task metric
CPUExecutionProvider / cpu	fp32	PASS	5005.385395001213 ms	5076.764100005676 ms	0.19978481597015124 samples/s	+467062784 B total (+358121472 B session init)	N/A (goal ceiling L2)
CPUExecutionProvider / cpu	w8a16	PASS	5581.969174998085 ms	5745.492499998363 ms	0.1791482483420097 samples/s	+311275520 B total (+25878528 B session init)	N/A (goal ceiling L2)

11. Component / op-level data

• Analyze command: ./.venv/Scripts/winml.exe analyze --model temp\impira_layoutlm-document-qa\fp32\model.onnx --ep all --output temp\impira_layoutlm-document-qa\analyze_fp32.json
• Artifact: temp\impira_layoutlm-document-qa\analyze_fp32.json
• Model summary: 395 total operators across 16 unique operator types.
• Per-EP classification:
  • QNNExecutionProvider / NPU: 16 supported / 0 partial / 0 unsupported / 0 unknown (395 total ops covered)
  • OpenVINOExecutionProvider / NPU: 15 supported / 1 partial / 0 unsupported / 0 unknown; partial op: OP/ai.onnx/Slice
  • VitisAIExecutionProvider / NPU: 0 supported / 0 partial / 0 unsupported / 16 unknown (no rule data)

12. Reproducible commands

C:\Users\qiowu\source\repos\winml-cli\.venv\Scripts\winml.exe build -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\baseline_offline --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
.\.venv\Scripts\winml.exe build -c examples\recipes\impira_layoutlm-document-qa\question-answering_fp32_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\fp32 --ep cpu --device cpu --rebuild
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py inspect --model temp\impira_layoutlm-document-qa\fp32\model.onnx --build-config temp\impira_layoutlm-document-qa\fp32\winml_build_config.json --label fp32
.\.venv\Scripts\winml.exe build -c examples\recipes\impira_layoutlm-document-qa\question-answering_w8a16_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\w8a16 --rebuild
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py inspect --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --build-config temp\impira_layoutlm-document-qa\w8a16\winml_build_config.json --label w8a16
.\.venv\Scripts\winml.exe perf --model temp\impira_layoutlm-document-qa\fp32\model.onnx --ep cpu --device cpu
.\.venv\Scripts\winml.exe perf --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --ep cpu --device cpu
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py perf --model temp\impira_layoutlm-document-qa\fp32\model.onnx --model-id impira/layoutlm-document-qa --label fp32
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py perf --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --model-id impira/layoutlm-document-qa --label w8a16
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py compare --model temp\impira_layoutlm-document-qa\fp32\model.onnx --model-id impira/layoutlm-document-qa --label fp32
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py compare --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --model-id impira/layoutlm-document-qa --label w8a16
.\.venv\Scripts\winml.exe analyze --model temp\impira_layoutlm-document-qa\fp32\model.onnx --ep all --output temp\impira_layoutlm-document-qa\analyze_fp32.json
python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py summarize-analyze --analyze-json temp\impira_layoutlm-document-qa\analyze_fp32.json

[DingmaomaoBJTU]

REQUEST_CHANGES

I re-checked PR #1093 from the PR head (5bd5ad4b) and reviewed the actual diff/files (src\winml\modelkit\models\hf\layoutlm.py, src\winml\modelkit\onnx\io.py, both recipes, README, and the two unit-test files). The code/tests look right-sized for effort=L1-light, the scope matches Outcome L1, the w8a16 artifact in temp\impira_layoutlm-document-qa\w8a16\winml_build_config.json still preserves quant={mode: static, weight_type: uint8, activation_type: uint16}, and PR-body items 8/10/11/12 match the tester’s corrected evidence.

Blocking fixes before I can approve:

1. Refresh the baseline on current origin/main and update the PR body. Reviewer.md requires the cited baseline head to equal review-time origin/main. Your baseline was run at defa3f63ffdc5c4cd3c252a4b3f60fcc9939f953, but review-time origin/main is f102649b63e6bf013a180523d15e6cab75906dd9, and git rev-list --count defa3f63ffdc5c4cd3c252a4b3f60fcc9939f953..origin/main returns 1. Please rebase/re-run the baseline gate on current origin/main, then refresh item 3/8 with the new command output and heads.

2. Correct PR-body item 6 (Optimum-coverage probe). The exact Step-1 probe on this PR head returns {'vendor': ['feature-extraction', 'fill-mask', 'text-classification', 'token-classification'], 'after_winml': ['feature-extraction', 'fill-mask', 'question-answering', 'text-classification', 'token-classification'], 'added_by_winml': ['question-answering']}. Item 6 currently says after_winml is unchanged and added_by_winml: none, which is inaccurate. Please replace item 6 with the real probe output after the refresh above.

What I did verify locally:

• gh pr checkout 1093; diff scope is exactly the 8 expected files, with no Lane A leakage.
• pytest tests\unit\export\test_onnx_config_overrides.py tests\unit\export\test_pytorch_export.py -q → 68 passed.
• uv run pre-commit run insert-license --all-files, uv run ruff check src/ tests/, and uv run mypy -p winml.modelkit all passed locally.
• Artifact spot-check: fp32 ONNX has the expected 4 inputs / 2 outputs; w8a16 build config retains non-null quant, so the _meta-060 quant-verification guardrail is satisfied by the cited recipe-driven build command.

[DingmaomaoBJTU]

APPROVE

Re-check complete on head b945df393ab5f0b2f729ad5a55855c03fd807e72 against current origin/main f102649b63e6bf013a180523d15e6cab75906dd9.

The two prior blocking reviewer items are resolved:

• Baseline freshness: PR item 3 now cites a refreshed isolated baseline on current origin/main, and the quoted failure is the expected AutoModelForNextSentencePrediction / LayoutLMConfig mismatch.
• Probe correctness: independently reproduced the task-registration delta with TasksManager: vendor ['feature-extraction','fill-mask','text-classification','token-classification'], after WinML import ['feature-extraction','fill-mask','text-classification','token-classification','question-answering'], added ['question-answering'].

Spot checks also passed:

• git merge-base HEAD origin/main == f102649b63e6bf013a180523d15e6cab75906dd9
• diff scope is unchanged and limited to the expected 8 recipe/code/test files
• winml inspect --list-tasks --model-type layoutlm includes question-answering
• targeted validation re-passed: 68 passed

Reviewer blocking issues are resolved. From the independent-review gate, this draft is ready to be converted to ready-for-review by the orchestrator.

[xieofxie]

Overall this PR makes sense — it correctly identifies and fixes the LayoutLM QA export failure (invalid token_type_ids + missing bbox ordering), with good test coverage and validation evidence. A few comments below.

[xieofxie]

All comments addressed. LGTM.