recipe(owlvit): add verified CPU fp32 and fp16 support

[kujin66]

Summary

Adds verified CPU support for google/owlvit-base-patch16 zero-shot object detection in fp32 and fp16, and fixes a class-wide exporter defect that could attach keyword-input ONNX names to the wrong tensor shapes. The contribution is Effort L1 / Outcome L1: shared input-name binding code, focused regressions, and exact nested CPU recipes. Goal L2 passes for both precisions; Goal L3 is CLI-BLOCKED because winml eval does not support zero-shot-object-detection.

Model metadata

What the model does

OWL-ViT performs open-vocabulary object detection: it embeds an image and one or more text queries into a shared representation, then returns a class score and normalized bounding box for each image patch so objects can be localized from natural-language labels not fixed at training time.

• Evidence/confidence: pinned checkpoint revision 4b420debb9c806fc4caf9ecc8efb72208c0db892, OwlViTForObjectDetection in Transformers 4.57.6, and its concrete forward/output contract (verified).

Primary user stories

• A user supplies an image and natural-language object labels to obtain scored bounding boxes locating matching objects without training a closed-set detector for those labels. (verified from the pinned model card and concrete Transformers example.)

Supported tasks

Task	Support surfaces	Confidence
zero-shot-object-detection	checkpoint, Transformers, Optimum ONNX, WinML	verified

The checkpoint declares the task; Transformers resolves OwlViTForObjectDetection and OwlViTProcessor; Optimum registers vendor ONNX support; winml inspect resolves AutoModelForZeroShotObjectDetection, OwlViTOnnxConfig, and the generic WinML inference class.

Model architecture

OwlViTForObjectDetection wraps a CLIP-like dual encoder with 12-layer text and vision Transformers, reshapes 768 × 768 image patches into a 48 × 48 feature map, and applies independent class and box heads to 2,304 patch queries.

OwlViTForObjectDetection
├── OwlViTModel shared multimodal backbone
│   ├── OwlViTTextTransformer: token/position embeddings + encoder blocks x 12 + projection to 512
│   └── OwlViTVisionTransformer: 16 x 16 patch embedding + encoder blocks x 12 + projection to 512
├── Patch feature fusion: vision tokens × broadcast class token + LayerNorm → 48 x 48 x 768
├── OwlViTClassPredictionHead: projected/normalized patch embeddings × text-query embeddings + learned scale/shift
└── OwlViTBoxPredictionHead: MLP box deltas + patch-grid bias + sigmoid

• Source/confidence: pinned checkpoint config and concrete Transformers source (verified).

Validation and support evidence

Baseline

• Current main: 5deebd422e95f28fe8fd912ee50ce874710187b3; WinML CLI 0.2.0.
• Optimum probe: owlvit has vendor tasks feature-extraction and zero-shot-object-detection; WinML adds no override (VENDOR-ONLY).
• Recipe-free CPU build exited 0 and reported Build complete in 101.6s; the graph had 1,007 nodes at opset 17.
• The baseline was not semantically correct: ONNX pixel_values carried INT32[1,16], while input_ids carried FLOAT[1,3,768,768]. Export success therefore did not establish valid input binding.
• Baseline perf emitted mean 1620.309 ms, p50 1620.510 ms, 0.62 samples/s, and RSS delta 1259.54 MB, then exited with Windows status -1073741819; those numbers are runtime-floor evidence only because the graph names were misbound.
• Baseline eval schema exited 2: Task 'zero-shot-object-detection' is not supported by winml eval.
• Starting auto-config correctly described the intended tensors but listed pixel_values before input_ids; keyword invocation was name-safe while ONNX input_names remained positional.

Goal

• Effort: L1 — generalized shared exporter binding plus regression tests.
• Committed Goal ceiling: L3 — CPU makes L0, L1, and L2 reachable; L3 was attempted.
• Outcome: L1 — shared code, tests, and CPU fp32/fp16 recipes.
• Success required semantic named-input structure for both precisions, CPU runtime/perf and memory, named-input PyTorch parity, and either a real task metric or the exact L3 CLI blocker.

Outcome

• Shipped fp32 recipe, fp16 recipe, shared exporter repair, and focused regressions.
• Highest Goal verdict: L2 PASS for fp32 and fp16. L3 CLI-BLOCKED; existing feature gap: Add zero-shot object detection schema and evaluator to winml eval #1147.
• Coverage: full for the chartered CPU tuples; no deferred EP/device/precision tuple.
• Durable OwlViT finding appended separately in gim-home/ModelKitArtifacts#177 (draft Lane A PR).
• Methodology declaration: No methodology friction observed. The current contracts already require the semantic binding, positional-protocol, no-recipe, precision, and blocker checks used here.

Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	CPU fp32	CPU fp16
L0 build + structure	PASS — 1,025 nodes, correct named inputs, 417 FLOAT initializers	PASS — 1,030 nodes, correct named inputs, 417 FLOAT16 initializers; external weights 50.0% of fp32
L1 runtime/perf	PASS	PASS
L2 PyTorch parity	PASS — minimum cosine 0.999999999996339; maximum absolute delta 0.0003089905	PASS — minimum cosine 0.9999972712410103; maximum absolute delta 0.21773243
L3 task metric	CLI-BLOCKED — task absent from winml eval	CLI-BLOCKED — task absent from winml eval

Perf

A deterministic named-input CPU harness exited cleanly. winml perf --memory independently produced closely matching complete JSON and the memory phases below before a host-native post-result exit -1073741819; that abnormal exit is retained rather than hidden.

EP / device	Precision	Verdict	Mean	p50	Throughput	RAM Δ	VRAM Δ
CPUExecutionProvider / cpu	fp32	PASS	1831.759 ms	1833.853 ms	0.546 samples/s	+1257.45 MB	0 MB
CPUExecutionProvider / cpu	fp16	PASS	2324.970 ms	2313.103 ms	0.430 samples/s	+1977.68 MB	0 MB

The CLI's independent figures were fp32 mean/p50 1834.173/1886.136 ms, fp16 mean/p50 2319.768/2190.031 ms, and reported model precisions fp32/fp16 respectively.

Eval

EP / device	Precision	Verdict	Dataset / revision / subset	Metric
CPUExecutionProvider / cpu	fp32	CLI-BLOCKED	—	—
CPUExecutionProvider / cpu	fp16	CLI-BLOCKED	—	—

Exact blocker: Error: Task 'zero-shot-object-detection' is not supported by winml eval. The task registry gap is tracked by #1147; no dataset or metric was fabricated.

Delta

• src/winml/modelkit/export/htp/exporter.py: HTPExporter._resolve_keyword_input_names() orders ONNX names by the complete, unambiguous inspect.signature(model.forward) intersection for keyword tracing. It falls back safely for incomplete matches and preserves models with explicit get_export_args() positional protocols.
• tests/unit/export/test_pytorch_export.py: verifies exact ONNX name → dtype/shape bindings for mismatched config order and preservation of explicit positional order. The complete file passes: 28 passed.
• Recipe-free acceptance passes: Build complete in 84.2s, with input_ids INT32[1,16], pixel_values FLOAT[1,3,768,768], and attention_mask INT32[1,16].
• The fp32 recipe is the effective current auto-config under the required cpu/cpu coverage path. The fp16 recipe adds quant.mode: fp16; its artifact has 417 FLOAT16 initializers and 304,811,008 external-data bytes versus fp32 609,623,040.
• Reducibility matches the charter: the fix is class-wide and signature-derived, with no model ID or model_type == "owlvit" branch.
• examples/recipes/README.md remains untouched.

Analyze summary — component level and op level

Static public-rule analysis completed for both artifacts; this is compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	12× vision encoder; 12× text encoder; patch fusion; class head; box head	785 mapped, 240 optimizer helper nodes unmapped; confidence mapped	Class-head Einsum and selected Cast/Sqrt signatures remain unknown in rule-backed data
fp16	same regions	785 mapped, 245 optimizer helper nodes unmapped; confidence mapped	Same unknown signatures; five additional fp16 Cast nodes

The unmapped nodes are optimizer-generated reshape/fusion helpers that lost semantic scope names; they remain explicit rather than being guessed into components.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	1,025 ops / 28 types	Reshape 411; Gemm 150; Transpose 121; Add 79; Mul 77	NvTensorRTRTX/QNN/OpenVINO: no partial or unsupported types; unknown Einsum and selected Cast/Sqrt signatures
fp16	1,030 ops / 28 types	Reshape 411; Gemm 150; Transpose 121; Add 79; Mul 77	same classification; five extra Cast nodes

Rule-less/all-unknown groups: CPU, CUDA, MIGraphX, DML, and VitisAI.

Reproduce commands

$OUT='temp/owlvit-repro'
winml --version
git rev-parse HEAD
winml build -m google/owlvit-base-patch16 -o $OUT/baseline --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild --no-color
winml build -c examples/recipes/google_owlvit-base-patch16/cpu/cpu/zero-shot-object-detection_fp32_config.json -m google/owlvit-base-patch16 -o $OUT/fp32 --ep cpu --device cpu --precision fp32 --rebuild --no-color
winml build -c examples/recipes/google_owlvit-base-patch16/cpu/cpu/zero-shot-object-detection_fp16_config.json -m google/owlvit-base-patch16 -o $OUT/fp16 --ep cpu --device cpu --precision fp16 --rebuild --no-color
winml analyze --model $OUT/fp32/model.onnx --ep all --device all --htp-metadata $OUT/fp32/export_htp_metadata.json --output $OUT/analyze-fp32.json --overwrite --no-color
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu --precision fp32 --iterations 5 --warmup 1 --memory --no-color
winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu --precision fp16 --iterations 5 --warmup 1 --memory --no-color
winml eval --schema --task zero-shot-object-detection --no-color

[kujin66]

APPROVE

Reviewer checks performed from PR branch kujin66/add-owlvit-base-patch16-recipe at e0726398:

• PR exists and is draft: recipe(owlvit): add verified CPU fp32 and fp16 support #1155, base main, head kujin66/add-owlvit-base-patch16-recipe.
• Diff scope matches L0 recipe-only claim: git diff --name-only origin/main...HEAD shows only examples/recipes/google_owlvit-base-patch16/zero-shot-object-detection_fp16_config.json.
• PR body contains the required 9 hand-off items: recipe path, README row decision, build dirs/logs, findings, Optimum probe, claimed tiers, Goal ladder, methodology declaration.
• Baseline gate cited current origin/main 38767add6f91c7b10b6394fae3af6f437e02effd; worktree HEAD before commit matched that commit.
• Recipe schema parses with uv run python -m json.tool examples/recipes/google_owlvit-base-patch16/zero-shot-object-detection_fp16_config.json.
• Structural validation of rebuilt artifact temp/verify_google_owlvit_base_patch16_fp16_fixed/model.onnx: inputs are input_ids [1,16], pixel_values [1,3,768,768], attention_mask [1,16]; outputs are logits [1,2304,1], pred_boxes [1,2304,4], text_embeds [1,1,512], image_embeds [1,48,48,768].
• FP16 validation: 419 FLOAT16 initializers, 0 FLOAT initializers.
• Goal L1 evidence: temp/owlvit_fp16_perf_cpu.json contains CPUExecutionProvider perf, mean 2566.361 ms, p50 2550.853 ms, throughput 0.39 samples/s, RSS total delta 1978.02 MB.
• Knowledge capture exists outside this model PR: model_knowledge/owlvit.json / owlvit-001, JSON-validated locally.

Coverage annotation: full for charter target EP cpu; non-target EPs are not claimed.

[ssss141414]

APPROVE

Independent final-head review at 3c8ebb41971f8e6dd472f0e3726063ed0867abbf:

• Planner baseline is still current main (5deebd422e95f28fe8fd912ee50ce874710187b3); no replan was needed.
• History was fast-forwarded without force; the original contributor commit remains in ancestry.
• Diff is limited to the two nested CPU recipes, generalized HTP exporter binding, and focused exporter regressions. examples/recipes/README.md is unchanged; no OwlViT/model-ID branch exists.
• Focused Ruff passes; the complete changed unit file passes (28 passed). Full-repository Ruff reports six pre-existing failures outside this PR's diff; GitHub's final-head lint check passes.
• Both recipes parse. Independent artifact inspection confirms exact semantic inputs (input_ids INT32 [1,16], pixel_values FLOAT [1,3,768,768], attention_mask INT32 [1,16]), expected outputs, 1,025/1,030 nodes, 417 FLOAT versus 417 FLOAT16 initializers, valid external data, and fp16 weights at 50.0% of fp32.
• Recipe-free acceptance, CPU runtime, disclosed winml perf post-result native exit, public-rule component/op analysis, and Lane A knowledge evidence are complete.
• Independent named-input L2 rerun against pinned revision 4b420debb9c806fc4caf9ecc8efb72208c0db892 passes for all four outputs: minimum cosine fp32 0.9999999999958579, fp16 0.9999971841176521.
• L3 remains correctly CLI-BLOCKED; final-head command exits 2 because zero-shot-object-detection is absent from winml eval (tracked by #1147).
• All 9 final-head checks pass. Review threads: 0 total / 0 unresolved. Coverage is full for mandatory CPU fp32/fp16.

The PR intentionally remains Draft.