recipe(distilbert): add MNLI text classification recipes

[kujin66]

Summary

Adds nested CPU fp32 and fp16 recipes for typeform/distilbert-base-uncased-mnli text classification while preserving the contributor's original commits. This is an Effort L0, Outcome L0 recipe-only contribution; both required tuples reach Goal L3 with CPU runtime, PyTorch parity, and pinned GLUE MNLI evaluation evidence. The production recipe README remains unchanged.

Model metadata

What the model does

A compact DistilBERT natural-language-inference classifier that accepts a premise/hypothesis text pair and returns logits for entailment, neutral, or contradiction.

• Evidence/confidence: pinned checkpoint config and model card at revision cfa538a0fddbbd978fefe8966c1aeff7ad409c90, plus DistilBertForSequenceClassification source (verified).

Primary user stories

• A user supplies a premise and hypothesis to classify whether the premise entails, is neutral toward, or contradicts the hypothesis. Checkpoint MNLI identity, model card, and three-label config (verified).
• A user supplies a candidate label as a hypothesis with an input statement as the premise to score zero-shot classification labels through NLI. Checkpoint pipeline_tag=zero-shot-classification and model card (verified).

Supported tasks

• text-classification: checkpoint, Transformers, Optimum ONNX, and WinML; evidenced by the concrete checkpoint class, Optimum registry, and WinML inspect/config (verified).
• zero-shot-classification: checkpoint and Transformers; evidenced by the checkpoint pipeline tag and model card (verified).

Model architecture

DistilBertForSequenceClassification
├── DistilBertModel
│   ├── Embeddings (vocab 30,522; positions 512; width 768)
│   └── Transformer encoder stack x 6
│       ├── Multi-head self-attention (12 heads)
│       ├── Feed-forward (768 → 3072 → 768, GELU)
│       └── Residual + LayerNorm
└── Sequence-classification head
    ├── First-token pooling + pre-classifier (768 → 768, ReLU)
    └── Classifier (768 → 3: entailment / neutral / contradiction)

• Source/confidence: pinned config, Transformers 4.57.6 source, and hierarchy-tagged ONNX graph (mapped).

Validation and support evidence

Baseline

• Current main: 5deebd422e95f28fe8fd912ee50ce874710187b3 (feat(optim): add Conv channel-affine folding (#1167)); WinML CLI 0.2.0.
• Recipe-free CPU build: PASS, 36.4 s, 224-node ONNX graph.
• Recipe-free CPU fp32 perf floor: JSON emitted with mean 129.218 ms, p50 130.150 ms, 7.74 samples/s, and +275.83 MB RSS. The host process then exited during native cleanup after a broken auto-installed OpenVINO provider registration; these values are preserved as emitted evidence, not represented as a clean process exit.
• Recipe-free L3 floor: pinned nyu-mll/glue@bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c, mnli/validation_matched, first 10 unshuffled rows, accuracy 1.0. JSON was emitted before the same post-result native cleanup exit.
• Starting auto-config already resolved text classification with int32 input_ids and attention_mask at [1,512], fp32/no quantization, and no optimization or compilation. The PR adds explicit precision coverage and pinned evaluation configuration.
• Optimum probe: VENDOR-ONLY; WinML adds no task registration. Vendor tasks include text classification.

Goal

• Committed Effort: L0 (recipe-only precision coverage declaration).
• Goal ceiling: L3. Success requires both CPU/cpu fp32 and fp16 tuples to build, run concrete L1 perf, match pinned PyTorch logits/decisions at L2, and preserve semantically valid accuracy on the authoritative pinned MNLI split at L3.
• Outcome: L0 implementation effort; no WinML source override was required.

Outcome

• Highest Goal reached: L3 PASS for both required tuples.
• Coverage: CPUExecutionProvider/cpu fp32 and fp16; no deferred required tuples.
• Shipped recipes:
  • examples/recipes/typeform_distilbert-base-uncased-mnli/cpu/cpu/text-classification_fp32_config.json
  • examples/recipes/typeform_distilbert-base-uncased-mnli/cpu/cpu/text-classification_fp16_config.json
• The obsolete flat fp16 and CPU w8a16 declarations are removed. No Python production code and no recipe README are modified.
• Model knowledge is recorded separately in Draft Lane A PR https://github.com/gim-home/ModelKitArtifacts/pull/178.
• Methodology audit: no new methodology friction trigger; existing contracts already cover nested recipe paths, CPU fp32/fp16 breadth, host EP registration failures, post-output native failures, and partial static-analysis exits.

Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	CPU / Precision	Verdict	Evidence
L0	CPUExecutionProvider/cpu fp32	PASS	Build exit 0 in 49.419 s; 224 nodes; 267,823,104-byte external data
L0	CPUExecutionProvider/cpu fp16	PASS	Build exit 0 in 94.360 s; 225 nodes; 133,911,552-byte external data, exactly half fp32
L1	CPUExecutionProvider/cpu fp32	PASS	100 iterations + 10 warmups with named semantic inputs
L1	CPUExecutionProvider/cpu fp16	PASS	100 iterations + 10 warmups with named semantic inputs
L2	CPUExecutionProvider/cpu fp32	PASS	cosine 0.999999999999422; max abs 1.0728836059570313e-05; 6/6 argmax
L2	CPUExecutionProvider/cpu fp16	PASS	cosine 0.9999998731168774; max abs 0.005286037921905518; 6/6 argmax
L3	CPUExecutionProvider/cpu fp32	PASS	pinned MNLI accuracy 0.76 on 100 rows
L3	CPUExecutionProvider/cpu fp16	PASS	pinned MNLI accuracy 0.76 on 100 rows

Perf

EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM Δ	VRAM Δ
CPUExecutionProvider / cpu	fp32	PASS	118.287 ms	117.835 ms	8.45 samples/s	+276.08 MB	0 MB
CPUExecutionProvider / cpu	fp16	PASS	146.715 ms	144.972 ms	6.82 samples/s	+294.72 MB	0 MB

Eval

The checkpoint label map 0=ENTAILMENT, 1=NEUTRAL, 2=CONTRADICTION matches the authoritative GLUE MNLI map by index and case-normalized name, so this L3 comparison is semantically valid.

EP / Device	Precision	Verdict	Dataset / revision / subset	Samples	Metric
CPUExecutionProvider / cpu	fp32	PASS	nyu-mll/glue@bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c, mnli/validation_matched	first 100, unshuffled	accuracy 0.76
CPUExecutionProvider / cpu	fp16	PASS	nyu-mll/glue@bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c, mnli/validation_matched	first 100, unshuffled	accuracy 0.76

Delta

Relative to current-main auto-config, both recipes add the same /eval object: task text-classification; dataset nyu-mll/glue; name mnli; split validation_matched; samples 100; mappings input_column=premise and second_input_column=hypothesis.

• fp32 keeps /quant: null; export, optimizer, compile, and loader blocks are otherwise identical to the current-main auto-config.
• fp16 changes /quant from null to mode=fp16, pins the checkpoint model id/type, and keeps graph I/O types with /quant/fp16_keep_io_types: true.
• Resolved builds only materialize five defaults absent from each recipe: /auto=false, /eval/device=auto, /eval/skip_build=true, /eval/dataset/shuffle=true, and /eval/dataset/seed=42. No optimization is auto-added.
• Relative to the old PR shape, the flat fp16 path moves under cpu/cpu, CPU w8a16 is removed, and mandatory CPU fp32 is added.
• The change is reducible to per-model recipes; recipe-free acceptance on current main already passes. No class-wide code behavior changes.
• The production recipe README remains untouched.

Analyze summary — component level and op level

Both artifacts are ANALYZE-PARTIAL-SUCCESS: complete parseable 11-target JSON was emitted and exit 1 reflects partial/unknown static classifications. This is static compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	embeddings; 6x attention/FFN/normalization; classifier	203/224 mapped (90.62%); 21 exporter/plumbing nodes retained	QNN NPU/GPU partial rule types Div, Erf, Add, Mul affect attention/FFN; no unsupported types
fp16	embeddings; 6x attention/FFN/normalization; classifier	203/225 mapped (90.22%); 22 exporter/plumbing nodes retained, including output Cast	QNN NPU/GPU partial rule types Div, Erf, Add, Mul affect attention/FFN; no unsupported types

HTP metadata agrees across precisions: 66,955,779 parameters, 96 modules, 28 traced modules (29.17%), and 56 execution steps.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	224 ops / 17 types	Add 61; MatMul 48; Mul 24; Reshape 24; Transpose 24; LayerNormalization 13	NvTensorRTRTX GPU and OpenVINO NPU/GPU/CPU: 17 supported; QNN NPU/GPU: 15 supported + 4 partial rule types; none unsupported
fp16	225 ops / 17 types	Add 61; MatMul 48; Mul 24; Reshape 24; Transpose 24; LayerNormalization 13	NvTensorRTRTX GPU and OpenVINO NPU/GPU/CPU: 17 supported; QNN NPU/GPU: 15 supported + 4 partial rule types; none unsupported

Rule-less CUDA, MIGraphX, DML, CPU, and VitisAI rows classify all 17 types unknown; those rows are retained internally and are not runtime-failure claims.

Reproduce commands

$MODEL = 'typeform/distilbert-base-uncased-mnli'
$DATASET_REV = 'bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c'
$OUT = 'temp/distilbert-mnli-review'

winml build -c examples/recipes/typeform_distilbert-base-uncased-mnli/cpu/cpu/text-classification_fp32_config.json -m $MODEL -o "$OUT/fp32" --ep cpu --device cpu --precision fp32 --no-analyze --no-compile --rebuild --no-color
winml build -c examples/recipes/typeform_distilbert-base-uncased-mnli/cpu/cpu/text-classification_fp16_config.json -m $MODEL -o "$OUT/fp16" --ep cpu --device cpu --precision fp16 --no-analyze --no-compile --rebuild --no-color

$env:DISTILBERT_REVIEW_OUT = $OUT
@'
import os
import numpy as np
from transformers import AutoTokenizer

model = "typeform/distilbert-base-uncased-mnli"
revision = "cfa538a0fddbbd978fefe8966c1aeff7ad409c90"
tokenizer = AutoTokenizer.from_pretrained(model, revision=revision)
encoded = tokenizer(
  "A soccer game with multiple males playing.",
  "Some men are playing a sport.",
  padding="max_length",
  truncation=True,
  max_length=512,
  return_tensors="np",
)
np.savez(
  os.path.join(os.environ["DISTILBERT_REVIEW_OUT"], "semantic_inputs.npz"),
  input_ids=encoded["input_ids"].astype(np.int32),
  attention_mask=encoded["attention_mask"].astype(np.int32),
)
'@ | uv run python -

winml perf -m "$OUT/fp32/model.onnx" --ep cpu --device cpu --precision fp32 --input-data "$OUT/semantic_inputs.npz" --iterations 100 --warmup 10 --memory --output "$OUT/perf-fp32.json" --overwrite --format json --no-color
winml perf -m "$OUT/fp16/model.onnx" --ep cpu --device cpu --precision fp16 --input-data "$OUT/semantic_inputs.npz" --iterations 100 --warmup 10 --memory --output "$OUT/perf-fp16.json" --overwrite --format json --no-color

@'
from __future__ import annotations
import argparse, json
from pathlib import Path
import numpy as np
import onnxruntime as ort
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL = "typeform/distilbert-base-uncased-mnli"
REVISION = "cfa538a0fddbbd978fefe8966c1aeff7ad409c90"
PAIRS = [
  ("A soccer game with multiple males playing.", "Some men are playing a sport."),
  ("A person on a horse jumps over a broken airplane.", "A person is outdoors on an animal."),
  ("Two women are embracing while holding to-go packages.", "The sisters are hugging goodbye."),
  ("A man is inspecting the uniform of a figure in some East Asian country.", "The man is sleeping."),
  ("A black race car starts up in front of a crowd of people.", "A man is driving down a lonely road."),
  ("A smiling costumed woman is holding an umbrella.", "A happy woman in a fairy costume holds an umbrella."),
]

def cosine(a, b):
  x, y = a.astype(np.float64).reshape(-1), b.astype(np.float64).reshape(-1)
  return float(np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y)))

parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True)
parser.add_argument("--precision", required=True)
parser.add_argument("--output", required=True)
args = parser.parse_args()
tokenizer = AutoTokenizer.from_pretrained(MODEL, revision=REVISION)
pt = AutoModelForSequenceClassification.from_pretrained(MODEL, revision=REVISION).eval()
session = ort.InferenceSession(args.model, providers=["CPUExecutionProvider"])
rows, all_pt, all_onnx = [], [], []
for premise, hypothesis in PAIRS:
  encoded = tokenizer(premise, hypothesis, padding="max_length", truncation=True, max_length=512, return_tensors="pt")
  with torch.inference_mode():
    pt_logits = pt(**encoded).logits.detach().cpu().numpy()
  inputs = {"input_ids": encoded["input_ids"].numpy().astype(np.int32), "attention_mask": encoded["attention_mask"].numpy().astype(np.int32)}
  onnx_logits = session.run(["logits"], inputs)[0]
  all_pt.append(pt_logits); all_onnx.append(onnx_logits)
  rows.append({"pt_argmax": int(pt_logits.argmax()), "onnx_argmax": int(onnx_logits.argmax()), "argmax_match": bool(np.array_equal(pt_logits.argmax(-1), onnx_logits.argmax(-1))), "cosine": cosine(pt_logits, onnx_logits), "max_abs": float(np.max(np.abs(pt_logits - onnx_logits)))})
x, y = np.concatenate(all_pt), np.concatenate(all_onnx)
result = {"model_id": MODEL, "revision": REVISION, "precision": args.precision, "providers": session.get_providers(), "sample_count": len(rows), "label_order": {"0": "ENTAILMENT", "1": "NEUTRAL", "2": "CONTRADICTION"}, "aggregate": {"cosine": cosine(x, y), "max_abs": float(np.max(np.abs(x - y))), "argmax_matches": sum(r["argmax_match"] for r in rows), "argmax_total": len(rows)}, "rows": rows}
Path(args.output).write_text(json.dumps(result, indent=2), encoding="utf-8")
print(json.dumps(result, indent=2))
'@ | Set-Content -Encoding utf8 "$OUT/compare_pytorch.py"

uv run python "$OUT/compare_pytorch.py" --model "$OUT/fp32/model.onnx" --precision fp32 --output "$OUT/parity-fp32.json"
uv run python "$OUT/compare_pytorch.py" --model "$OUT/fp16/model.onnx" --precision fp16 --output "$OUT/parity-fp16.json"

winml eval --model "$OUT/fp32/model.onnx" --model-id $MODEL --dataset nyu-mll/glue --dataset-revision $DATASET_REV --dataset-name mnli --task text-classification --device cpu --ep cpu --samples 100 --split validation_matched --no-shuffle --streaming --column input_column=premise --column second_input_column=hypothesis --output "$OUT/eval-fp32.json" --overwrite --format json --no-color
winml eval --model "$OUT/fp16/model.onnx" --model-id $MODEL --dataset nyu-mll/glue --dataset-revision $DATASET_REV --dataset-name mnli --task text-classification --device cpu --ep cpu --samples 100 --split validation_matched --no-shuffle --streaming --column input_column=premise --column second_input_column=hypothesis --output "$OUT/eval-fp16.json" --overwrite --format json --no-color

$env:WINMLCLI_RULES_DIR = 'C:\path\to\populated\rules'
winml analyze --model "$OUT/fp32/model.onnx" --ep all --device all --htp-metadata "$OUT/fp32/export_htp_metadata.json" --format json --no-color | Set-Content -Encoding utf8 "$OUT/analyze-fp32.json"
winml analyze --model "$OUT/fp16/model.onnx" --ep all --device all --htp-metadata "$OUT/fp16/export_htp_metadata.json" --format json --no-color | Set-Content -Encoding utf8 "$OUT/analyze-fp16.json"
Remove-Item Env:WINMLCLI_RULES_DIR

[kujin66]

Reviewer verdict: APPROVE fallback comment / Lane B model PR is ready for maintainer review.

Checked items:

• PR body contains the 9-item adding-model-support report.
• Diff scope is L0 recipe-only: README plus two typeform/distilbert-base-uncased-mnli recipe JSON files.
• Branch is based on current origin/main 38767add6f91c7b10b6394fae3af6f437e02effd; git rev-list --count 38767add6f91c7b10b6394fae3af6f437e02effd..origin/main returned 0.
• CI is green: Analyze, CodeQL, CLA, lint, and all test jobs passed.
• Goal L0 evidence is present: baseline auto-config and recipe builds produced loadable ONNX artifacts with input_ids / attention_mask [1,512] and logits [1,3].
• Goal L1 evidence is present for CPU: fp16 perf mean 189.806 ms / 5.27 samples/sec; w8a16 perf mean 279.577 ms / 3.58 samples/sec.
• Precision evidence is present: fp16 artifact has 109/115 FLOAT16 initializers; w8a16 artifact has QuantizeLinear=219, DequantizeLinear=328.
• PR is mergeable according to GitHub. GitHub-side approval metadata may not be set if the same account cannot approve its own pull request.

Methodology note:

• No new methodology friction observed. Existing _meta-057 was applied for artifact-level precision validation, and _meta-056 covers the analyze parquet host gap.
• Local Skill knowledge was updated as model_knowledge/distilbert.json / distilbert-001; Skill Lane A files are intentionally not included in this Lane B winml-cli PR.

[ssss141414]

APPROVE — independent final-head review

Reviewed SHA: 33a018c68703ba57a6181557f7fa7283e2db51a9.

• Scope: current-main merge preserves the contributor history; the merge-base diff contains only the nested CPU fp32/fp16 recipe JSON files. The production recipe README and Python source are unchanged.
• Baseline: the cited main SHA 5deebd422e95f28fe8fd912ee50ce874710187b3 still equals origin/main (0 commits ahead at final review).
• Fresh final-head builds: fp32 and fp16 both emitted Build complete; graphs are 224/225 nodes with int32 [1,512] named inputs and [1,3] logits. fp16 has 109 FLOAT16 initializers and 133,911,552-byte external data versus fp32's 267,823,104 bytes.
• Independent L1 spot check: fp32 mean/p50 117.904/117.767 ms, 8.48 samples/s, +275.12 MB RSS; fp16 mean/p50 139.600/138.830 ms, 7.16 samples/s, +294.20 MB RSS. Both are within the tester's expected host variance.
• Independent L2 rerun exactly reproduced fp32 cosine 0.999999999999422, max-abs 1.0728836059570312e-05, 6/6 argmax and fp16 cosine 0.9999998731168774, max-abs 0.005286037921905518, 6/6 argmax.
• Independent L3 rerun exactly reproduced accuracy 0.76 for both artifacts on pinned nyu-mll/glue@bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c, mnli/validation_matched, first 100 unshuffled rows.
• Analyze: independent fp16 scan reproduced 225 operators / 17 types and all 11 requested target records when JSON was captured from stdout. The first review pass found that --output retained only the first target; the explainer repaired the public commands to capture stdout and added self-contained semantic-input and L2 harness generation. The PR body, not the stale earlier comment, is the source of truth.
• Quality: fresh review checkout passed uv run ruff check src/ tests/ and uv run mypy -p winml.modelkit (409 source files). Exact-head GitHub checks all completed successfully: CodeQL/Analyze, lint, analyze/models/optim/commands/remaining tests, and CLA. Producer-side exact-head partitions also passed: 1,351 models tests and 2,373 commands tests.
• Feedback reconciliation: 0 line comments, 1 conversation comment, 0 formal reviews, 0 review threads, and 0 unresolved threads. The old conversation comment is explicitly superseded because it describes the prior flat fp16/w8a16/README state and old measurements.
• Knowledge: separate Draft Lane A PR https://github.com/gim-home/ModelKitArtifacts/pull/178 contains only model_knowledge/distilbert.json; no methodology friction trigger was observed.
• Shipment: PR remains Draft and retains the actual model-scale-by-skill label.

Highest Goal verdict: L3 PASS for CPU/cpu fp32 and fp16. No deferred required tuples and no unresolved failures.