PP-DocLayoutV3 FP32 ONNX falls back entirely to CPU with VitisAI EP BF16 compile on Ryzen AI MAX+ 395

[AIwork4me]

Summary

I am an AMD Ryzen AI MAX+ 395 AI PC customer trying to build a local Document AI pipeline on Ryzen AI. The layout detection stage uses PaddleOCR's PP-DocLayoutV3 model. I followed the intended Windows/Ryzen AI route:

FP32 ONNX -> VitisAIExecutionProvider -> BF16 compile -> NPU + CPU hybrid partition -> cached compiled model

However, PP-DocLayoutV3 currently falls back entirely to CPU on my machine. The same environment successfully compiles and runs a ResNet50 FP32 ONNX control model on the NPU via VitisAI EP BF16 compile, so this does not look like a general local environment setup problem. Download PP-DocLayoutV3.

This is a blocking issue for Document AI application development on Ryzen AI MAX+ 395. Layout detection is a core stage for OCR/VLM document pipelines, and without NPU execution the end-to-end latency and value proposition of the AI PC platform are substantially reduced.

Hardware / OS environment

• Machine: ASUS ProArt PX13 HN7306EA
• CPU/APU: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
• CPU cores / threads: 16 cores / 32 logical processors
• Integrated GPU: AMD Radeon(TM) 8060S Graphics
• GPU driver: 32.0.22032.6002
• NPU device: NPU Compute Accelerator Device
• NPU PCI ID: PCI\VEN_1022&DEV_17F0&SUBSYS_20CF1043&REV_...
• OS: Windows 11 Home Chinese, 64-bit
• OS version/build: 10.0.26200, build 26200

Software environment

• Python: 3.12.13
• ONNX Runtime: 1.24.5
• ONNX: 1.17.0
• WinML Python package: windowsml from local venv
• AMD WinML GPU EP package:
  • MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_1.8.55.0_x64__8wekyb3d8bbwe
  • EP library: migraphx-ep.dll
• AMD WinML NPU EP package:
  • MicrosoftCorporationII.WinML.AMD.NPU.EP.1.8_1.8.59.0_x64__8wekyb3d8bbwe
  • EP library: onnxruntime_vitisai_ep.dll
• Vitis AI / AI Engine compiler observed during the ResNet50 control run:
  • AI Engine Compiler Version 2026.1 (windows64-bit)
  • SW Build 4ccfd0b on 2026-03-16-20:43:50

ort.get_ep_devices() discovers:

CPUExecutionProvider
DmlExecutionProvider
MIGraphXExecutionProvider
VitisAIExecutionProvider

Model under test: PP-DocLayoutV3 ONNX

The model is the official PaddleOCR/PaddleX PP-DocLayoutV3 layout detection model, exported to ONNX outside Windows with a compatible Paddle2ONNX toolchain, then copied back to the Ryzen AI Windows machine.

ONNX model properties:

file: inference.onnx
size: 131,731,131 bytes
IR version: 8
opset: ai.onnx 17
node count: 6440
initializer count: 0

Inputs:

im_shape      tensor(float) [DynamicDimension.0, 2]
image         tensor(float) [DynamicDimension.1, 3, 800, 800]
scale_factor  tensor(float) [DynamicDimension.2, 2]

Outputs:

fetch_name_0  tensor(float) [DynamicDimension.3, 7]
fetch_name_1  tensor(int64) [DynamicDimension.4]
fetch_name_2  tensor(int64) [DynamicDimension.5, 200, 200]

VitisAI EP BF16 config used

I used an explicit VAIML/BF16 compile route. The most important variant used this config:

{
  "passes": [
    {
      "name": "init",
      "plugin": "vaip-pass_init"
    },
    {
      "name": "vaiml_partition",
      "plugin": "vaip-pass_vaiml_partition",
      "vaiml_config": {
        "optimize_level": 2,
        "preferred_data_storage": "auto",
        "enable_f32_to_bf16_conversion": true
      }
    }
  ],
  "target": "VAIML",
  "targets": [
    {
      "name": "VAIML",
      "pass": ["init", "vaiml_partition"]
    }
  ]
}

Provider options:

provider_options = {
    "config_file": "<path-to-vai_ep_config.json>",
    "cache_dir": "<cache-dir>",
    "cache_key": "bf16_auto_o2_legacy_force",
    "enable_cache_file_io_in_mem": "0",
    "ai_analyzer_visualization": "true",
    "ai_analyzer_profiling": "true",
}

I create the ONNX Runtime session through WinML EP devices, i.e. SessionOptions.add_provider_for_devices() with VitisAIExecutionProvider first and CPUExecutionProvider as fallback.

Reproduction steps

1. Install the Windows AMD WinML EP packages so that VitisAIExecutionProvider is visible through ort.get_ep_devices().
2. Prepare/export PP-DocLayoutV3 to FP32 ONNX. In my case the valid ONNX file is inference.onnx, 131,731,131 bytes.
3. Register Windows ML EP libraries from Python:

import onnxruntime as ort
from windowsml import EpCatalog

with EpCatalog() as catalog:
    for provider in catalog.find_all_providers():
        provider.ensure_ready()
        if provider.name in ("VitisAIExecutionProvider", "MIGraphXExecutionProvider"):
            ort.register_execution_provider_library(provider.name, str(provider.library_path))

4. Build VitisAI EP session options:

ep_devices = list(ort.get_ep_devices())
vitis = [d for d in ep_devices if d.ep_name == "VitisAIExecutionProvider"]
cpu = [d for d in ep_devices if d.ep_name == "CPUExecutionProvider"]

so = ort.SessionOptions()
so.enable_profiling = True
so.profile_file_prefix = "ort_profile.json"
so.add_provider_for_devices(vitis, provider_options)
so.add_provider_for_devices(cpu, {})

sess = ort.InferenceSession("inference.onnx", sess_options=so)

5. Run a dummy inference with valid PP-DocLayoutV3 input shapes:

feeds = {
    "image": np.random.randn(1, 3, 800, 800).astype(np.float32),
    "im_shape": np.array([[800, 800]], dtype=np.float32),
    "scale_factor": np.array([[1.0, 1.0]], dtype=np.float32),
}
outputs = sess.run(None, feeds)
profile_file = sess.end_profiling()

6. Inspect the ORT profile and the VitisAI cache artifacts.

Actual PP-DocLayoutV3 result

The session is created with both providers:

session_providers = ["VitisAIExecutionProvider", "CPUExecutionProvider"]

But the runtime profile shows all nodes on CPU:

profile_provider_events = {
  "CPUExecutionProvider": {
    "count": 1346,
    "dur_us": ~340000
  }
}

There are no VitisAIExecutionProvider kernel events for PP-DocLayoutV3.

The VAIML partition result indicates that no usable NPU overlay/subgraph was formed:

unsupported_ops_count = 1262
ops_per_overlay = {}
aie_required_clusters = null

I tried several BF16/VAIML variants and got the same result:

Variant	Session providers	Runtime provider events	Unsupported ops	ops_per_overlay
bf16_auto_o1	VitisAI + CPU	CPU only, 1346 events	1262	{}
bf16_vectorized_o1	VitisAI + CPU	CPU only, 1346 events	1262	{}
bf16_unvectorized_o1	VitisAI + CPU	CPU only, 1346 events	1262	{}
bf16_auto_o2_legacy_force	VitisAI + CPU	CPU only, 1346 events	1262	{}

Example unsupported op names include many Add.* nodes, e.g. Add.0, Add.10, Add.102, Add.104, Add.106, etc.

Control experiment: ResNet50 proves the local NPU/BF16 path works

To rule out a local setup issue, I generated a standard ResNet50-style FP32 ONNX control model and ran it through the same Windows/ORT/VitisAI EP path, with the same BF16 compile approach.

ResNet50 control model:

FP32 ONNX
input: tensor(float) [1, 3, 224, 224]
model size: 102,030,656 bytes
nodes: 122
initializers: 55

Result:

AI Engine Compiler: Compilation Complete
session_providers = ["VitisAIExecutionProvider", "CPUExecutionProvider"]
profile_provider_counts = {"VitisAIExecutionProvider": 3}
profile_provider_dur_us = {"VitisAIExecutionProvider": 19098}
run_times_s = [0.0082, 0.0056, 0.0059]

The compiled model cache was also created successfully:

.rai
subgraphs_cache.json
partition_io_shapes.json
vaiml_par_0/0/ctrl.elf
vaiml_par_0/0/unified-4x4.xclbin
vaiml_par_0/0/wts32.bin

Cache hit test:

first session create: 503.555 s
cached session create: 0.413 s
cached profile_provider_counts = {"VitisAIExecutionProvider": 2}

This confirms that the local VitisAI EP, BF16 compilation, NPU runtime, and cache path are operational on this Ryzen AI MAX+ 395 system.

Expected behavior

For PP-DocLayoutV3 FP32 ONNX, I expected VitisAI EP to do at least one of the following:

1. Compile the supported parts of the graph into one or more NPU subgraphs, with unsupported ops falling back to CPU;
2. Produce actionable diagnostics explaining why no NPU partition is possible;
3. Provide a documented export/conversion workaround for this model class so Document AI layout detection can run on Ryzen AI NPUs.

Actual behavior

PP-DocLayoutV3 creates a VitisAI EP session but runs entirely on CPU. The analyzer reports 1262 unsupported ops and an empty ops_per_overlay, so no NPU subgraph is formed.

Why this matters

I bought and am actively developing on an AMD Ryzen AI MAX+ 395 AI PC specifically to run local AI workloads. Document AI is one of the most important practical use cases for an AI PC: OCR, layout detection, table/figure/text block detection, and VLM-based document understanding.

PP-DocLayoutV3 is the layout model used by PaddleOCR/PaddleOCR-VL style pipelines. If this model cannot use the NPU, Document AI applications on Ryzen AI lose a major performance and efficiency advantage. The ResNet50 control result shows the platform can execute BF16-compiled ONNX workloads on the NPU, but this real Document AI model currently cannot benefit from it.

Could AMD please investigate whether this is due to unsupported ONNX ops, graph patterns emitted by Paddle2ONNX, dynamic shapes, missing BF16 lowering coverage, or a VitisAI EP partitioning limitation? A model-specific recommendation or compiler/runtime fix would be very valuable to customers building Document AI applications on Ryzen AI MAX+ 395 systems.

I am happy to provide the full profile JSON files, cache directories, and the exact ONNX model metadata if that helps triage.

[AIwork4me]

Additional local reproduction: PP-DocLayoutV3 reaches VAIML partitioning after graph cleanup, but AIE backend fails at BF16 placement

I continued the investigation on the same Ryzen AI MAX+ 395 AIPC and can now provide a more precise, reproducible root-cause chain. The short version is: the local NPU/ORT setup is working, a ResNet50 FP32 -> BF16 control model runs on VitisAIExecutionProvider, but PP-DocLayoutV3 still cannot produce a usable AIE runtime artifact. After removing front-end blockers in the ONNX graph, VAIML recognizes ~99.1% of the model GOPs, then the AIE backend fails during L1 placement for a MatMul/GEMM layer and ORT falls back to CPU.

Environment

• Hardware: AMD Ryzen AI MAX+ 395 AIPC
• NPU detected by AMD quicktest: STX
• OS: Windows
• Ryzen AI Software: C:\Program Files\RyzenAI\1.7.1
• Conda env used for all tests: ryzen-ai-1.7.1
• Python executable: C:\Users\rocm\miniconda3\envs\ryzen-ai-1.7.1\python.exe
• ONNX Runtime in this env: 1.23.3.dev20260320
• ORT providers available:
  • VitisAIExecutionProvider
  • DmlExecutionProvider
  • CPUExecutionProvider

AMD quicktest passes:

conda run -n ryzen-ai-1.7.1 python "C:\Program Files\RyzenAI\1.7.1\quicktest\quicktest.py"

Output:

Detected NPU type: STX
Test Finished

The STX quicktest creates ORT sessions with:

providers = ["VitisAIExecutionProvider"]
provider_options = [{}]
session = ort.InferenceSession(model, providers=providers, provider_options=provider_options)

Control result: ResNet50 FP32 -> BF16 works on NPU

As a control, a ResNet50 FP32 ONNX model with BF16 compile works on the same environment. It generates normal AIE runtime artifacts including:

• ctrl.elf
• unified-4x4.xclbin
• .rai
• wts32.bin

ORT profiling for the control model shows VitisAIExecutionProvider runtime kernel events, for example:

"profile_provider_counts": {
  "VitisAIExecutionProvider": 3
}

So this does not look like a broken NPU driver, broken conda env, or wrong ORT provider setup.

Original PP-DocLayoutV3 ONNX behavior

Model:

PP-DocLayoutV3 FP32 ONNX
Inputs:
  im_shape: [N, 2], float32
  image: [N, 3, 800, 800], float32
  scale_factor: [N, 2], float32
Outputs:
  fetch_name_0: [?, 7], float32
  fetch_name_1: [?], int32
  fetch_name_2: [?, 200, 200], int32

The original exported graph has:

• opset 17
• 6440 nodes
• 0 graph initializers
• 2519 Constant nodes
• 118 Conv
• 123 MatMul
• 105 BatchNormalization
• 18 GridSample
• 3 Where
• 3 TopK

Running it with VitisAIExecutionProvider creates a session, but the ORT profile contains only CPUExecutionProvider kernel events.

Reproduction command used for PP-DocLayoutV3

All tests below were run with conda run -n ryzen-ai-1.7.1:

$env:PYTHONPATH='src'
conda run -n ryzen-ai-1.7.1 python -m paddleocr_vl_local.vitisai_layout_probe `
  --input test_images\handwrite_ch_demo.png `
  --model models\PP-DocLayoutV3-onnx\inference.onnx `
  --output logs\pp_doclayoutv3_vitisai_171_default `
  --enable-profiling `
  --runs 1 `
  --log-severity 0

Observed runtime profile:

CPUExecutionProvider: all kernel events
VitisAIExecutionProvider: 0 kernel events

With BF16 config and compile cache enabled:

conda run -n ryzen-ai-1.7.1 python -m paddleocr_vl_local.vitisai_layout_probe `
  --input test_images\handwrite_ch_demo.png `
  --model models\PP-DocLayoutV3-onnx\inference.onnx `
  --output logs\pp_doclayoutv3_dynamic_vitisai_171_bf16_force `
  --config-file configs\vitisai-bf16-ppdoclayout-stx-o2-force.json `
  --cache-dir logs\pp_doclayoutv3_dynamic_vitisai_171_bf16_force\cache `
  --cache-key ppdoclayoutv3_dynamic_bf16_force `
  --enable-cache-file-io-in-mem 0 `
  --enable-profiling `
  --runs 1 `
  --log-severity 1

The dynamic ONNX graph fails in VAIML front-end conversion:

ERROR: [VAIML-COMPILE 1000] loc("Cast.82"): 'onnx.Cast' op requires the same shape for all operands and results
Warning for operation onnx.Reshape: [Shape inference, dim 0] the inferred dim (300) is different from the existing dim (1). Use the existing dim instead.
ERROR: [VAIML-COMPILE 1006] Failed to convert ONNX to MLIR: lowerONNXToONNXMLIR

Graph cleanup performed to isolate the backend problem

To separate model-export front-end issues from backend AIE compilation issues, I generated a semantically equivalent VitisAI-friendly variant for fixed 1x3x800x800 layout input:

1. Fixed dynamic input/output shapes.
2. Converted Constant weight nodes to graph initializers.
3. Ran ONNX simplification.
4. Rewrote Where(cond, A, B) into float-mask arithmetic:

Where(cond, A, B) -> mask * A + (1 - mask) * B

This specifically avoids exposing ONNX BOOL tensors to the VAIML front-end. Without this rewrite, the static graph fails with:

F... graph.cpp:1058] unknown type 9 name=p2o.pd_op.cast.2.0

Here type 9 is ONNX BOOL.

The prepared graph has:

• 1183 nodes
• 510 initializers
• 3 Where nodes removed/replaced
• no bool initializers left in the tested path

VAIML front-end result after cleanup

After the cleanup, VAIML no longer fails immediately on dynamic shape / BOOL / Where. It identifies most of the model as supported:

----- Preliminary Summary of WCR EP VAIML Pass ------
Model data type: float32
Device data type: bfloat16
Number of operators in the model: 1209
GOPs of the model: 181.898
Number of operators supported by VAIML: 1157 (95.699%)
GOPs supported by VAIML: 180.261 / 181.898 (99.100%)
Number of subgraphs supported by VAIML: 11

For the main partition:

{
  "aie_partition_call_order": ["forward_outlined_part_0"],
  "num_aie_partitions": 1
}

and:

{
  "offload_map": {
    "AIE": 100,
    "CPU": 0
  }
}

This is important: after graph cleanup, the PP-DocLayoutV3 graph is largely accepted by the VAIML front end. The remaining failure happens later in the AIE backend.

Backend failure with BF16 compile

Using optimize_level=1 + enable_f32_to_bf16_conversion=true:

ERROR: [VAIML-BE-SKGEN 11005] Flexible L1 placement failure in layer 105 : L-105
ERROR: [VAIML-COMPILE 1000] vaiml_compile_v4 error: Flexible L1 placement failure in layer 105 : L-105

The backend dot files identify this as:

MatMul.2/MatMulAddFusion
IncoreChain Kernels Type: Gemm
ConstWTSGEMM
DataType: BF16

Using a config closer to the default RyzenAI 1.7.1 STX vaip_config.json:

{
  "device": "stx",
  "optimize_level": 2,
  "preferred_data_storage": "auto",
  "enable_f32_to_bf16_conversion": true,
  "fe_experiment": "enable-outer-garbage-propagation-through-reshape=1",
  "aiecompiler_args": " -Xelfgen=-j16 "
}

The failure changes layer number but remains the same class of backend failure:

ERROR: [VAIML-BE-SKGEN 11005] Flexible L1 placement failure in layer 93 : L-93
ERROR: [VAIML-COMPILE 1000] vaiml_compile_v4 error: Flexible L1 placement failure in layer 93 : L-93

No complete NPU artifacts are generated for PP-DocLayoutV3. In contrast to the ResNet50 control, I do not get a usable ctrl.elf, unified-4x4.xclbin, or .rai for this model. The final ORT profile still shows only CPU runtime kernels:

Counter({'CPUExecutionProvider': 1137})

Why this matters

PP-DocLayoutV3 is a key layout model for PaddleOCR-VL / Document AI pipelines. On this Ryzen AI MAX+ 395 AIPC, layout detection is exactly the kind of workload that should benefit from local NPU acceleration. At the moment, even after making the ONNX graph much more VAIML-friendly and reaching 99.1% supported GOPs in the front-end summary, the AIE backend cannot produce a runnable BF16 NPU partition due to L1 placement failure in a MatMul/GEMM layer.

This blocks a lightweight no-Paddle Document AI runtime from using the NPU for layout detection and makes the local AI PC development experience much worse than expected.

Requested help from AMD

Could the AMD Ryzen AI team please help with one of the following?

1. Confirm whether PP-DocLayoutV3-style FP32 ONNX -> BF16 VAIML compilation is expected to work on STX / Ryzen AI 1.7.1.
2. Investigate the AIE backend Flexible L1 placement failure for the MatMul/GEMM layer in this model.
3. Provide a config option or recommended workflow to exclude the failing MatMul/GEMM layer from AIE partitioning and let it fall back to CPU, while keeping the rest of the supported subgraph on NPU.
4. Provide guidance on the recommended export boundary for this model, e.g. whether PP-DocLayoutV3 should be split into backbone/neck/head on NPU and decode/mask/postprocess on CPU.

I can provide the prepared ONNX variant, VAIML cache directory, preliminary-vaiml-pass-summary.txt, partition-info.json, fail_safe_summary.json, and backend dot evidence for L-93 / L-105 if useful.

ppdoclayoutv3_ryzenai_issue374_evidence_slim.zip