[Regression] 2026.3.1-gpu hangs with 200% CPU in sched_yield / libopencl-clang on dynamic shape embeddings (working fine in 2026.3-gpu)

[xy111a]

Describe the bug
In openvino/model_server:2026.3.1-gpu, serving embedding models on Intel iGPU with multiple workers (--rest_workers > 1, e.g. 4 or 16) causes OVMS to hang indefinitely when handling variable-length texts (or concurrent dynamic shape requests).

The container CPU stays pegged at ~200% indefinitely (two threads busy-spinning at 100% each in sched_yield) and stops responding to HTTP requests.

Attaching GDB to the hanging process reveals a spinlock deadlock between libigdrcl.so (Intel OpenCL runtime) and libopencl-clang2.so.16:

• Thread 8 (LWP 96) is stuck inside libopencl-clang2.so.16 (Compile()).
• Worker threads (e.g. Thread 22 LWP 82) are stuck in a busy spinloop in sched_yield() inside libigdrcl.so.

To Reproduce
Steps to reproduce the behavior:

1. Model: OpenVINO/Qwen3-Embedding-0.6B-int8-ov (or any dynamic shape embedding model).
2. Launch OVMS on Intel GPU with multiple workers:

   docker run --device /dev/dri/renderD128:/dev/dri/renderD128 \
     -p 9200:9200 \
     openvino/model_server:2026.3.1-gpu \
     --model_repository_path /models \
     --source_model OpenVINO/Qwen3-Embedding-0.6B-int8-ov \
     --task embeddings \
     --pooling LAST \
     --target_device GPU \
     --rest_workers 4 \
     --rest_port 9200 \
     --model_name qwen3-embedding-0.6b

3. Send variable-length text requests in quick succession or concurrently:

   curl -X POST http://localhost:9200/v3/embeddings \
     -H "Content-Type: application/json" \
     -d '{"model": "qwen3-embedding-0.6b", "input": "testing variable length dynamic shape sentence"}'

4. OVMS hangs, container CPU pins at 200%, and requests never return.

Expected behavior
The server should properly synchronize dynamic shape OpenCL compilation without deadlocking or busy-spinning in sched_yield.

GDB Backtrace
GDB backtrace of the spinning process in 2026.3.1-gpu:

Thread 8 (LWP 96 - Compilation):

#0  0x00007fc... in ... from /usr/local/lib/libopencl-clang2.so.16
#1  0x00007fc... in Compile () from /usr/local/lib/libopencl-clang2.so.16
#2  0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so

Thread 22 (LWP 82 - Worker spinning in sched_yield):

#0  0x00007fc... in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#2  0x00007fc... in ... from /ovms/lib/libopenvino_intel_gpu_plugin.so

Environment & Hardware Configuration

1. Host CPU: 12th Gen Intel Core i5-1235U (10 cores, 12 threads)
2. GPU: Intel Iris Xe Graphics (Alder Lake-UP3 GT2, 80 EU, PCI ID [8086:46a8])
3. RAM: 64 GB
4. Host OS: Linux 6.6.x (Debian 12)
5. Model: OpenVINO/Qwen3-Embedding-0.6B-int8-ov
6. Docker Image: openvino/model_server:2026.3.1-gpu (Broken) vs openvino/model_server:2026.3-gpu (Working)

Root Cause Analysis & Verified Workarounds
The root trigger is that when multiple worker threads (--rest_workers > 1) handle concurrent embedding requests with variable-length text, multiple threads simultaneously call into the Intel OpenCL driver (libigdrcl.so) and trigger JIT compilation in libopencl-clang2.so.16, resulting in a spinlock deadlock (sched_yield). Furthermore, in 2026.3.1-gpu, once dynamic compilation is triggered, background compiler threads can remain permanently spinning at 200% CPU even after HTTP calls complete.

Workarounds:

1. Reverting to openvino/model_server:2026.3-gpu with --rest_workers 1 and --truncate true: requests execute smoothly at ~40ms latency and CPU drops back to 0.07% immediately upon completion.
2. Under 2026.3.1-gpu, setting --rest_workers 1 and --truncate true serializes requests at the HTTP layer, but libopencl-clang threads may still leak CPU loops after JIT compilation.