Concurrency

[emi-dm]

Hi! I want to kindly ask how is concurrency managed by docker model runner when you are using VLLM as a backend.

Thanks!

[dmedovich]

Hi! I want to kindly ask how is concurrency managed by docker model runner when you are using VLLM as a backend.

Thanks!

Hi! Docker Model Runner does not serialize inference requests when using the vLLM backend.

For each model/backend/mode combination, Model Runner starts one vLLM server process. Concurrent requests for the same model are proxied to that process concurrently. Model Runner's reference counting only prevents the runner from being unloaded while requests are active—it does not impose a request-level concurrency limit.

The actual inference concurrency is therefore managed by vLLM itself through its scheduler and continuous batching. The main controls are:

• --max-num-seqs: maximum number of sequences processed in one scheduler iteration.
• --max-num-batched-tokens: token budget for one iteration.
• Available KV-cache/GPU memory, context lengths, and request sizes.

Requests that cannot be scheduled immediately remain queued inside vLLM. By default, Docker Model Runner does not override these limits, so the defaults of the bundled vLLM version are used.

You can tune them per model, for example:

docker model configure my-model -- \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192

Optionally, GPU memory allocation can also be configured:

docker model configure --gpu-memory-utilization 0.9 my-model

Different models normally use separate runner processes. Model Runner supports several runner slots, but that slot count limits loaded model processes—not concurrent requests to one model. Running multiple vLLM models concurrently may require lowering their GPU-memory utilization to avoid contention.

So, in short: Docker Model Runner handles model-process lifecycle and routing, while vLLM handles request queueing, scheduling, and continuous batching.

An important detail is that --max-num-seqs limits the number of sequences processed concurrently, not the number of accepted HTTP requests. Additional requests can wait in vLLM's queue.

[emi-dm]

Thank you!