LLMs

Utilities for launching a local vLLM service and calling it through an async OpenAI-compatible Python client.

Overview

This repository contains:

• start_vllm.sh: a configurable launch script for a local vLLM server.
• vllm_api.py: VLLM_API, a thin async wrapper around openai.AsyncOpenAI.
• test/test_vllm.py: an interactive multi-turn chat smoke test.

The client sends requests to a vLLM endpoint exposed with the OpenAI-compatible API, typically at http://127.0.0.1:8331/v1.

Requirements

The current workflow assumes an existing Conda environment named vllm under:

/u01/yuzhaoxin/miniconda3

The environment should include vllm, openai, and their runtime dependencies. No requirements.txt or pyproject.toml is currently provided.

Start the vLLM Server

Run with defaults:

bash start_vllm.sh

Useful environment overrides:

MODEL_PATH=/u01/yuzhaoxin/plms/Qwen3-32B \
SERVED_MODEL_NAME=Qwen3-32B \
PORT=8331 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
TENSOR_PARALLEL_SIZE=8 \
bash start_vllm.sh

The script validates that CUDA_VISIBLE_DEVICES matches TENSOR_PARALLEL_SIZE.

Interactive Test

After the server starts, run:

python test/test_vllm.py Qwen3-32B 8331

Commands inside the chat:

• /exit or /quit: leave the session.
• /clear: reset conversation history.

Python Usage

import asyncio

from LLMs.vllm_api import VLLM_API


async def main() -> None:
    api = VLLM_API(
        base_url="http://127.0.0.1:8331/v1",
        model="Qwen3-32B",
        max_tokens=512,
        temperature=0.7,
    )

    result = await api.chat([
        {"role": "user", "content": "Explain vLLM in one paragraph."}
    ])
    print(result["content"])


asyncio.run(main())

Concurrency Model

VLLM_API.chat() sends one request per call. It does not implement batching, request queues, or concurrency limits internally. Because it is an async coroutine, callers can run multiple requests concurrently:

results = await asyncio.gather(
    api.chat(messages_a, request_id="a"),
    api.chat(messages_b, request_id="b"),
)

Server-side GPU parallelism is controlled by vLLM launch settings such as CUDA_VISIBLE_DEVICES and TENSOR_PARALLEL_SIZE.

Returned Result

chat() returns a serializable dictionary with:

• content: assistant text.
• finish_reason: model stop reason.
• usage: token usage when returned by the server.
• latency_s: request latency in seconds.
• error: structured error information, or None on success.

Notes

• The default API key is EMPTY, matching common local vLLM usage.
• Thinking behavior can be controlled through enable_thinking, which is passed via chat_template_kwargs.
• Keep private model paths, API keys, and machine-specific settings out of committed code.