[Feature] Add support for audio modality and MMSU benchmark support

[TianhaoLiang2000]

This PR adds a first-class AUDIO modality path to VLMEvalKit and lands MMSU as the first supported audio benchmark.

The implementation keeps audio separate from existing image and video flows. It introduces an audio dataset base class, audio-aware dataset registration, audio routing in both local and API inference paths, OpenAI-compatible audio payload formatting for LMDeploy/vLLM API models, and MMSU dataset loading, prompt construction, and exact-matching evaluation.

The target end-to-end path validated in this PR is:

MMSU dataset
  -> AudioBaseDataset
  -> AUDIO dataset registration
  -> run.py / inference_api.py audio route
  -> LMDeployAPI / OpenAI-compatible chat completions payload
  -> vLLM-served Qwen3-Omni endpoint
  -> prediction TSV
  -> MMSU exact-matching evaluation and breakdown report

Motivation

VLMEvalKit already has mature modality-specific paths for image and video benchmarks. Audio benchmarks should not be represented as image-like or video-like datasets with special-case media fields. This PR adds audio as a peer modality so future audio benchmarks can reuse a clear and extensible contract.

The first supported benchmark is MMSU, a 5,000-sample spoken language understanding and reasoning benchmark with audio plus multiple-choice questions.

What Changed

1. Added AudioBaseDataset

New file:

vlmeval/dataset/audio_base.py

AudioBaseDataset defines the shared audio dataset contract:

[
    {"type": "audio", "value": "/abs/or/relative/path/to/audio.wav"},
    {"type": "text", "value": "Question... Options..."},
]

It handles:

• MODALITY = 'AUDIO'.
• Loading dataset metadata through the existing TSV/data root conventions.
• Resolving audio paths from audio_path.
• Dumping embedded base64 audio in audio to LMUData/audios/<dataset>/.
• Supporting one or multiple audio clips per sample.
• Falling back from relative audio_path to the dataset audio root.
• Returning audio prompt items that are consumed by the model wrapper through the same model.generate(message=..., dataset=...) contract used by other modalities.

2. Registered AUDIO datasets

Changed file:

vlmeval/dataset/__init__.py

This PR adds:

AUDIO_DATASET = [MMSUDataset]

and includes AUDIO_DATASET in DATASET_CLASSES, so:

• MMSU is included in SUPPORTED_DATASETS.
• DATASET_MODALITY('MMSU') returns AUDIO.
• DATASET_TYPE('MMSU') returns MCQ.

3. Added audio inference routing

Changed files:

run.py
vlmeval/inference_api.py
vlmeval/inference_audio.py

The local route imports infer_data_job_audio and dispatches datasets with MODALITY == 'AUDIO' to the audio route.

The async API pipeline now includes:

• DatasetType = Literal["image", "video", "audio", "mt"].
• dataset_type="audio" task production.
• _produce_audio_tasks() for building audio prompts without video-specific subprocess/frame logic.
• Worker execution through the existing model.generate() path.
• Checkpoint reuse and failed-result retry behavior consistent with other modalities.
• Final result writing that drops embedded audio data when present, so prediction TSVs do not duplicate large media payloads.

inference_audio.py reuses the established inference pattern while preserving audio-specific final result cleanup.

4. Added OpenAI-compatible audio payload support

Changed file:

vlmeval/api/lmdeploy.py

LMDeployWrapper now accepts audio in allowed_types and maps internal audio prompt items to OpenAI-compatible audio_url chat content.

When --local-media is sent, the audio file is encoded into a data URL:

{
  "type": "audio_url",
  "audio_url": {
    "url": "data:audio/wav;base64,..."
  }
}

When --local-media is not sent, the local audio path is converted to a file URI:

{
  "type": "audio_url",
  "audio_url": {
    "url": "file:///abs/path/audio.wav"
  }
}

The wrapper also passes through existing media URLs unchanged:

• http://...
• https://...
• file://...
• data:...

5. Added MMSU dataset support

New file:

vlmeval/dataset/mmsu.py

The new MMSUDataset is implemented as:

class MMSUDataset(AudioBaseDataset):
    TYPE = 'MCQ'
    MODALITY = 'AUDIO'

Evaluation uses mcq_vanilla_eval with exact matching by default and reports:

• Overall accuracy.
• Accuracy by category.
• Accuracy by sub-category.
• Accuracy by sub-sub-category.
• Accuracy by task_name.
• Accuracy by linguistics_sub_discipline.

Supported Audio Formats

The first implementation supports these audio file extensions and MIME mappings:

Extension	MIME type
.wav	audio/wav
.mp3	audio/mpeg
.m4a	audio/mp4
.flac	audio/flac
.ogg	audio/ogg
.aac	audio/aac
.oga	audio/ogg
.opus	aaudio/ogg

Supported input representations:

• audio_path as an absolute path.
• audio_path as a path relative to LMUData/audios/<dataset>/.
• audio as embedded base64 content in a TSV.
• audio as bytes or Hugging Face audio payloads during MMSU conversion.
• Already formatted http://, https://, file://, or data: media URLs.

Supported API payload modes:

• Base64 data URL mode, used when local_media=False.
• Local file URI mode, used when local_media=True.

The local file URI mode requires the remote vLLM/Lmdeploy server to be able to read the referenced media path, for example through a shared filesystem and an appropriate allowed local media path configuration. The base64 mode avoids that filesystem coupling at the cost of larger request payloads.

CLI and API Behavior

run.py --base-url builds LMDeployAPI dynamically, so this PR does not require adding a hard-coded Qwen3-Omni model entry to vlmeval/config.py.

Relevant API-mode arguments used in validation:

python3 run.py \
  --api-mode \
  --data MMSU \
  --model Qwen3-Omni-30B-A3B-Instruct \
  --base-url "${BASE_URL}" \
  --key EMPTY \
  --max-tokens 16384 \
  --retry 3 \
  --timeout 3600 \
  --work-dir "${OUTPUT_BASE}" \
  --mode all \
  --reuse \
  --api-nproc 64 \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --judge exact_matching \
  --custom-prompt qwen3

Validation

Single-sample endpoint validation

One MMSU sample was tested against a vLLM-served Qwen3-Omni-30B-A3B-Instruct OpenAI-compatible endpoint.

with --local-media sent:

{"type": "audio_url", "audio_url": {"url": "file:///mnt/.../sample.wav"}}

without --local-media sent:

{"type": "audio_url", "audio_url": {"url": "data:audio/wav;base64,..."}}

Both modes returned normal model content through the same LMDeployAPI wrapper.

Full MMSU end-to-end validation

Validation environment:

• Dataset: MMSU
• Samples: 5,000
• Model: Qwen3-Omni-30B-A3B-Instruct
• Backend: vLLM OpenAI-compatible chat completions endpoint
• Judge: exact_matching
• API_NPROC: 64
• temperature: 0.6
• top_p: 0.95
• top_k: 20
• max_tokens: 16384
• Prediction format: TSV

Total-score comparison

Run	Audio transport	MMSU score	Delta vs official
Official Qwen model card	N/A	69.00	0.00
Full run 1	file:// local media	70.04	+1.04
Full run 2	base64 data URL	70.52	+1.52

The two runs used sampling parameters with temperature=0.6, so the small metric delta should be treated as sampling variation rather than evidence that one transport encoding is systematically better. The important validation result is that both file:// and base64 audio payload modes complete the full 5,000-sample MMSU run with zero inference and judge failures.