Repository navigation
Bug: blank lines in wav.scp / jsonl file lists crash prepare_data_iterator (IndexError / JSONDecodeError) #3757
Description
🐛 Bug
AutoModel.generate() with a file list (wav.scp, text.txt, file.jsonl, ...) crashes if the list contains a blank or whitespace-only line. prepare_data_iterator() in funasr/auto/auto_model.py iterates over every line without skipping empty ones:
- the
id \t databranch doeslines = line.strip().split(maxsplit=1)and then indexeslines[0], so an empty line raisesIndexError: list index out of range; - the
.jsonlbranch callsjson.loads(line.strip())on the empty line and raisesjson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0).
Blank lines are easy to produce in practice — lists concatenated from several sources, lists edited by hand, or files written with an extra separator line — and one of them aborts the whole batch instead of being ignored.
To Reproduce
- Install from source:
pip install -e . - Run the snippet below (it calls
prepare_data_iteratordirectly, so no model weights or download are needed). - See
IndexErrorfor the.scplist andJSONDecodeErrorfor the.jsonllist.
Code sample
from funasr.auto.auto_model import prepare_data_iterator
open("wav.scp", "w", encoding="utf-8").write("utt1 /a.wav\n\nutt2 /b.wav\n")
open("data.jsonl", "w", encoding="utf-8").write(
'{"key": "k1", "source": "/a.wav"}\n\n{"key": "k2", "source": "/b.wav"}\n'
)
prepare_data_iterator("wav.scp") # IndexError: list index out of range
prepare_data_iterator("data.jsonl") # json.decoder.JSONDecodeErrorThe same crash reaches users through model.generate(input="wav.scp").
Expected behavior
Blank lines carry no utterance, so they should be skipped; the two real entries should come back as (['utt1', 'utt2'], ['/a.wav', '/b.wav']) and (['k1', 'k2'], ['/a.wav', '/b.wav']), exactly as they do when the blank line is deleted manually.
Error logs
Traceback (most recent call last):
File "repro.py", line 8, in <module>
prepare_data_iterator("wav.scp")
File ".../funasr/auto/auto_model.py", line 479, in prepare_data_iterator
data = lines[1] if len(lines) > 1 else lines[0]
IndexError: list index out of range
Traceback (most recent call last):
File "repro.py", line 9, in <module>
prepare_data_iterator("data.jsonl")
File ".../funasr/auto/auto_model.py", line 474, in prepare_data_iterator
lines = json.loads(line.strip())
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Environment
- OS: Windows 11 26200 (the file-list reader is pure Python, so this reproduces on Linux too)
- Python version: 3.11.14
- FunASR version: source
mainat66d7a4c264a5993a2a63ed00c1f402c296ee521a - ModelScope version: not used by this code path
- PyTorch / torchaudio version: 2.14.1+cpu
- Install method (
pip, source, Docker): source,pip install -e .equivalent - Device (
cuda,cpu,mps): cpu - GPU model: n/a
- CUDA/cuDNN version: n/a
- Docker image tag, if used: n/a
Audio details
No audio is needed — the crash happens while parsing the list file itself.
Activity
LauraGPT commentedon Oct 8, 2026
Thanks for the self-contained reproduction and fix. #3758 is now merged in 22b4282.
I reproduced both parser failures against current main before the change (IndexError for .scp and JSONDecodeError for .jsonl). With the fix, the file-list/byte-loading suites pass 84 tests and 2 subtests. Additional CPU-only checks cover blank-only lists, Unicode whitespace, CRLF, key order and multiword text preservation. Malformed nonblank JSON still raises rather than being silently ignored. Hosted NumPy 1.26.4 and 2.4.0 checks at the PR head each pass 122 tests without failures or skips.
GitHub auto-closed this issue on merge; I have reopened it with needs feedback under our issue-lifecycle policy. Please confirm the source fix with your original Windows 11 / Python 3.11.14 file lists. Our validation used Linux/Python 3.12, not a real Windows installation, and this is a source-main fix, not a claim that a new PyPI release is available. No audio or model download is needed to check prepare_data_iterator.