Skip to content

macOS: benchmark run can hang indefinitely with default multiprocessing/data loader settings #1

Description

@aiwantaozi

Description

On macOS, benchmark_runner may hang indefinitely during benchmark run when using default runtime settings.
The process reaches the benchmark phase but does not make progress for several minutes (or longer), and no explicit error is raised.

Environment

  • OS: macOS (Apple Silicon)
  • Python: 3.11
  • benchmark-runner: v0.0.2
    Backend: OpenAI-compatible HTTP endpoint
    Data: synthetic (prompt_tokens=...,output_tokens=...)

Reproduction

python -m benchmark_runner.main benchmark run \
  --target http://<server>:<port> \
  --sample-requests 0 \
  --output-dir .cache \
  --outputs full.json \
  --processor <local_tokenizer_path> \
  --data prompt_tokens=8192,output_tokens=100 \
  --profile constant \
  --rate 1 \
  --max-requests 10

Expected behavior

Benchmark starts and either: progresses normally, or fails fast with a clear error.

Actual behavior

The command appears stuck/hung (often at the benchmark stage) without completion or explicit failure.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions