Local LLM Workload Benchmark

This project measures how small, open-weight language models handle practical work when run locally on Apple Silicon. It compares task quality, instruction control, latency, throughput, and memory use across model families and GGUF quantizations.

The results apply to this workload, hardware, model cohort, and inference configuration. They are not universal model rankings.

Study design

The study evaluates five model families at four quantizations: Q3, Q4, Q6, and Q8. Every run uses a 16K context window, temperature 0, seed 42, the same output limits, and one generation per item.

Models

• Llama 3.1 8B Instruct
• Qwen3 8B
• Mistral 7B Instruct v0.3
• Granite 3.3 8B Instruct
• Qwen2.5-Coder 7B Instruct

Workloads

Stage	Workload	What it measures	Items per configuration	Local generations
Practical workloads	Applied reasoning	Arithmetic, logic, ordering, and probability	48	960
Practical workloads	Code debugging and repair	Function implementation, diagnosis, and repair	80	1,600
Practical workloads	Messy text to schema	Structured extraction from noisy text	48	960
Practical workloads	Constraint-load curve	Compliance as simultaneous rules increase	48	960
Practical workloads	Tool use	Tool selection and exact arguments	48	960
Practical workloads	Email to action	Inbox routing and structured action extraction	40	800
Retrieval	Long-text retrieval	Evidence retrieval from long documents	48	960
Grounded compression	Faithful compression	Concise summaries supported by supplied sources	30	600

The complete study contains 390 benchmark items and 7,800 local generations across 20 model and quantization configurations.

Evaluation design

The benchmark uses the strongest evaluator available for each task:

• Code tasks run executable tests.
• Structured outputs use schema and exact-field checks.
• Reasoning tasks use exact, numeric, rational, date, or set comparisons.
• Tool tasks check tool choice, arguments, and response protocol separately.
• Constraint tasks check mechanical rules and declared semantic requirements.
• Grounded summaries use the configured LLM judge and deterministic word-limit checks.

The practical and retrieval stages run deterministic scoring first. The pipeline then sends eligible ambiguous failures and semantic requirements to the calibrated LLM judge. Grounded compression uses the same judge directly. Authoritative code, schema, exact-answer, and tool-argument checks are never overridden by the judge.

What is recorded

Every response is saved with its prompt identity, raw output, evaluated output, score details, model configuration, generation settings, and runtime telemetry. The aggregate artifacts include:

• quality score, strict pass rate, and semantic pass rate;
• protocol compliance, parse failures, and recovery rate;
• time to first token and end-to-end latency;
• prompt and decode throughput;
• model-load and peak process memory;
• per-workload, per-model, and per-quantization summaries.

Requirements

• macOS on Apple Silicon with Metal support
• Python 3.11 or newer
• uv
• about 110 GiB for the pinned GGUF model cohort
• a Cerebras API key for judge calibration and judged evaluation

Setup

Install uv if needed:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install the project and download the pinned model files:

scripts/setup.sh

The setup script resumes partial downloads. It verifies the expected byte count and SHA-256 hash for all 20 GGUF files.

Add the judge credential to .env before starting the study:

CEREBRAS_API_KEY=your-key-here

If Hugging Face requires authentication for a model file, add HF_TOKEN to the same file.

Run the complete study

Run the full pipeline with one command:

caffeinate -dims scripts/run_study.sh

The command performs these steps in order:

1. Build and validate the datasets.
2. Calibrate the configured LLM judge against the reference-labelled fixture.
3. Run the practical, retrieval, and grounded-compression matrices.
4. Adjudicate saved responses and recover grounded judge failures without regenerating model responses.
5. Export normalized metrics, generate stage plots, and generate the combined figure bundle.

The pipeline saves its state after each step. If it stops, run the same command again. It resumes the active study instead of starting over.

You can also run the three stages separately:

caffeinate -dims scripts/run_study.sh workloads
caffeinate -dims scripts/run_study.sh retrieval
caffeinate -dims scripts/run_study.sh grounded

The first stage calibrates the judge. Later stages reuse that completed calibration. The combined plots are generated when all three stages are complete.

Check progress without running work:

scripts/run_study.sh --status

Use --restart only when you want to abandon an incomplete study and start a new one. Completed studies start fresh automatically.

Results and artifacts

• Pipeline state is stored in runs/study.json.
• Raw responses and telemetry are stored under runs/<study-id>-<stage>/.
• Normalized metrics are stored under each run's artifacts/data/ directory.
• Stage plots are stored under each run's artifacts/plots/ directory.
• Combined publication plots are stored under final_figures/plots/.
• Figure provenance is stored in final_figures/manifest.json.

See FINDINGS.md for the current study results, interpretation, and limitations.

Development checks

Run the test suite without model inference or external judge calls:

UV_CACHE_DIR=/tmp/uv-cache uv run --offline pytest