An experimental small model for translating English requests into Linux shell commands.
The project notebook records setup, experiment results, decisions, and proposed next steps. Each milestone or topic has a descriptive Markdown document in docs/.
Use WSL Ubuntu from this repository directory. Keep the Python environment on the WSL filesystem; the model download is cached there too.
export UV_PROJECT_ENVIRONMENT="$HOME/.venvs/shellm-0.6b"
uv sync --locked
uv run python baseline.py
uv run python baseline.py "enter the root directory"The baseline script loads Qwen/Qwen3-0.6B-Base in float16 on CUDA and uses the proposed Request: ...\nCommand: format. Its output is a pretraining baseline, not a trained shell translator. The script only prints generated text; it never executes it.
uv run python baseline.py --model Qwen/Qwen3-0.6B
uv run python baseline.py --model Qwen/Qwen3-0.6B --prompt-style chatThe first command keeps the raw prompt identical to the base-model run. The second uses Qwen's chat template in non-thinking mode with an instruction to return one command. The script's twelve fixed requests are an exploratory smoke test, not a benchmark score.
See the full comparison and limitations.
The pilot suite covers 112 requests across 14 command families. Commands run in disposable Docker containers and are checked on two filesystem fixtures. The untouched Base model passed 30/112 requests functionally; only 14/112 matched the reference command string.
With Docker Desktop running, use the same WSL environment:
uv run python -m shellbench build
uv run python -m shellbench validate
uv run python -m unittest discover -s tests -v
uv run python -m shellbench evaluate \
--predictions reports/base-functional-baseline/predictions.jsonl \
--output reports/baseline-replay.jsonThe evaluator guide explains generation and replay. The baseline report links to saved predictions, settings, and detailed results.
The training dataset keeps 1,008 training and 112 validation examples in topic/family folders, with editable intent catalogs and fully rendered JSONL releases. See the contribution guide to extend it.
The first LoRA experiment completed three epochs on the RTX 2060 in about 15 minutes, with 1.40 GiB peak PyTorch tensor allocation. Functional accuracy improved from 30/112 (26.8%) to 90/112 (80.4%). This is a development-suite result; quoting, path interpretation, and search composition still need work. The separate smoke request enter the root directory currently prints pwd, showing that broader wording also needs attention.
uv run python -m shellm_data build --release pilot-v1 --check
uv run python -m shellm_data validate --release pilot-v1
uv run python -m shellm_training \
--config configs/training/lora-pilot-v1.json \
--run-id my-lora-experiment
uv run python -m shellbench generate \
--adapter checkpoints/my-lora-experiment/epoch-3 \
--output reports/my-lora-experiment/evaluations/epoch-3
uv run python -m shellbench evaluate \
--predictions reports/my-lora-experiment/evaluations/epoch-3/predictions.jsonl
uv run python tools/report_history.pyTry the saved best overall checkpoint inside WSL:
uv run python translate.py "enter the root directory" \
--adapter checkpoints/2026-10-01-lora-pilot-v1/epoch-3translate.py prints the command. Use uv run python tools/evaluate_checkpoints.py --run-id my-lora-experiment to generate and evaluate every saved epoch of another completed run.
Use a fresh run ID for each experiment. Checkpoints are ignored by Git; dataset releases, configurations, training logs, and evaluation reports are intended to be tracked. New prediction directories and evaluation files refuse overwrites to preserve history. reports/history.csv is a regenerable chart index; original reports remain authoritative.
ShellBench Extra v1 is a separate, frozen benchmark of 300 new requests across 20 categories. It covers compositions, pipelines, unusual filenames, precise text processing, recursive exclusions, permissions, links, and conditional commands. All reference commands passed independent expectations in two new filesystem worlds (600 executions). Its requests and concrete reference commands have no exact overlap with the pilot data or original benchmark.
Use training validation splits for primary development and checkpoint decisions. Measure Extra periodically at completed milestones to track broader progress; keep its cases out of training and validation.
uv run python tools/evaluate_extra.py \
--training-run 2026-10-01-lora-pilot-v1 \
--report-id my-extra-benchmark-snapshot --batch-size 8The construction and initial measurements document the evaluation protocol and limitations. Predictions, results, chart data, and exported PNG/SVG charts are archived separately in reports/2026-10-02-shellbench-extra-v1/.
The first Extra comparison scored Base 20/300 (6.7%), epoch 1 51/300 (17.0%), epoch 2 64/300 (21.3%), and epoch 3 65/300 (21.7%) on both fixtures. The final test suite passed 29 tests. These broader cases give a starting point for measuring later milestones independently of the original pilot score.
The csanad-shenanigans branch expanded the release to 2,072 training and 372 validation examples across 74 families. The fresh LoRA experiment completed three epochs in 41.2 minutes on the RTX 2060. Its final checkpoint scored 95/112 (84.8%) on ShellBench v1 and 77/300 (25.7%) on Extra v1, against matched fresh Base scores of 26.8% and 6.7%. Success requires both fixtures. Raw predictions, reports, source provenance, dataset hashes, logs, and Base-versus-final-model PNG/SVG charts are preserved under reports/2026-10-03-csanad-fresh-lora/.
uv run python tools/evaluate_final.py --run-id 2026-10-03-csanad-fresh-lora --epoch 3 --batch-size 8
uv run --with matplotlib python tools/plot_final_comparison.py --run-id 2026-10-03-csanad-fresh-lora --epoch 3