[Neonkraft]

Summary

Makes the LlamaFactory backend usable for real SLURM runs. Before this PR, LlamaFactory submissions fell back to TRL defaults in several places and could not resume after a requeue. The backend now targets LlamaFactory v0.9.5.

Configuration

• All training settings come from the llamafactory block, using LlamaFactory's own keys. Any TRL section (model, training, data, ...) is now rejected, and method must match llamafactory.stage.
• Relative input paths are made absolute at submission, because the job now runs in the run directory.

Run directory and resume

• Checkpoints go to <run_dir>/checkpoints/, so requeued and resubmitted jobs resume from the last checkpoint. The submission owns output_dir; setting it is an error.
• The DeepSpeed JSON, dataset_info.json and the tokenize script are frozen into <run_dir>/llamafactory/. A later submit into the same run directory keeps them, and the review screen warns if a source changed.

Prefetch and job script

• The login-node prefetch downloads the LlamaFactory model and datasets (not the TRL defaults), and the job loads the local snapshots.
• The job script loads modules, honors offline, always launches through torchrun, checks the HF cache variables, and no longer requeues twice on failure.

--tokenize-only for LlamaFactory

• New: tokenizes the datasets into llamafactory.tokenized_path on 1 GPU and exits before the model load. A later training submit with the same config loads the saved data.
• The submission requires tokenized_path and refuses one that already holds data. Data is written to a .partial directory first, so an interrupted job leaves nothing behind.

Review screen and configs

• The submission review shows the llamafactory block, with defaults marked, and warns about non-resumable stages and kept frozen inputs.
• New configs/llamafactory/prelude-sft.yaml (Prelude 9B SFT on LUMI), and README sections on LlamaFactory configuration and the run directory.

Type of change

• Bug fix
• New feature
• Refactor
• Performance
• Documentation
• Maintenance

Validation

• pytest: 332 passed, including end-to-end submit.py tests without sbatch. ruff and black pass.

[Copilot]

🟡 Changes recommended

Unresolved concurrency, authentication, and input-handling issues can expose incomplete tokenized data or prevent valid jobs from running.

7 open issues

• Lock shared tokenize-only targets through publication · New
• Normalize adapter paths as lists · New
• Skip source registry validation when tokenized data exists · New
• Apply media directory default when explicitly set to null · New
• Reject unsupported PPO checkpoint resume instead of advertising it · New
• Use the frozen registry for resubmission prefetch · New
• Pass configured Hub token to model and dataset prefetch · New

What changed in this PR

This PR expands the LlamaFactory backend for SLURM training with native configuration, frozen run inputs, checkpoint handling, and tokenization-only jobs targeting v0.9.5.

Changes:

• Adds LlamaFactory-specific validation, asset prefetching, path handling, and submission review.
• Adds tokenization-only execution and improves container environment checks and failure handling.
• Adds regression tests, usage documentation, and a Prelude SFT configuration.

File	Description
tests/​test_submit_llamafactory.py	Tests submission, resubmission, and tokenization setup.
tests/​test_submit_hf_check.py	Tests submission-time cache environment checks.
tests/​test_slurm_render.py	Tests LlamaFactory job rendering and environment settings.
tests/​test_prefetch.py	Tests native model and dataset prefetching.
tests/​test_guardrails_llamafactory.py	Tests review output and frozen-input warnings.
tests/​test_container_job_script.py	Exercises container scripts with stub SLURM commands.
tests/​test_config.py	Tests rejection and omission of TRL sections.
tests/​test_backend.py	Tests validation, frozen inputs, paths, and run naming.
src/​post_training/​utils/​prefetch.py	Adds LlamaFactory asset download planning.
src/​post_training/​utils/​guardrails.py	Adds LlamaFactory-specific submission review.
src/​post_training/​slurm/​llamafactory_tokenize.py	Adds standalone dataset tokenization.
src/​post_training/​slurm/​launcher.py	Passes tokenization and environment settings into templates.
src/​post_training/​slurm/​job_trl_container.sh.jinja	Adds cache checks and corrects failure handling.
src/​post_training/​slurm/​job_llamafactory.sh.jinja	Updates launch, environment, and requeue behavior.
src/​post_training/​config.py	Validates explicit keys and saves backend-specific configuration.
src/​post_training/​backend.py	Adds native validation, input freezing, and checkpoint configuration.
scripts/​submit.py	Integrates cache checks and tokenization-only submission.
README.md	Documents native configuration and run-directory behavior.
data/​llamafactory/​dataset_info.json	Registers two additional SFT datasets.
configs/​llamafactory/​prelude-sft.yaml	Adds a Prelude 9B SFT example for LUMI.
configs/​llamafactory/​long-context.yaml	Removes output settings now owned by submission.

_{Review effort: Balanced}

[ReinforcedKnowledge]

I was able to reproduce the issues about tokenization race, frozen-registry/prefetch mismatch and rejection of saved tokenized data without its original registry in #65 (comment), #65 (comment), #65 (comment) respectively. So I won't duplicate comments.

One other issue is this dolci_instruct_sft entry does not preserve raw Dolci’s tool schema. Reproduced with LF 0.9.5, template: qwen3, enable_thinking: true and mask_history: true, using the Paris/Madrid weather example from #63. The definitions in the system message’s functions field and the calls in the assistant’s function_calls field disappear during ShareGPT conversion. The converted example has:

aligned["_tools"] == ""
aligned["_prompt"][1] == {"role": "assistant", "content": None}

The environment result is retained as an observation. SFT preprocessing then fails in remove_thought: TypeError: expected string or bytes-like object, got 'NoneType'

I think we should either remove this entry for now or even better normalize raw Dolci into LF-compatible tool definitions and calls before conversion. Replacing the null content with an empty string would still leave the tool data missing.

PS: The default Excellent recipe is unaffected by this issue.