TL;DR
- LLaMA-Factory (hiyouga/LLaMA-Factory) is a comprehensive fine-tuning framework that combines a CLI, a YAML interface, and a Gradio-based web UI for training open-weight LLMs.
- Bundles support for 100+ pretrained model families, six fine-tuning paradigms (continued pretraining, SFT, reward modelling, PPO, DPO, KTO), and most PEFT methods.
- Particularly strong for users who want a point-and-click workflow or who need to fine-tune Chinese-language models — its template registry and dataset zoo are unusually thorough.
- Apache 2.0 licensed; widely used in academic and industry settings across China and increasingly globally.
Positioning#
Where Axolotl is opinionated toward reproducible YAML and Unsloth is opinionated toward single-GPU speed, LLaMA-Factory is opinionated toward completeness. It tries to cover every reasonable combination of base model, fine-tuning method, and post-training algorithm with curated defaults — and to make all of them accessible through both code and a UI.
The web UI in particular distinguishes the project: users can pick a base model, upload or select a dataset, choose a method, and launch a run without writing code. For teaching and rapid prototyping this is materially faster than wiring up Transformers manually.
Supported Methods#
| Stage | Methods |
|---|---|
| Continued pretraining | Full FT, LoRA, QLoRA |
| SFT | Full FT, LoRA, QLoRA, freeze-tuning |
| Reward modelling | Full FT, LoRA, QLoRA |
| PPO | LoRA + frozen base |
| DPO / ORPO / KTO / SimPO | Full FT, LoRA, QLoRA |
| PEFT add-ons | DoRA, PiSSA, rsLoRA, LoftQ, GaLore |
Strengths#
- 100+ supported base models with pre-registered chat templates — Llama, Qwen, GLM, Baichuan, Yi, InternLM, DeepSeek, Mistral, Gemma, Phi, ChatGLM, and more.
- Curated dataset zoo with auto-formatters for Alpaca, ShareGPT, OpenAI, and custom formats.
- Gradio web UI exposes every config knob without code; YAML and CLI paths cover the same surface for scripted runs.
- Native integrations with DeepSpeed, FSDP, vLLM (for inference test of trained adapters), and llama.cpp export.
LLaMA-Factory's `LLaMA Board` web UI is one of the fastest ways to demo a fine-tuning workflow to non-engineers. Spin it up, point it at a small dataset and a 7B base, and you have a clickable end-to-end demo in under an hour.
Trade-offs#
- Pro: by far the broadest method and model coverage in any single open fine-tuning framework.
- Pro: the UI and CLI share a config schema — prototype in the UI, ship in CLI.
- Con: the breadth comes at the cost of opinionatedness — there are usually multiple ways to do the same thing.
- Con: per-run throughput is competitive but not class-leading (Unsloth wins on single-GPU speed).
When to Use LLaMA-Factory#
Use LLaMA-Factory when you need broad method coverage in one place, when you want a UI for non-engineer collaborators, or when fine-tuning Chinese-language models where its template registry has the deepest coverage. For pure throughput, pair with Unsloth-style optimisations. For reproducible CI-driven pipelines, Axolotl's narrower surface is often easier to maintain.
References
- LLaMA-Factory on GitHub · GitHub
- LLaMA-Factory: Unified Efficient Fine-Tuning of 100+ Language Models · arXiv (Zheng et al., 2024)