Summary

LoRA reduces fine-tuning parameters to 0.1%-1% through low-rank matrix decomposition, enabling 7B model fine-tuning on a single A100 GPU
QLoRA builds on LoRA with 4-bit quantization and paged optimizers, making 13B model fine-tuning on 24GB VRAM a reality
RLHF (Reinforcement Learning from Human Feedback) is the core technique for LLM alignment, but reward model training remains the biggest bottleneck
Dataset quality determines the ceiling of fine-tuning: 1,000 high-quality samples outperform 10,000 noisy ones
This article provides a complete fine-tuning pipeline from data preparation to model deployment, including training scripts and evaluation strategies

Three Approaches to LLM Fine-Tuning
LoRA Low-Rank Adaptation: Principles and Practice
QLoRA Quantized Fine-Tuning: Extreme VRAM Compression
RLHF Alignment Training: Making Models Safer
Dataset Preparation: The Lifeline of Fine-Tuning
Fine-Tuned Model Evaluation and Deployment
Summary and Further Reading

Three Approaches to LLM Fine-Tuning

Full Fine-Tuning vs LoRA vs QLoRA

Dimension	Full FT	LoRA	QLoRA
Trainable Parameters	100%	0.1%-1%	0.1%-1%
VRAM Required (7B)	120GB+	28GB	16GB
VRAM Required (13B)	240GB+	48GB	24GB
Training Speed	Slow	Fast (1.2-1.5x)	Medium (0.8-1.0x)
Accuracy	Highest	Near Full FT	Near Full FT
Hardware Threshold	4xA100	1xA100	1xA100/4090
Deployment	Merge weights	Merge or standalone	Merge or standalone
Multi-task Switching	Difficult	Easy (swap LoRA)	Easy (swap LoRA)

Fine-Tuning Approach Decision Tree

LoRA Low-Rank Adaptation: Principles and Practice

Core Principles of LoRA

The core idea of LoRA (Low-Rank Adaptation): the fine-tuning increment DeltaW of the pre-trained weight matrix W can be approximated by the product of two low-rank matrices: DeltaW = A x B, where A in R^(d x r), B in R^(r x k), and r is much smaller than d and k.

+----------------------------------------------------------+ | LoRA Low-Rank Decomposition Principle | | | | Original Weight W (frozen): | | +------------------------+ | | | d x k = 4096x4096 | = 16M parameters | | | (not updated) | | | +------------------------+ | | | | LoRA Increment DeltaW = A x B: | | +----------+ +----------+ | | | d x r | x | r x k | = 2x4096x8 = 65K params | | | 4096x8 | | 8x4096 | (only 0.4% of original) | | +----------+ +----------+ | | | | Output = W*x + DeltaW*x = W*x + A*(B*x) | +----------------------------------------------------------+

LoRA Training Code

`python from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments from peft import LoraConfig, get_peft_model, TaskType from trl import SFTTrainer from datasets import load_dataset

model_id = "Qwen/Qwen2.5-7B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, )

lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], bias="none", )

model = get_peft_model(model, lora_config) model.print_trainable_parameters()

trainable params: 19,594,240 || all params: 7,615,078,400 || trainable%: 0.2573%

dataset = load_dataset("json", data_files="train_data.jsonl", split="train")

def format_example(example): return { "text": f"<|im_start|>system\nYou are a professional K8s operations assistant<|im_end|>\n<|im_start|>user\n{example['input']}<|im_end|>\n<|im_start|>assistant\n{example['output']}<|im_end|>" }

dataset = dataset.map(format_example)

training_args = TrainingArguments( output_dir="./lora-output", num_train_epochs=3, per_device_train_batch_size=4, gradient_accumulation_steps=8, learning_rate=2e-4, weight_decay=0.01, warmup_ratio=0.03, lr_scheduler_type="cosine", bf16=True, logging_steps=10, save_strategy="epoch", evaluation_strategy="no", gradient_checkpointing=True, optim="adamw_torch", report_to="tensorboard", )

trainer = SFTTrainer( model=model, args=training_args, train_dataset=dataset, processing_class=tokenizer, max_seq_length=2048, )

trainer.train() trainer.save_model("./lora-output/final") `

LoRA Hyperparameter Tuning

Hyperparameter	Recommended Value	Description
r (rank)	8-64	Larger = more capacity but slower; 16 recommended for 7B models
lora_alpha	2xr	Scaling factor, typically set to 2x r
lora_dropout	0.05	Prevents overfitting
target_modules	q/k/v/o/gate/up/down_proj	More modules = stronger but slower
learning_rate	1e-4 ~ 3e-4	LoRA typically uses higher learning rates
batch_size	4-8	Combined with gradient_accumulation

QLoRA Quantized Fine-Tuning: Extreme VRAM Compression

Three Innovations of QLoRA

Innovation	Description	Effect
4-bit NormalFloat	New data type, optimal quantization for normal distributions	Accuracy loss < 1%
Double Quantization	Quantize the quantization constants again	Saves 0.37 bit per parameter
Paged Optimizers	Offload to CPU when GPU VRAM is insufficient	Avoids OOM

QLoRA Training Code

`python from transformers import AutoModelForCausalLM, BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, )

model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-13B-Instruct", quantization_config=bnb_config, device_map="auto", trust_remote_code=True, )

model = prepare_model_for_kbit_training(model)

model = get_peft_model(model, lora_config) model.print_trainable_parameters()

trainable params: 19,594,240 || all params: 7,615,078,400 || trainable%: 0.2573%

VRAM Comparison

Model	Full FT	LoRA (BF16)	QLoRA (4-bit)
7B	120GB	28GB	16GB
13B	240GB	48GB	24GB
72B	1.2TB	280GB	48GB

RLHF Alignment Training: Making Models Safer

Three-Stage RLHF Pipeline

RLHF Training Code (TRL Framework)

`python from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead from trl import create_reference_model from transformers import AutoTokenizer

model_id = "./sft-model"

tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLMWithValueHead.from_pretrained(model_id) ref_model = create_reference_model(model)

ppo_config = PPOConfig( model_name=model_id, learning_rate=1.41e-5, batch_size=128, mini_batch_size=4, gradient_accumulation_steps=4, ppo_epochs=4, max_grad_norm=0.5, kl_coef=0.2, target_kl=6.0, )

ppo_trainer = PPOTrainer( config=ppo_config, model=model, ref_model=ref_model, tokenizer=tokenizer, dataset=preference_dataset, reward_model=reward_model, )

for epoch in range(ppo_config.ppo_epochs): for batch in ppo_trainer.dataloader: query_tensors = batch["input_ids"] response_tensors = ppo_trainer.generate(query_tensors) rewards = reward_model(batch, response_tensors) stats = ppo_trainer.step(query_tensors, response_tensors, rewards) ppo_trainer.log_stats(stats, batch, rewards) `

DPO: A Simplified Alternative to RLHF

DPO (Direct Preference Optimization) bypasses the reward model and directly optimizes the policy model using preference data.

`python from trl import DPOTrainer, DPOConfig

dpo_config = DPOConfig( output_dir="./dpo-output", per_device_train_batch_size=4, learning_rate=5e-7, beta=0.1, max_length=2048, num_train_epochs=1, gradient_checkpointing=True, bf16=True, )

dpo_trainer = DPOTrainer( model=model, ref_model=ref_model, args=dpo_config, train_dataset=preference_dataset, processing_class=tokenizer, )

dpo_trainer.train() `

Method	Requires Reward Model	Training Stability	VRAM Requirement	Effectiveness
PPO	Yes	Unstable	High	Strongest
DPO	No	Stable	Medium	Strong
KTO	No	Most Stable	Low	Moderately Strong

Dataset Preparation: The Lifeline of Fine-Tuning

Data Quality > Data Quantity

Data Volume	Quality	Model Performance
10,000 samples	Low (noisy)	Poor
5,000 samples	Medium	Average
1,000 samples	High (human-reviewed)	Good
500 samples	Very High (expert-annotated)	Very Good

Dataset Format

json {"input": "How to configure K8s GPU time-slicing?", "output": "K8s GPU time-slicing configuration requires installing the NVIDIA GPU Operator..."} {"input": "What is the difference between vLLM and TGI?", "output": "vLLM is based on PagedAttention, achieving 90%+ GPU utilization..."}

Data Cleaning Pipeline

`python import json import re from pathlib import Path

def clean_dataset(input_path: str, output_path: str): cleaned = [] with open(input_path, "r", encoding="utf-8") as f: for line in f: item = json.loads(line.strip()) if not item.get("input") or not item.get("output"): continue if len(item["input"]) < 5 or len(item["output"]) < 20: continue if len(item["output"]) > 4096: continue item["input"] = re.sub(r"\s+", " ", item["input"]).strip() item["output"] = re.sub(r"\s+", " ", item["output"]).strip() cleaned.append(item)

with open(output_path, "w", encoding="utf-8") as f:
    for item in cleaned:
        f.write(json.dumps(item, ensure_ascii=False) + "\n")

print(f"Cleaned: {len(cleaned)} items")

Fine-Tuned Model Evaluation and Deployment

Evaluation Metrics

Metric	Description	Tool
Perplexity	Lower is better	lm-eval-harness
BLEU	Translation/generation quality	sacrebleu
ROUGE	Summarization quality	rouge-score
Human Evaluation	Ultimate quality standard	Annotation platform
Domain Benchmark	Domain-specific evaluation	Custom

LoRA Weight Merging and Deployment

`python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto", )

model = PeftModel.from_pretrained(base_model, "./lora-output/final") merged_model = model.merge_and_unload() merged_model.save_pretrained("./merged-model") tokenizer.save_pretrained("./merged-model") `

Deploying Fine-Tuned Models with vLLM

ash python -m vllm.entrypoints.openai.api_server \ --model ./merged-model \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.92 \ --max-model-len 8192 \ --enable-prefix-caching

Summary and Further Reading

The three pillars of LLM fine-tuning each have their ideal use cases: LoRA is the general-purpose choice, QLoRA is the savior for VRAM-constrained scenarios, and RLHF/DPO is the essential path for alignment. The core principle: data quality determines the ceiling, parameter efficiency determines the barrier to entry, and alignment techniques determine safety.

Key Takeaways:

LoRA reduces trainable parameters to 0.1%-1% through low-rank decomposition, enabling 7B fine-tuning on a single A100
QLoRA adds 4-bit quantization on top of LoRA, enabling 13B fine-tuning with 24GB VRAM
DPO is more stable than PPO and is recommended as an alternative to RLHF
1,000 high-quality samples > 10,000 noisy samples
Post-fine-tuning evaluation is mandatory; training loss alone is insufficient

Related Reading:

LLM Inference Acceleration Benchmark: vLLM vs TensorRT-LLM vs SGLang — Inference deployment optimization for fine-tuned models
Python AI Multimodal RAG in Practice — Fine-tuned models in RAG applications
MCP Protocol in Practice: Building AI Agent Toolchains with Model Context Protocol — Fine-tuned models as Agent brains

Authoritative References:

LLM Fine-Tuning in Practice: LoRA, QLoRA, and RLHF for Domain Adaptation

Summary

Table of Contents

Three Approaches to LLM Fine-Tuning

Full Fine-Tuning vs LoRA vs QLoRA

Fine-Tuning Approach Decision Tree

LoRA Low-Rank Adaptation: Principles and Practice

Core Principles of LoRA

LoRA Training Code

trainable params: 19,594,240 || all params: 7,615,078,400 || trainable%: 0.2573%

LoRA Hyperparameter Tuning

QLoRA Quantized Fine-Tuning: Extreme VRAM Compression

Three Innovations of QLoRA

QLoRA Training Code

trainable params: 19,594,240 || all params: 7,615,078,400 || trainable%: 0.2573%

VRAM Comparison

RLHF Alignment Training: Making Models Safer

Three-Stage RLHF Pipeline

RLHF Training Code (TRL Framework)

DPO: A Simplified Alternative to RLHF

Dataset Preparation: The Lifeline of Fine-Tuning

Data Quality > Data Quantity

Dataset Format

Data Cleaning Pipeline

Fine-Tuned Model Evaluation and Deployment

Evaluation Metrics

LoRA Weight Merging and Deployment

Deploying Fine-Tuned Models with vLLM

Summary and Further Reading