LLM Knowledge Distillation in Practice: The Compression Path from Teacher to Student Model

Abstract

• Knowledge distillation is the critical bridge for LLM production deployment: compressing a 70B-parameter teacher model to a 7B student model reduces inference cost by 90%+ while retaining 85%+ performance
• 3 major distillation strategies: white-box distillation (logit matching), gray-box distillation (feature alignment), and black-box distillation (output imitation), each suited to different scenarios
• 3-stage progressive distillation: 70B→30B→7B, preserving 90%+ knowledge at each stage, with final model performance approaching direct distillation
• Multi-task distillation in practice: distilling general and specialized capabilities simultaneously while avoiding catastrophic forgetting
• This article provides full LLaMA distillation pipeline code and an analysis of the DeepSeek-R1 distillation approach

---

Table of Contents

• Knowledge Distillation: The Essential Path for LLM Production Deployment
• 3 Major Distillation Strategies Explained
• White-Box Distillation: Logit Matching in Practice
• Gray-Box Distillation: Feature Alignment in Practice
• 3-Stage Progressive Distillation
• Multi-Task Distillation and Catastrophic Forgetting
• Summary and Resources

---

Knowledge Distillation: The Essential Path for LLM Production Deployment

Why Knowledge Distillation?

Dimension	Teacher Model (70B)	Student Model (7B)	Distillation Benefit
Inference Latency	800ms/token	80ms/token	10×
GPU Memory	4×A100	1×RTX4090	4×
Deployment Cost	$5/1K requests	$0.5/1K requests	10×
General Capability	92 pts	78 pts	-15%
Specialized Capability	88 pts	82 pts	-7%

Knowledge Distillation Evolution

Stage	Period	Method	Representative Work
Classic Distillation	2015	Soft Label	Hinton's KD
Feature Distillation	2018-2020	Intermediate Layer Alignment	FitNets, PKT
LLM Distillation	2023	Logit + Feature	Alpaca, Vicuna
Systematic Distillation	2024-2026	Progressive + Multi-task	DeepSeek-R1, Qwen2.5

Mainstream Distillation Approaches in 2026

Approach	Teacher Model	Student Model	Capability Retention	Open Source
DeepSeek-R1	R1-671B	R1-Distill-8B	82%	Yes
Qwen2.5 Distillation	Qwen2.5-72B	Qwen2.5-7B	85%	Yes
LLaMA Distillation	LLaMA-3-70B	LLaMA-3-8B	80%	Partial
GPT-4 Distillation	GPT-4	GPT-4o-mini	N/A	No
Claude Distillation	Claude-3.5	Claude-3-Haiku	N/A	No

---

3 Major Distillation Strategies Explained

Strategy Overview

┌─────────────────────────────────────────────────────────────┐
│              3 Major Distillation Strategies                  │
│                                                               │
│  1. White-Box Distillation                                    │
│  ┌─────────────────────────────────────────────────────┐     │
│  │  Access teacher model internals: logits, hidden layers, attention weights  │     │
│  │  Loss = α·L_logit + β·L_feature + γ·L_attn          │     │
│  │  Advantage: Best distillation performance             │     │
│  │  Disadvantage: Requires teacher model weights, high compute cost  │     │
│  └─────────────────────────────────────────────────────┘     │
│                                                               │
│  2. Gray-Box Distillation                                     │
│  ┌─────────────────────────────────────────────────────┐     │
│  │  Partial access to teacher model: logits only or features only  │     │
│  │  Loss = α·L_logit + β·L_CE                            │     │
│  │  Advantage: Balances performance and cost             │     │
│  │  Disadvantage: Performance inferior to white-box      │     │
│  └─────────────────────────────────────────────────────┘     │
│                                                               │
│  3. Black-Box Distillation                                    │
│  ┌─────────────────────────────────────────────────────┐     │
│  │  Access teacher model output only: generated text + scores  │     │
│  │  Loss = L_CE(student output, teacher generated text)  │     │
│  │  Advantage: No teacher weights needed, API sufficient  │     │
│  │  Disadvantage: Worst performance, strong data dependency  │     │
│  └─────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘

Strategy Selection Decision Tree

Condition	Recommended Strategy
Have teacher model weights + sufficient GPU	White-box distillation
Have teacher model weights + limited GPU	Gray-box distillation (logit)
API access only	Black-box distillation
Very large teacher model (>100B)	Black-box distillation + data augmentation

---

White-Box Distillation: Logit Matching in Practice

Core Principle

The core of white-box distillation is having the student model learn the teacher model's output distribution (soft labels) rather than hard labels.

import torch
import torch.nn as nn
import torch.nn.functional as F

class LogitDistillationLoss(nn.Module):
    def __init__(self, temperature=2.0, alpha=0.7):
        super().__init__()
        self.temperature = temperature
        self.alpha = alpha
    
    def forward(self, student_logits, teacher_logits, labels):
        soft_loss = F.kl_div(
            F.log_softmax(student_logits / self.temperature, dim=-1),
            F.softmax(teacher_logits / self.temperature, dim=-1),
            reduction="batchmean",
        ) * (self.temperature ** 2)
        
        hard_loss = F.cross_entropy(student_logits, labels)
        
        return self.alpha * soft_loss + (1 - self.alpha) * hard_loss

class LLMDistillationTrainer:
    def __init__(
        self,
        teacher_model,
        student_model,
        tokenizer,
        temperature=2.0,
        alpha=0.7,
    ):
        self.teacher = teacher_model
        self.student = student_model
        self.tokenizer = tokenizer
        self.loss_fn = LogitDistillationLoss(temperature, alpha)
        self.teacher.eval()
    
    def distill_step(self, input_ids, attention_mask, labels):
        with torch.no_grad():
            teacher_outputs = self.teacher(
                input_ids=input_ids,
                attention_mask=attention_mask,
            )
            teacher_logits = teacher_outputs.logits
        
        student_outputs = self.student(
            input_ids=input_ids,
            attention_mask=attention_mask,
        )
        student_logits = student_outputs.logits
        
        shift_teacher = teacher_logits[..., :-1, :].contiguous()
        shift_student = student_logits[..., :-1, :].contiguous()
        shift_labels = labels[..., 1:].contiguous()
        
        loss = self.loss_fn(shift_student, shift_teacher, shift_labels)
        return loss

Temperature Parameter Tuning

Temperature T	Soft Label Distribution	Distillation Effect	Applicable Scenario
1.0	Sharp (close to hard labels)	Poor	Not recommended
2.0	Moderate	Good	General recommendation
4.0	Smooth	Moderate	When knowledge is rich
8.0	Over-smoothed	Poor	Not recommended

Alpha Parameter Tuning

α Value	Soft Label Weight	Hard Label Weight	Applicable Scenario
0.3	30%	70%	Poor data quality
0.5	50%	50%	Balanced
0.7	70%	30%	Good data quality (recommended)
0.9	90%	10%	Very strong teacher

---

Gray-Box Distillation: Feature Alignment in Practice

Intermediate Layer Feature Alignment

class FeatureDistillationLoss(nn.Module):
    def __init__(self, teacher_layers, student_layers, projection_dim=2048):
        super().__init__()
        self.teacher_layers = teacher_layers
        self.student_layers = student_layers
        self.layer_mapping = self._build_layer_mapping()
        
        self.projections = nn.ModuleDict({
            str(s_layer): nn.Linear(
                student_dim, projection_dim, bias=False
            )
            for s_layer, student_dim in student_layers
        })
    
    def _build_layer_mapping(self):
        t_count = len(self.teacher_layers)
        s_count = len(self.student_layers)
        return {
            s_idx: int(s_idx * t_count / s_count)
            for s_idx in range(s_count)
        }
    
    def forward(self, teacher_hidden, student_hidden):
        total_loss = 0.0
        for s_idx, t_idx in self.layer_mapping.items():
            s_feat = student_hidden[s_idx]
            t_feat = teacher_hidden[t_idx]
            
            s_proj = self.projections[str(s_idx)](s_feat)
            
            s_norm = F.normalize(s_proj, dim=-1)
            t_norm = F.normalize(t_feat, dim=-1)
            
            total_loss += (2 - 2 * (s_norm * t_norm).sum(dim=-1)).mean()
        
        return total_loss / len(self.layer_mapping)

class AttentionDistillationLoss(nn.Module):
    def __init__(self, num_heads=32):
        super().__init__()
        self.num_heads = num_heads
    
    def forward(self, teacher_attn, student_attn):
        t_heads = teacher_attn.shape[1]
        s_heads = student_attn.shape[1]
        
        head_mapping = {
            s: int(s * t_heads / s_heads)
            for s in range(s_heads)
        }
        
        total_loss = 0.0
        for s_h, t_h in head_mapping.items():
            s_attn = student_attn[:, s_h]
            t_attn = teacher_attn[:, t_h]
            total_loss += F.mse_loss(s_attn, t_attn)
        
        return total_loss / len(head_mapping)

Combined Distillation Loss

class CombinedDistillationLoss(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.logit_loss = LogitDistillationLoss(
            temperature=config.temperature,
            alpha=config.logit_alpha,
        )
        self.feature_loss = FeatureDistillationLoss(
            teacher_layers=config.teacher_layers,
            student_layers=config.student_layers,
        )
        self.attn_loss = AttentionDistillationLoss(
            num_heads=config.num_heads,
        )
        self.w_logit = config.w_logit
        self.w_feature = config.w_feature
        self.w_attn = config.w_attn
    
    def forward(self, student_outputs, teacher_outputs, labels):
        l_logit = self.logit_loss(
            student_outputs.logits, teacher_outputs.logits, labels
        )
        l_feature = self.feature_loss(
            teacher_outputs.hidden_states,
            student_outputs.hidden_states,
        )
        l_attn = self.attn_loss(
            teacher_outputs.attentions,
            student_outputs.attentions,
        )
        return (
            self.w_logit * l_logit
            + self.w_feature * l_feature
            + self.w_attn * l_attn
        )

Loss Combination	w_logit	w_feature	w_attn	Effect
Logit Only	1.0	0	0	Baseline
Logit + Feature	0.5	0.5	0	+3%
Logit + Attention	0.5	0	0.5	+2%
Full Combination	0.4	0.35	0.25	+5%

---

3-Stage Progressive Distillation

3-Stage Pipeline

┌──────────────────────────────────────────────────────────────┐
│              3-Stage Progressive Distillation                  │
│                                                                │
│  Stage 1: 70B → 30B                                           │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Teacher: 70B full parameters → Student: 30B          │    │
│  │  Strategy: White-box distillation (logit + feature + attention)  │    │
│  │  Data: 5M high-quality instructions                    │    │
│  │  Knowledge retention: 93%                              │    │
│  └──────────────────────────────────────────────────────┘    │
│                         ↓                                     │
│  Stage 2: 30B → 14B                                           │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Teacher: 30B distilled model → Student: 14B          │    │
│  │  Strategy: Gray-box distillation (logit + feature)    │    │
│  │  Data: 3M + Stage 1 data mixture                      │    │
│  │  Knowledge retention: 91%                              │    │
│  └──────────────────────────────────────────────────────┘    │
│                         ↓                                     │
│  Stage 3: 14B → 7B                                            │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Teacher: 14B distilled model → Student: 7B           │    │
│  │  Strategy: Gray-box distillation (logit) + online data augmentation  │    │
│  │  Data: 2M + data from previous two stages              │    │
│  │  Knowledge retention: 88%                              │    │
│  └──────────────────────────────────────────────────────┘    │
│                                                                │
│  Total knowledge retention: 93% × 91% × 88% ≈ 74%            │
│  Direct 70B→7B distillation: ~65%                              │
│  Progressive advantage: +9%                                    │
└──────────────────────────────────────────────────────────────┘

Progressive Distillation Implementation

class ProgressiveDistillation:
    def __init__(self, config):
        self.stages = config.stages
        self.current_stage = 0
    
    def get_stage_config(self, stage_idx):
        stage_configs = [
            {
                "teacher_size": "70B",
                "student_size": "30B",
                "strategy": "white_box",
                "data_size": 5_000_000,
                "epochs": 3,
                "lr": 5e-5,
                "loss_weights": {
                    "logit": 0.4, "feature": 0.35, "attn": 0.25
                },
            },
            {
                "teacher_size": "30B",
                "student_size": "14B",
                "strategy": "gray_box",
                "data_size": 3_000_000,
                "epochs": 4,
                "lr": 3e-5,
                "loss_weights": {
                    "logit": 0.5, "feature": 0.5, "attn": 0.0
                },
            },
            {
                "teacher_size": "14B",
                "student_size": "7B",
                "strategy": "gray_box_logit",
                "data_size": 2_000_000,
                "epochs": 5,
                "lr": 2e-5,
                "loss_weights": {
                    "logit": 0.7, "feature": 0.3, "attn": 0.0
                },
            },
        ]
        return stage_configs[stage_idx]
    
    def run_stage(self, stage_idx, teacher_model, student_model, dataset):
        config = self.get_stage_config(stage_idx)
        
        trainer = DistillationTrainer(
            teacher_model=teacher_model,
            student_model=student_model,
            train_dataset=dataset,
            args=DistillationArguments(
                num_train_epochs=config["epochs"],
                learning_rate=config["lr"],
                per_device_train_batch_size=4,
                gradient_accumulation_steps=8,
                warmup_ratio=0.03,
                bf16=True,
                logging_steps=100,
                save_strategy="epoch",
                strategy=config["strategy"],
                loss_weights=config["loss_weights"],
            ),
        )
        
        trainer.train()
        return trainer.model
    
    def run_all_stages(self, initial_teacher, dataset):
        current_teacher = initial_teacher
        
        for stage_idx in range(len(self.stages)):
            stage_config = self.get_stage_config(stage_idx)
            student = self._init_student(stage_config["student_size"])
            
            distilled = self.run_stage(
                stage_idx, current_teacher, student, dataset
            )
            
            current_teacher = distilled
            self._save_checkpoint(distilled, stage_idx)
        
        return current_teacher

Progressive vs Direct Distillation Comparison

Method	70B→7B	Training GPU Hours	Final MMLU	Final HumanEval
Direct Distillation	1 step	2000	58.2	42.1
Progressive (2-stage)	70B→14B→7B	2400	61.5	46.8
Progressive (3-stage)	70B→30B→14B→7B	2800	63.1	48.3
Progressive + Data Augmentation	3-stage + augmentation	3200	65.2	51.7

---

Multi-Task Distillation and Catastrophic Forgetting

Multi-Task Distillation Architecture

class MultiTaskDistillation:
    def __init__(self, teacher, student, task_weights):
        self.teacher = teacher
        self.student = student
        self.task_weights = task_weights
        self.task_buffers = {}
    
    def compute_loss(self, batch, task_id):
        with torch.no_grad():
            teacher_out = self.teacher(**batch)
        
        student_out = self.student(**batch)
        
        logit_loss = F.kl_div(
            F.log_softmax(student_out.logits / 2.0, dim=-1),
            F.softmax(teacher_out.logits / 2.0, dim=-1),
            reduction="batchmean",
        ) * 4.0
        
        return logit_loss
    
    def ewc_penalty(self, task_id):
        if task_id not in self.task_buffers:
            return 0.0
        
        buffer = self.task_buffers[task_id]
        penalty = 0.0
        for name, param in self.student.named_parameters():
            if name in buffer["params"]:
                penalty += (
                    (param - buffer["params"][name]) ** 2
                ).sum() * buffer["importance"][name]
        
        return penalty
    
    def save_task_buffer(self, task_id, dataloader):
        params = {}
        importance = {}
        
        for name, param in self.student.named_parameters():
            params[name] = param.data.clone()
            importance[name] = torch.zeros_like(param.data)
        
        self.student.eval()
        for batch in dataloader:
            self.student.zero_grad()
            loss = self.student(**batch).loss
            loss.backward()
            
            for name, param in self.student.named_parameters():
                if param.grad is not None:
                    importance[name] += param.grad.data ** 2
        
        for name in importance:
            importance[name] /= len(dataloader)
        
        self.task_buffers[task_id] = {
            "params": params,
            "importance": importance,
        }

Catastrophic Forgetting Defense Strategies

Strategy	Principle	Compute Overhead	Effect
EWC	Parameter importance regularization	Low	Moderate
Replay Buffer	Old task data replay	Medium	Good
LoRA Adapter	Per-task independent adapters	Low	Good
Progressive Freezing	Layer-wise freezing	Low	Moderate
Hybrid Strategy	EWC + Replay + LoRA	Medium	Best

DeepSeek-R1 Distillation Approach Analysis

The DeepSeek-R1 distillation approach is one of the most successful LLM distillation cases in 2026:

Stage	Teacher Model	Student Model	Data Volume	Key Technique
Stage 1	R1-671B	R1-Distill-70B	800K	Black-box distillation + CoT data
Stage 2	R1-Distill-70B	R1-Distill-32B	600K	White-box logit distillation
Stage 3	R1-Distill-32B	R1-Distill-8B	400K	Gray-box distillation + data augmentation

Key innovations:

1. CoT Data Distillation: Distilling not only the final answer but also the reasoning process
2. Rejection Sampling: The teacher model generates multiple reasoning paths, selecting the optimal path for distillation
3. Curriculum Learning: Gradually increasing distillation data difficulty from simple to complex

---

Summary and Resources

Key Takeaways

1. Strategy Selection: White-box delivers the best performance but at high cost; black-box is the most flexible but with lower performance; gray-box is the optimal balance point
2. Progressive Distillation: 3-stage progressive distillation improves knowledge retention by 9%+ over direct distillation
3. Multi-Task Distillation: The EWC + Replay + LoRA hybrid strategy effectively prevents catastrophic forgetting
4. DeepSeek-R1: CoT data distillation + rejection sampling is the current state-of-the-art distillation approach

Distillation Approach Recommendations

Scenario	Recommended Approach	Expected Effect
Have teacher weights + sufficient GPU	White-box 3-stage progressive	85%+ knowledge retention
Have teacher weights + limited GPU	Gray-box 2-stage	80%+ knowledge retention
API access only	Black-box + data augmentation	70%+ knowledge retention
Multi-task scenario	Hybrid strategy + EWC	Balanced across tasks

Need to handle JSON format conversion for distillation data? Try our JSON Formatter and Text Diff tools to quickly process distillation datasets.

Further Reading

• LLM Fine-Tuning Trio: LoRA/QLoRA/RLHF — Complementary approach to post-distillation fine-tuning
• LLM Inference Acceleration: Three-Engine Showdown — Inference deployment of distilled models
• LLM Evaluation Framework in Practice — Quality evaluation of distilled models
• Distilling Step-by-Step Paper — Step-by-step distillation methodology
• DeepSeek-R1 Technical Report — R1 distillation approach in detail