Summary
- Speculative decoding is the secret weapon for LLM inference acceleration: 2-3x speedup with zero accuracy loss, now a standard feature in inference services as of 2026
- 3 major speculation strategies: Draft Model, Medusa multi-head, and Tree Speculation, each with optimal use cases
- Draft model selection key: draft model speed must be >= 5x the target model, acceptance rate >= 70% to achieve 2x+ speedup
- Medusa multi-head requires no additional model: add prediction heads to the target model with low training cost and simple deployment
- This article provides vLLM + speculative decoding deployment solutions and custom draft model training in practice
Table of Contents
Speculative Decoding: The Secret Weapon for LLM Inference Acceleration
The Bottleneck of Autoregressive Decoding
The core bottleneck of LLM inference is the serial nature of autoregressive decoding: generating each token requires a complete forward pass, resulting in extremely low GPU utilization.
| Model Size |
Per-Token Latency |
GPU Utilization |
Throughput |
| 7B |
15ms |
25% |
40 tok/s |
| 14B |
28ms |
18% |
22 tok/s |
| 70B |
80ms |
12% |
8 tok/s |
| 405B |
250ms |
5% |
2.5 tok/s |
Core Idea of Speculative Decoding
┌──────────────────────────────────────────────────────────────┐
│ Speculative Decoding vs Autoregressive Decoding │
│ │
│ Autoregressive Decoding (token-by-token generation) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ [BOS] → f() → t1 → f() → t2 → f() → t3 → f() → t4 │ │
│ │ 4forward passes, 4tokens │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ Speculative Decoding (draft + verify) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Draft model fast generation: t1' t2' t3' t4' (1forward pass) │ │
│ │ Target model parallel verify: [t1' t2' t3' t4'] → f() (1forward pass) │ │
│ │ Assumet1't2't3'correct, t4'wrong: │ │
│ │ Acceptt1't2't3', rejectt4', correct tot4 │ │
│ │ 2forward passes, 4tokens → 2×speedup │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
2026 Speculative Decoding Approach Comparison
| Approach |
speedup |
Accuracy Loss |
Extra VRAM |
Deployment Complexity |
| Draft Model |
2-3× |
Zero |
30-50% |
Medium |
| Medusa Multi-Head |
1.5-2.5× |
Zero |
5-10% |
Low |
| Eagle Tree Speculation |
2.5-3.5× |
Zero |
10-15% |
Medium |
| Self-Speculation |
1.5-2× |
Zero |
0% |
Low |
| Speculative Sampling + Quantization |
2-3.5× |
Minimal |
15-25% |
Medium |
Speculative Decoding Principles and Mathematical Foundations
Acceptance Rate and speedup Ratio
The speedup ratio of speculative decoding depends on the acceptance rate of draft tokens:
Theoretical speedup = γ / (1 + (1-α) × γ)
Where:
- γ = Draft length (number of tokens per speculation)
- α = Acceptance rate (probability of draft tokens being accepted by the target model)
Examples:
- γ=5, α=0.8: speedup = 5 / (1 + 0.2×5) = 5/2 = 2.5×
- γ=5, α=0.9: speedup = 5 / (1 + 0.1×5) = 5/1.5 = 3.3×
- γ=5, α=0.6: speedup = 5 / (1 + 0.4×5) = 5/3 = 1.67×
- γ=3, α=0.8: speedup = 3 / (1 + 0.2×3) = 3/1.6 = 1.875×
Acceptance Rate vs speedup Ratio
| Acceptance Rate α |
γ=3 |
γ=5 |
γ=7 |
γ=10 |
| 0.5 |
1.2× |
1.25× |
1.27× |
1.25× |
| 0.6 |
1.36× |
1.47× |
1.52× |
1.47× |
| 0.7 |
1.58× |
1.82× |
1.94× |
1.96× |
| 0.8 |
1.88× |
2.5× |
2.92× |
3.33× |
| 0.9 |
2.31× |
3.33× |
4.38× |
5.88× |
Verification Algorithm Implementation
import torch
import torch.nn.functional as F
def speculative_decode(
target_model,
draft_model,
input_ids,
draft_length=5,
temperature=1.0,
):
draft_tokens = []
draft_probs = []
current_ids = input_ids.clone()
for _ in range(draft_length):
with torch.no_grad():
outputs = draft_model(current_ids)
next_token_logits = outputs.logits[:, -1, :]
next_token_prob = F.softmax(next_token_logits / temperature, dim=-1)
next_token = torch.multinomial(next_token_prob, num_samples=1)
draft_tokens.append(next_token)
draft_probs.append(next_token_prob)
current_ids = torch.cat([current_ids, next_token], dim=-1)
draft_token_ids = torch.cat(draft_tokens, dim=-1)
with torch.no_grad():
target_outputs = target_model(current_ids)
target_logits = target_outputs.logits[:, -(draft_length + 1):, :]
target_probs = F.softmax(target_logits / temperature, dim=-1)
accepted_tokens = []
n_accepted = 0
for i in range(draft_length):
draft_token = draft_token_ids[:, i]
draft_prob = draft_probs[i].gather(1, draft_token.unsqueeze(-1))
target_prob = target_probs[:, i].gather(1, draft_token.unsqueeze(-1))
accept_prob = torch.min(
torch.ones_like(target_prob),
target_prob / (draft_prob + 1e-10),
)
if torch.rand(1, device=accept_prob.device) < accept_prob:
accepted_tokens.append(draft_token)
n_accepted += 1
else:
residual_prob = F.relu(target_probs[:, i] - draft_probs[i])
residual_prob = residual_prob / residual_prob.sum(dim=-1, keepdim=True)
corrected_token = torch.multinomial(residual_prob, num_samples=1)
accepted_tokens.append(corrected_token)
break
if n_accepted == draft_length:
bonus_prob = target_probs[:, -1]
bonus_token = torch.multinomial(bonus_prob, num_samples=1)
accepted_tokens.append(bonus_token)
result = torch.cat(accepted_tokens, dim=-1)
return result, n_accepted
Draft Model Selection and Training
Draft Model Selection Principles
| Principle |
Requirement |
Reason |
| Speed |
≥5×Target Model |
Draft generation must not become a bottleneck |
| Acceptance Rate |
≥70% |
Below 70% speedup is less than 2x |
| Size |
≤1/5Target Model |
Controllable VRAM overhead |
| Vocabulary |
Same as target model |
Avoid vocabulary mapping loss |
Mainstream Draft Model Pairings
| Target Model |
Draft Model |
Speed Ratio |
Acceptance Rate |
speedup |
| LLaMA-3-70B |
LLaMA-3-8B |
6× |
75% |
2.3× |
| Qwen2.5-72B |
Qwen2.5-7B |
5.5× |
78% |
2.5× |
| DeepSeek-V3 |
DeepSeek-V2-Lite |
7× |
72% |
2.2× |
| Mistral-Large |
Mistral-7B |
6× |
76% |
2.4× |
| Gemma-2-27B |
Gemma-2-2B |
8× |
70% |
2.1× |
Draft Model Training
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer
from datasets import load_dataset
class DraftModelTrainer:
def __init__(self, target_model_path, draft_model_path, output_path):
self.target = AutoModelForCausalLM.from_pretrained(
target_model_path, torch_dtype=torch.float16, device_map="auto"
)
self.draft = AutoModelForCausalLM.from_pretrained(
draft_model_path, torch_dtype=torch.float16, device_map="auto"
)
self.tokenizer = AutoTokenizer.from_pretrained(target_model_path)
self.output_path = output_path
def generate_distillation_data(self, prompts, num_samples=50000):
distill_data = []
for prompt in prompts:
inputs = self.tokenizer(prompt, return_tensors="pt").to(self.target.device)
with torch.no_grad():
outputs = self.target.generate(
**inputs,
max_new_tokens=256,
temperature=0.7,
do_sample=True,
num_return_sequences=5,
)
for output in outputs:
text = self.tokenizer.decode(output, skip_special_tokens=True)
distill_data.append({"text": text})
if len(distill_data) >= num_samples:
break
return distill_data
def train(self, distill_data, epochs=3, lr=5e-5):
from trl import SFTTrainer, SFTConfig
sft_config = SFTConfig(
output_dir=self.output_path,
num_train_epochs=epochs,
per_device_train_batch_size=4,
gradient_accumulation_steps=8,
learning_rate=lr,
warmup_ratio=0.03,
bf16=True,
logging_steps=100,
save_strategy="epoch",
max_seq_length=2048,
)
trainer = SFTTrainer(
model=self.draft,
train_dataset=distill_data,
args=sft_config,
)
trainer.train()
self.draft.save_pretrained(self.output_path)
return self.draft
Draft Model Acceptance Rate Optimization
| Optimization Method |
Acceptance Rate Improvement |
Approach |
| Distillation Fine-tuning |
+10-15% |
Train draft model with target model outputs |
| Vocabulary Alignment |
+5-8% |
Ensure draft and target vocabularies match |
| Temperature Alignment |
+3-5% |
Match sampling temperatures between draft and target |
| Context Alignment |
+5-10% |
Use the same system prompt for the draft model |
| Combined Optimization |
+20-30% |
All of the above |
Medusa Multi-Head Speculation in Practice
Medusa Architecture
┌──────────────────────────────────────────────────────────────┐
│ Medusa Multi-Headarchitecture │
│ │
│ Original LLM Backbone │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Input → Transformer Layers → Hidden State │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ │
│ Original LM Head (predict t+1) │
│ ┌──────────┐ │
│ │ LM Head │ → P(t+1) │
│ └──────────┘ │
│ │
│ Medusa Head 0 (predict t+2) │
│ ┌──────────────┐ │
│ │ Linear+Softmax│ → P(t+2) │
│ └──────────────┘ │
│ │
│ Medusa Head 1 (predict t+3) │
│ ┌──────────────┐ │
│ │ Linear+Softmax│ → P(t+3) │
│ └──────────────┘ │
│ │
│ Medusa Head K-1 (predict t+K) │
│ ┌──────────────┐ │
│ │ Linear+Softmax│ → P(t+K) │
│ └──────────────┘ │
└──────────────────────────────────────────────────────────────┘
Medusa Implementation
import torch
import torch.nn as nn
from transformers import PreTrainedModel
class MedusaHead(nn.Module):
def __init__(self, hidden_size, vocab_size):
super().__init__()
self.linear = nn.Linear(hidden_size, vocab_size, bias=False)
def forward(self, hidden_states):
return self.linear(hidden_states)
class MedusaModel(nn.Module):
def __init__(self, base_model, num_heads=4):
super().__init__()
self.base_model = base_model
self.num_heads = num_heads
hidden_size = base_model.config.hidden_size
vocab_size = base_model.config.vocab_size
self.medusa_heads = nn.ModuleList([
MedusaHead(hidden_size, vocab_size)
for _ in range(num_heads)
])
for head in self.medusa_heads:
nn.init.xavier_uniform_(head.linear.weight)
def forward(self, input_ids, attention_mask=None):
outputs = self.base_model.model(
input_ids=input_ids,
attention_mask=attention_mask,
use_cache=True,
)
hidden_states = outputs.last_hidden_state
base_logits = self.base_model.lm_head(hidden_states)
medusa_logits = []
for head in self.medusa_heads:
medusa_logits.append(head(hidden_states))
return base_logits, medusa_logits, outputs.past_key_values
class MedusaDecoding:
def __init__(self, model, tokenizer, num_heads=4, top_k=10):
self.model = model
self.tokenizer = tokenizer
self.num_heads = num_heads
self.top_k = top_k
def generate(self, input_ids, max_new_tokens=256):
generated = input_ids.clone()
for _ in range(max_new_tokens):
base_logits, medusa_logits, _ = self.model(generated)
base_prob = torch.softmax(base_logits[:, -1, :], dim=-1)
next_token = torch.argmax(base_prob, dim=-1, keepdim=True)
candidates = [next_token]
for head_idx in range(self.num_heads):
head_prob = torch.softmax(
medusa_logits[head_idx][:, -1, :], dim=-1
)
top_tokens = torch.topk(head_prob, self.top_k)
candidates.append(top_tokens.indices[:, :1])
candidate_ids = torch.cat(candidates, dim=-1)
with torch.no_grad():
verify_outputs = self.model.base_model(candidate_ids)
verify_logits = self.model.base_model.lm_head(
verify_outputs.last_hidden_state
)
accepted = [next_token]
for i, cand in enumerate(candidates[1:]):
verify_prob = torch.softmax(verify_logits[:, i, :], dim=-1)
if verify_prob.gather(1, cand).item() > 0.1:
accepted.append(cand)
else:
corrected = torch.argmax(verify_prob, dim=-1, keepdim=True)
accepted.append(corrected)
break
new_tokens = torch.cat(accepted, dim=-1)
generated = torch.cat([generated, new_tokens], dim=-1)
if new_tokens.shape[-1] > 1:
break
return generated
Medusa Training
class MedusaTrainer:
def __init__(self, medusa_model, lr=1e-4):
self.model = medusa_model
self.optimizer = torch.optim.AdamW(
[p for p in self.model.medusa_heads.parameters()],
lr=lr,
weight_decay=0.01,
)
self.base_model = medusa_model.base_model
for param in self.base_model.parameters():
param.requires_grad = False
def train_step(self, input_ids, labels):
with torch.no_grad():
outputs = self.base_model.model(input_ids=input_ids)
hidden_states = outputs.last_hidden_state
total_loss = 0
for head_idx, head in enumerate(self.model.medusa_heads):
shift = head_idx + 1
head_logits = head(hidden_states[:, :-shift, :])
head_labels = labels[:, shift:]
loss = F.cross_entropy(
head_logits.reshape(-1, head_logits.size(-1)),
head_labels.reshape(-1),
)
total_loss += loss
total_loss /= len(self.model.medusa_heads)
total_loss.backward()
self.optimizer.step()
self.optimizer.zero_grad()
return total_loss.item()
| Medusa Config |
Heads |
Top-K |
speedup |
Extra VRAM |
| Medusa-2 |
2 |
5 |
1.5× |
3% |
| Medusa-4 |
4 |
10 |
2.0× |
5% |
| Medusa-4 |
4 |
20 |
2.2× |
5% |
| Medusa-8 |
8 |
10 |
2.5× |
10% |
Tree Speculation and the Eagle Approach
Eagle Tree Speculation Principles
┌──────────────────────────────────────────────────────────────┐
│ Eagle Tree Speculation │
│ │
│ Traditional Linear Speculation: │
│ t1 → t2 → t3 → t4 → t5 (1path) │
│ │
│ Eagle Tree Speculation: │
│ t1 │
│ / \ │
│ t2a t2b │
│ / \ / \ │
│ t3a t3b t3c t3d │
│ │
│ Multiple candidate paths verified in parallel, selecting the path with the highest acceptance rate │
│ Advantage: even if one path is rejected, others may still be accepted │
└──────────────────────────────────────────────────────────────┘
Eagle Implementation
class EagleSpeculativeDecoder:
def __init__(self, target_model, draft_model, tree_width=4, tree_depth=3):
self.target = target_model
self.draft = draft_model
self.tree_width = tree_width
self.tree_depth = tree_depth
def build_speculation_tree(self, input_ids):
tree_nodes = [{"ids": input_ids.clone(), "parent": None}]
leaves = []
for depth in range(self.tree_depth):
new_nodes = []
for node in tree_nodes[-self.tree_width ** depth:]:
with torch.no_grad():
outputs = self.draft(node["ids"])
probs = torch.softmax(outputs.logits[:, -1, :], dim=-1)
top_k = torch.topk(probs, self.tree_width)
for i in range(self.tree_width):
child_ids = torch.cat([
node["ids"],
top_k.indices[:, i:i+1],
], dim=-1)
child = {
"ids": child_ids,
"parent": node,
"prob": top_k.values[:, i].item(),
"token": top_k.indices[:, i],
}
new_nodes.append(child)
if depth == self.tree_depth - 1:
leaves.append(child)
tree_nodes.extend(new_nodes)
return tree_nodes, leaves
def verify_tree(self, tree_nodes, leaves):
best_path = []
best_score = 0
for leaf in leaves:
path = []
node = leaf
while node["parent"] is not None:
path.append(node)
node = node["parent"]
path.reverse()
with torch.no_grad():
full_ids = leaf["ids"]
outputs = self.target(full_ids)
target_probs = torch.softmax(outputs.logits, dim=-1)
score = 1.0
accepted = 0
for i, step in enumerate(path):
target_prob = target_probs[:, -(len(path) - i), :].gather(
1, step["token"]
).item()
score *= target_prob
if target_prob > 0.3:
accepted += 1
else:
break
if score > best_score:
best_score = score
best_path = path[:accepted]
return best_path
Tree Speculation vs Linear Speculation
| Approach |
Candidate Paths |
Acceptance Rate |
speedup |
Compute Overhead |
| Linearγ=5 |
1 |
70% |
2.3× |
Low |
| Tree w=2,d=3 |
8 |
82% |
2.8× |
Medium |
| Tree w=3,d=3 |
27 |
88% |
3.2× |
Medium-High |
| Tree w=4,d=3 |
64 |
92% |
3.5× |
High |
| Eagle(w=4,d=4) |
256 |
95% |
3.8× |
High |
Production Deployment: vLLM + Speculative Decoding
vLLM Speculative Decoding Configuration
from vllm import LLM, SamplingParams
llm = LLM(
model="Qwen/Qwen2.5-72B-Instruct",
speculative_model="Qwen/Qwen2.5-7B-Instruct",
num_speculative_tokens=5,
speculative_max_model_len=4096,
gpu_memory_utilization=0.9,
tensor_parallel_size=4,
trust_remote_code=True,
)
sampling_params = SamplingParams(
temperature=0.7,
top_p=0.9,
max_tokens=512,
)
outputs = llm.generate(
["Please explain the basic principles of quantum computing"],
sampling_params=sampling_params,
)
for output in outputs:
print(output.outputs[0].text)
| Configuration |
Target Model |
Draft Model |
Throughput (tok/s) |
speedup |
| No Speculation |
72B |
None |
8 |
1× |
| Draft Speculation |
72B |
7B |
18 |
2.25× |
| Draft Speculation |
72B |
0.5B |
22 |
2.75× |
| Medusa-4 |
72B |
None (built-in) |
16 |
2.0× |
| Eagle |
72B |
7B |
25 |
3.1× |
Production Deployment Considerations
| Consideration |
Description |
Solution |
| Draft ModelVRAM |
Additional 30-50% usage |
Quantize draft model to INT8 |
| Acceptance Rate Fluctuation |
Significant variation across tasks |
Dynamically adjust draft length |
| KV Cache Management |
KV of rejected tokens needs cleanup |
vLLM handles automatically |
| Batch Processing Compatibility |
Conflict between speculative decoding and continuous batching |
vLLM now supported |
| First-Token Latency |
Speculative decoding increases first-token latency |
Disable speculation in chat scenarios |
Dynamic Speculation Strategy
class DynamicSpeculativeConfig:
def __init__(self):
self.min_draft_length = 2
self.max_draft_length = 8
self.target_acceptance_rate = 0.75
self.current_draft_length = 5
self.history = []
def update(self, acceptance_rate):
self.history.append(acceptance_rate)
if len(self.history) > 100:
self.history = self.history[-100:]
avg_rate = sum(self.history) / len(self.history)
if avg_rate > 0.85:
self.current_draft_length = min(
self.current_draft_length + 1,
self.max_draft_length,
)
elif avg_rate < 0.65:
self.current_draft_length = max(
self.current_draft_length - 1,
self.min_draft_length,
)
return self.current_draft_length
Summary and Further Reading
Key Takeaways
- Speculative decoding is the free lunch of LLM inference acceleration: 2-3x speedup with zero accuracy loss
- Draft Model: Selection key is speed >= 5x + acceptance rate >= 70%, distillation fine-tuning can improve acceptance rate by 20-30%
- Medusa Multi-Head: No additional model needed, 5% VRAM overhead for 2x speedup, simplest deployment
- Eagle Tree Speculation: Multi-path parallel verification, 3.5x+ speedup, ideal for high-throughput scenarios
Recommended Approaches
| Scenario |
Recommended Approach |
speedup |
| Quick Deployment |
Medusa-4 |
2.0× |
| General Inference |
Draft Model(7B→72B) |
2.5× |
| Highthroughput |
Eagle Tree Speculation |
3.5× |
| VRAM Constrained |
Medusa + Quantized Draft |
2.2× |
Need to handle inference data format conversion? Try our JSON to YAML tool and Text Diff Comparison to quickly process speculative decoding debug data.
Further Reading