K8s 1.30+大模型推理彈性調度實戰：vLLM部署與KEDA自動伸縮深度指南
云原生
摘要
- K8s 1.30+的SidecarContainers、ReadWriteOncePod PVC等特性為大模型推理服務提供了更精細的資源控制能力
- vLLM的PagedAttention+ContinuousBatching是當前LLM推理服務的事實標準，單卡吞吐量可達傳統方案的3-5倍
- KEDA自定義指標伸縮（基於請求佇列深度、GPU利用率、TTFT延遲）比HPA CPU指標更精準匹配推理負載
- 推理服務3層彈性策略：Pod級HPA→Deployment級滾動更新→Cluster級集群伸縮
- 本文提供從vLLM K8s部署到KEDA自動伸縮的完整方案，含生產級YAML與性能調優參數
目錄
K8s 1.30+與大模型推理的新契合點
Kubernetes 1.30（Ubykah）引入了多項對AI推理工作負載至關重要的特性。SidecarContainers正式GA，使得日誌採集、指標暴露等Sidecar容器不再阻塞Pod終止，推理Pod的滾動更新速度提升60%。ReadWriteOncePod PVC存取模式讓模型權重檔案的儲存更加安全高效，避免多Pod同時掛載導致的寫入衝突。
┌──────────────────────────────────────────────────────────────────┐
│ K8s 1.30+ LLM推理架構 │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Inference Gateway (Nginx/Traefik) │ │
│ │ - 負載均衡: 最少連接數演算法 │ │
│ │ - 限流熔斷: 令牌桶 + 熔斷器 │ │
│ │ - A/B路由: 模型版本灰度發布 │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ vLLM Pod (x N) │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ vLLM Server│ │ Log Sidecar│ │ Metrics │ │ │
│ │ │ (主容器) │ │(1.30正式GA)│ │ Sidecar │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ GPU: NVIDIA A100/H100 │ │
│ │ 模型: RWX Pod PVC掛載 │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ KEDA Scaler │ │
│ │ - 指標: 請求佇列深度 / GPU顯存利用率 / TTFT │ │
│ │ - 最小副本: 2 (高可用) │ │
│ │ - 最大副本: 20 (彈性上限) │ │
│ │ - 冷卻: 300s (避免抖動) │ │
│ └──────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
K8s 1.30+關鍵特性對LLM推理的影響
| K8s特性 | 版本 | 對LLM推理的影響 |
|---|---|---|
| SidecarContainers GA | 1.30 | 推理Pod滾動更新加速60%，Sidecar不再阻塞終止 |
| ReadWriteOncePod PVC | 1.30 | 模型權重檔案獨佔掛載，避免寫入衝突 |
| PodReadyToStartContainers | 1.31 | 推理服務就緒檢測更精準，避免流量打到未初始化Pod |
| AppArmor GA | 1.31 | 推理Pod安全加固，防止模型權重洩露 |
| Structured Authorization | 1.32 | 多租戶推理集群的細粒度權限控制 |
vLLM推理引擎K8s部署
vLLM核心優化原理
vLLM透過PagedAttention和ContinuousBatching兩大核心技術，將GPU顯存利用率從傳統方案的30-40%提升到90%以上。PagedAttention借鑑作業系統虛擬記憶體的分頁機制，將KV Cache按塊管理，消除顯存碎片。ContinuousBatching在請求級別進行排程，新請求無需等待當前批次完成即可加入執行。
┌──────────────────────────────────────────────────────────────┐
│ vLLM PagedAttention + ContinuousBatching │
│ │
│ 傳統方案 (Static Batching): │
│ ┌────┬────┬────┬────┐ │
│ │ R1 │ R2 │ R3 │ R4 │ 等最長請求完成才能開始下一批 │
│ └────┴────┴────┴────┘ │
│ ████████████████████ GPU空閒等待 │
│ │
│ vLLM (Continuous Batching): │
│ ┌────┬────┬────┬────┐ │
│ │ R1 │ R2 │ R3 │ R4 │ R1完成後立即插入R5 │
│ └────┴────┴────┴────┘ │
│ ┌────┬────┬────┬────┐ │
│ │ R5 │ R2 │ R3 │ R4 │ R2完成後立即插入R6 │
│ └────┴────┴────┴────┘ │
│ ████████████████ GPU持續高利用率 │
│ │
│ PagedAttention KV Cache: │
│ ┌───┬───┬───┬───┬───┬───┬───┬───┐ │
│ │P0 │P1 │P2 │P3 │P4 │P5 │P6 │P7 │ 按需分配物理塊 │
│ └───┴───┴───┴───┴───┴───┴───┴───┘ │
│ R1→[P0,P1,P2] R2→[P3,P4] R3→[P5,P6,P7] │
│ 無碎片，顯存利用率90%+ │
└──────────────────────────────────────────────────────────────┘
vLLM K8s Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen2-72b
namespace: llm-inference
labels:
app: vllm
model: qwen2-72b-instruct
spec:
replicas: 2
selector:
matchLabels:
app: vllm
model: qwen2-72b-instruct
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: vllm
model: qwen2-72b-instruct
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
spec:
terminationGracePeriodSeconds: 120
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: vllm
model: qwen2-72b-instruct
topologyKey: kubernetes.io/hostname
containers:
- name: vllm-server
image: vllm/vllm-openai:v0.6.0
command:
- python
- -m
- vllm.entrypoints.openai.api_server
args:
- --model
- /models/Qwen2.5-72B-Instruct
- --tensor-parallel-size
- "2"
- --max-model-len
- "32768"
- --gpu-memory-utilization
- "0.92"
- --max-num-seqs
- "256"
- --enable-prefix-caching
- --enable-chunked-prefill
- --port
- "8000"
ports:
- containerPort: 8000
resources:
requests:
nvidia.com/gpu: "2"
cpu: "8"
memory: 32Gi
limits:
nvidia.com/gpu: "2"
cpu: "16"
memory: 64Gi
env:
- name: MODEL_NAME
value: "qwen2-72b-instruct"
- name: VLLM_WORKER_MULTIPROC_METHOD
value: "ray"
volumeMounts:
- name: model-storage
mountPath: /models
- name: shm
mountPath: /dev/shm
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180
periodSeconds: 15
failureThreshold: 5
- name: metrics-exporter
image: prometheus-exporter/vllm-metrics:v1.0
ports:
- containerPort: 9090
resources:
requests:
cpu: "100m"
memory: 128Mi
volumes:
- name: model-storage
persistentVolumeClaim:
claimName: model-weights-pvc
- name: shm
emptyDir:
medium: Memory
sizeLimit: 16Gi
模型權重PVC（ReadWriteOncePod）
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-weights-pvc
namespace: llm-inference
spec:
accessModes:
- ReadWriteOncePod
storageClassName: local-ssd
resources:
requests:
storage: 200Gi
vLLM性能調優參數
| 參數 | 預設值 | 推薦值 | 說明 |
|---|---|---|---|
--gpu-memory-utilization |
0.9 | 0.92 | GPU顯存利用率上限 |
--max-model-len |
模型預設 | 32768 | 最大序列長度，影響KV Cache大小 |
--max-num-seqs |
256 | 256 | 最大並發序列數 |
--enable-prefix-caching |
false | true | 前綴快取，相同prompt復用KV Cache |
--enable-chunked-prefill |
false | true | 分塊預填充，降低首Token延遲 |
--swap-space |
4GB | 8GB | CPU交換空間，長序列溢出時使用 |
KEDA自定義指標自動伸縮
為什麼HPA不夠
K8s原生HPA基於CPU/記憶體利用率伸縮，但LLM推理服務是GPU密集型，CPU利用率無法反映真實負載。一個推理Pod的CPU可能只有30%，但GPU顯存已滿、請求佇列積壓嚴重。KEDA（Kubernetes Event-Driven Autoscaling）支援自定義指標伸縮，可基於vLLM暴露的請求佇列深度、GPU利用率、TTFT（Time To First Token）等指標精準伸縮。
KEDA部署
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-scaler
namespace: llm-inference
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: vllm-qwen2-72b
minReplicaCount: 2
maxReplicaCount: 20
cooldownPeriod: 300
pollingInterval: 15
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 120
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: Pods
value: 3
periodSeconds: 60
- type: Percent
value: 50
periodSeconds: 60
selectPolicy: Max
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_num_requests_waiting
threshold: "10"
query: |
avg_over_time(vllm_num_requests_waiting{model="qwen2-72b-instruct"}[1m])
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_gpu_cache_usage_perc
threshold: "80"
query: |
avg_over_time(vllm_gpu_cache_usage_perc{model="qwen2-72b-instruct"}[2m])
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_e2e_request_latency_seconds
threshold: "5"
query: |
histogram_quantile(0.95, sum(rate(vllm_e2e_request_latency_seconds_bucket{model="qwen2-72b-instruct"}[2m])) by (le))
多指標伸縮策略
┌──────────────────────────────────────────────────────────────┐
│ KEDA多指標伸縮決策流程 │
│ │
│ 指標1: 請求佇列深度 │
│ vllm_num_requests_waiting > 10 → 擴容 │
│ │
│ 指標2: GPU KV Cache利用率 │
│ vllm_gpu_cache_usage_perc > 80% → 擴容 │
│ │
│ 指標3: P95端到端延遲 │
│ vllm_e2e_request_latency_seconds > 5s → 擴容 │
│ │
│ 決策: 任一指標觸發即擴容，所有指標正常才縮容 │
│ 冷卻: 擴容30s / 縮容600s (避免抖動) │
│ 步長: 擴容max(3 Pod, 50%) / 縮容1 Pod/2min │
└──────────────────────────────────────────────────────────────┘
GPU資源優化與多租戶隔離
NVIDIA MIG分區
在A100/H100上使用MIG（Multi-Instance GPU）將單張GPU劃分為多個實例，實現多租戶推理隔離。
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
name: mig-instance
namespace: llm-inference
spec:
devices:
- name: gpu
driver: nvidia.com/gpu
requests:
- name: mig-instance
deviceClassName: mig.nvidia.com/1g.10gb
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-small-model
namespace: llm-inference
spec:
replicas: 4
selector:
matchLabels:
app: vllm-small
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.0
args:
- --model
- /models/Qwen2.5-7B-Instruct
- --gpu-memory-utilization
- "0.90"
- --max-model-len
- "8192"
resources:
claims:
- name: mig-instance
resourceClaims:
- name: mig-instance
claimName: mig-instance
GPU時間片共享
對於低優先級推理任務，使用GPU時間片共享，多個Pod共享同一張GPU。
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-batch-inference
namespace: llm-inference
spec:
replicas: 2
selector:
matchLabels:
app: vllm-batch
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.0
args:
- --model
- /models/Qwen2.5-7B-Instruct
- --gpu-memory-utilization
- "0.45"
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "0"
- name: CUDA_MPS_PIPE_DIRECTORY
value: "/tmp/nvidia-mps"
- name: CUDA_MPS_LOG_DIRECTORY
value: "/tmp/nvidia-log"
多租戶GPU資源配額
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-quota-team-a
namespace: team-a-inference
spec:
hard:
requests.nvidia.com/gpu: "8"
limits.nvidia.com/gpu: "8"
requests.cpu: "32"
limits.cpu: "64"
requests.memory: 128Gi
limits.memory: 256Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-quota-team-b
namespace: team-b-inference
spec:
hard:
requests.nvidia.com/gpu: "4"
limits.nvidia.com/gpu: "4"
3層彈性策略與故障自癒
3層彈性架構
┌──────────────────────────────────────────────────────────────┐
│ 3層彈性策略架構 │
│ │
│ Layer 1: Pod級 (KEDA HPA) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 基於自定義指標的Pod副本數伸縮 │ │
│ │ 回應時間: 30-60s │ │
│ │ 粒度: 單個Deployment │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ 仍不足 │
│ Layer 2: Deployment級 (滾動更新) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 模型版本切換、參數調整的滾動更新 │ │
│ │ 回應時間: 5-15min │ │
│ │ 粒度: 單個模型服務 │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ 集群級 │
│ Layer 3: Cluster級 (Cluster Autoscaler) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 節點池擴縮容，新增GPU節點 │ │
│ │ 回應時間: 3-10min (含GPU驅動初始化) │ │
│ │ 粒度: 整個集群 │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
故障自癒：Pod異常檢測與恢復
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen2-72b
namespace: llm-inference
spec:
replicas: 2
selector:
matchLabels:
app: vllm
template:
spec:
containers:
- name: vllm-server
image: vllm/vllm-openai:v0.6.0
startupProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
failureThreshold: 30
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180
periodSeconds: 15
failureThreshold: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: vllm-pdb
namespace: llm-inference
spec:
minAvailable: 1
selector:
matchLabels:
app: vllm
model: qwen2-72b-instruct
推理服務預熱策略
import httpx
import asyncio
import logging
logger = logging.getLogger(__name__)
class InferenceWarmer:
def __init__(self, base_url: str, model: str, warmup_prompts: list[str]):
self.base_url = base_url
self.model = model
self.warmup_prompts = warmup_prompts
async def warmup(self, rounds: int = 3):
for round_num in range(rounds):
tasks = [self._send_request(prompt) for prompt in self.warmup_prompts]
results = await asyncio.gather(*tasks, return_exceptions=True)
success_count = sum(1 for r in results if not isinstance(r, Exception))
logger.info(f"Warmup round {round_num + 1}: {success_count}/{len(tasks)} succeeded")
async def _send_request(self, prompt: str) -> dict:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.post(
f"{self.base_url}/v1/completions",
json={
"model": self.model,
"prompt": prompt,
"max_tokens": 32,
"temperature": 0.0,
},
)
return response.json()
async def warmup_new_pod(pod_ip: str, model: str):
warmer = InferenceWarmer(
base_url=f"http://{pod_ip}:8000",
model=model,
warmup_prompts=[
"Hello, how are you?",
"What is the capital of France?",
"Explain quantum computing briefly.",
],
)
await warmer.warmup(rounds=3)
生產級推理集群可觀測性
vLLM Prometheus指標
apiVersion: v1
kind: ConfigMap
metadata:
name: vllm-alerts
namespace: llm-inference
data:
alerts.yml: |
groups:
- name: vllm.rules
rules:
- alert: VLLMHighQueueDepth
expr: vllm_num_requests_waiting > 20
for: 2m
labels:
severity: warning
annotations:
summary: "vLLM request queue depth is high"
description: "Queue depth {{ $value }} exceeds threshold 20"
- alert: VLLMHighTTFT
expr: histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "vLLM TTFT P95 is high"
description: "TTFT P95 {{ $value }}s exceeds 2s threshold"
- alert: VLLMGPUOOM
expr: vllm_gpu_cache_usage_perc > 0.95
for: 1m
labels:
severity: critical
annotations:
summary: "vLLM GPU cache near OOM"
description: "GPU cache usage {{ $value }}% exceeds 95%"
- alert: VLLMHighErrorRate
expr: rate(vllm_request_errors_total[5m]) / rate(vllm_request_total[5m]) > 0.05
for: 3m
labels:
severity: critical
annotations:
summary: "vLLM error rate is high"
description: "Error rate {{ $value | humanizePercentage }} exceeds 5%"
推理服務Grafana Dashboard關鍵面板
| 面板 | 指標 | 說明 |
|---|---|---|
| 請求吞吐量 | rate(vllm_request_total[1m]) |
每秒完成請求數 |
| 佇列深度 | vllm_num_requests_waiting |
等待中的請求數 |
| TTFT P95 | histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) |
首 Token延遲 |
| TPOT P95 | histogram_quantile(0.95, rate(vllm_time_per_output_token_seconds_bucket[5m])) |
每 Token延遲 |
| GPU Cache利用率 | vllm_gpu_cache_usage_perc |
KV Cache顯存佔用 |
| 並發序列數 | vllm_num_requests_running |
正在執行的序列數 |
| 錯誤率 | rate(vllm_request_errors_total[5m]) / rate(vllm_request_total[5m]) |
請求失敗率 |
推理服務性能基準（Qwen2.5-72B, 2xA100）
| 場景 | 並發數 | TTFT P50 | TTFT P95 | TPOT | 吞吐量(tokens/s) |
|---|---|---|---|---|---|
| 短文本(128 in, 256 out) | 16 | 0.3s | 0.8s | 25ms | 4096 |
| 中等文本(512 in, 512 out) | 8 | 0.5s | 1.2s | 30ms | 2048 |
| 長文本(2048 in, 1024 out) | 4 | 1.2s | 2.5s | 35ms | 1024 |
| 超長文本(8192 in, 2048 out) | 2 | 3.5s | 6.0s | 40ms | 512 |
總結與引流
K8s 1.30+為大模型推理服務提供了更精細的資源控制能力。vLLM的PagedAttention+ContinuousBatching是LLM推理的事實標準，KEDA自定義指標伸縮比HPA CPU指標更精準匹配推理負載。3層彈性策略（Pod級HPA→Deployment級滾動更新→Cluster級集群伸縮）確保推理服務在不同負載場景下的穩定運行。
開發要點回顧：
- vLLM部署：啟用prefix-caching和chunked-prefill，GPU顯存利用率設為0.92，max-num-seqs根據GPU規格調整
- KEDA伸縮：基於請求佇列深度、GPU Cache利用率、TTFT延遲三指標觸發，擴容30s冷卻/縮容600s冷卻
- GPU優化：大模型用獨佔GPU，小模型用MIG分區，批量推理用時間片共享
- 故障自癒：startupProbe→readinessProbe→livenessProbe三級探測，PDB保證最小可用副本
- 可觀測性：Prometheus+Grafana全鏈路監控，關鍵告警：佇列深度>20、TTFT>2s、GPU Cache>95%
相關閱讀：
- AI Agent多智能體編排實戰：從單Agent到生產級多Agent協作系統 — Agent編排中的推理服務調度
- K8s GPU調度實戰：MIG分區與GPU共享 — GPU資源管理與多租戶隔離
- 大模型RAG系統從零到生產級全鏈路實戰 — RAG系統中的推理服務整合
權威參考：
本站提供瀏覽器本地工具，免註冊即可試用 →
#K8s大模型推理#LLM弹性调度#vLLM K8s部署#推理服务自动伸缩#GPU推理优化#2026