K8s 1.30+大模型推理弹性调度实战：vLLM部署与KEDA自动伸缩深度指南
云原生
摘要
- K8s 1.30+的SidecarContainers、ReadWriteOncePod PVC等特性为大模型推理服务提供了更精细的资源控制能力
- vLLM的PagedAttention+ContinuousBatching是当前LLM推理服务的事实标准，单卡吞吐量可达传统方案的3-5倍
- KEDA自定义指标伸缩（基于请求队列深度、GPU利用率、TTFT延迟）比HPA CPU指标更精准匹配推理负载
- 推理服务3层弹性策略：Pod级HPA→Deployment级滚动更新→Cluster级集群伸缩
- 本文提供从vLLM K8s部署到KEDA自动伸缩的完整方案，含生产级YAML与性能调优参数
目录
K8s 1.30+与大模型推理的新契合点
Kubernetes 1.30（Ubykah）引入了多项对AI推理工作负载至关重要的特性。SidecarContainers正式GA，使得日志采集、指标暴露等Sidecar容器不再阻塞Pod终止，推理Pod的滚动更新速度提升60%。ReadWriteOncePod PVC访问模式让模型权重文件的存储更加安全高效，避免多Pod同时挂载导致的写入冲突。
┌──────────────────────────────────────────────────────────────────┐
│ K8s 1.30+ LLM推理架构 │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Inference Gateway (Nginx/Traefik) │ │
│ │ - 负载均衡: 最少连接数算法 │ │
│ │ - 限流熔断: 令牌桶 + 熔断器 │ │
│ │ - A/B路由: 模型版本灰度发布 │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ vLLM Pod (x N) │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ vLLM Server│ │ Log Sidecar│ │ Metrics │ │ │
│ │ │ (主容器) │ │(1.30正式GA)│ │ Sidecar │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ GPU: NVIDIA A100/H100 │ │
│ │ 模型: RWX Pod PVC挂载 │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ KEDA Scaler │ │
│ │ - 指标: 请求队列深度 / GPU显存利用率 / TTFT │ │
│ │ - 最小副本: 2 (高可用) │ │
│ │ - 最大副本: 20 (弹性上限) │ │
│ │ - 冷却: 300s (避免抖动) │ │
│ └──────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
K8s 1.30+关键特性对LLM推理的影响
| K8s特性 | 版本 | 对LLM推理的影响 |
|---|---|---|
| SidecarContainers GA | 1.30 | 推理Pod滚动更新加速60%，Sidecar不再阻塞终止 |
| ReadWriteOncePod PVC | 1.30 | 模型权重文件独占挂载，避免写入冲突 |
| PodReadyToStartContainers | 1.31 | 推理服务就绪检测更精准，避免流量打到未初始化Pod |
| AppArmor GA | 1.31 | 推理Pod安全加固，防止模型权重泄露 |
| Structured Authorization | 1.32 | 多租户推理集群的细粒度权限控制 |
vLLM推理引擎K8s部署
vLLM核心优化原理
vLLM通过PagedAttention和ContinuousBatching两大核心技术，将GPU显存利用率从传统方案的30-40%提升到90%以上。PagedAttention借鉴操作系统虚拟内存的分页机制，将KV Cache按块管理，消除显存碎片。ContinuousBatching在请求级别进行调度，新请求无需等待当前批次完成即可加入执行。
┌──────────────────────────────────────────────────────────────┐
│ vLLM PagedAttention + ContinuousBatching │
│ │
│ 传统方案 (Static Batching): │
│ ┌────┬────┬────┬────┐ │
│ │ R1 │ R2 │ R3 │ R4 │ 等最长请求完成才能开始下一批 │
│ └────┴────┴────┴────┘ │
│ ████████████████████ GPU空闲等待 │
│ │
│ vLLM (Continuous Batching): │
│ ┌────┬────┬────┬────┐ │
│ │ R1 │ R2 │ R3 │ R4 │ R1完成后立即插入R5 │
│ └────┴────┴────┴────┘ │
│ ┌────┬────┬────┬────┐ │
│ │ R5 │ R2 │ R3 │ R4 │ R2完成后立即插入R6 │
│ └────┴────┴────┴────┘ │
│ ████████████████ GPU持续高利用率 │
│ │
│ PagedAttention KV Cache: │
│ ┌───┬───┬───┬───┬───┬───┬───┬───┐ │
│ │P0 │P1 │P2 │P3 │P4 │P5 │P6 │P7 │ 按需分配物理块 │
│ └───┴───┴───┴───┴───┴───┴───┴───┘ │
│ R1→[P0,P1,P2] R2→[P3,P4] R3→[P5,P6,P7] │
│ 无碎片，显存利用率90%+ │
└──────────────────────────────────────────────────────────────┘
vLLM K8s Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen2-72b
namespace: llm-inference
labels:
app: vllm
model: qwen2-72b-instruct
spec:
replicas: 2
selector:
matchLabels:
app: vllm
model: qwen2-72b-instruct
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: vllm
model: qwen2-72b-instruct
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
spec:
terminationGracePeriodSeconds: 120
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: vllm
model: qwen2-72b-instruct
topologyKey: kubernetes.io/hostname
containers:
- name: vllm-server
image: vllm/vllm-openai:v0.6.0
command:
- python
- -m
- vllm.entrypoints.openai.api_server
args:
- --model
- /models/Qwen2.5-72B-Instruct
- --tensor-parallel-size
- "2"
- --max-model-len
- "32768"
- --gpu-memory-utilization
- "0.92"
- --max-num-seqs
- "256"
- --enable-prefix-caching
- --enable-chunked-prefill
- --port
- "8000"
ports:
- containerPort: 8000
resources:
requests:
nvidia.com/gpu: "2"
cpu: "8"
memory: 32Gi
limits:
nvidia.com/gpu: "2"
cpu: "16"
memory: 64Gi
env:
- name: MODEL_NAME
value: "qwen2-72b-instruct"
- name: VLLM_WORKER_MULTIPROC_METHOD
value: "ray"
volumeMounts:
- name: model-storage
mountPath: /models
- name: shm
mountPath: /dev/shm
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180
periodSeconds: 15
failureThreshold: 5
- name: metrics-exporter
image: prometheus-exporter/vllm-metrics:v1.0
ports:
- containerPort: 9090
resources:
requests:
cpu: "100m"
memory: 128Mi
volumes:
- name: model-storage
persistentVolumeClaim:
claimName: model-weights-pvc
- name: shm
emptyDir:
medium: Memory
sizeLimit: 16Gi
模型权重PVC（ReadWriteOncePod）
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-weights-pvc
namespace: llm-inference
spec:
accessModes:
- ReadWriteOncePod
storageClassName: local-ssd
resources:
requests:
storage: 200Gi
vLLM性能调优参数
| 参数 | 默认值 | 推荐值 | 说明 |
|---|---|---|---|
--gpu-memory-utilization |
0.9 | 0.92 | GPU显存利用率上限 |
--max-model-len |
模型默认 | 32768 | 最大序列长度，影响KV Cache大小 |
--max-num-seqs |
256 | 256 | 最大并发序列数 |
--enable-prefix-caching |
false | true | 前缀缓存，相同prompt复用KV Cache |
--enable-chunked-prefill |
false | true | 分块预填充，降低首Token延迟 |
--swap-space |
4GB | 8GB | CPU交换空间，长序列溢出时使用 |
KEDA自定义指标自动伸缩
为什么HPA不够
K8s原生HPA基于CPU/内存利用率伸缩，但LLM推理服务是GPU密集型，CPU利用率无法反映真实负载。一个推理Pod的CPU可能只有30%，但GPU显存已满、请求队列积压严重。KEDA（Kubernetes Event-Driven Autoscaling）支持自定义指标伸缩，可基于vLLM暴露的请求队列深度、GPU利用率、TTFT（Time To First Token）等指标精准伸缩。
KEDA部署
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-scaler
namespace: llm-inference
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: vllm-qwen2-72b
minReplicaCount: 2
maxReplicaCount: 20
cooldownPeriod: 300
pollingInterval: 15
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 120
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: Pods
value: 3
periodSeconds: 60
- type: Percent
value: 50
periodSeconds: 60
selectPolicy: Max
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_num_requests_waiting
threshold: "10"
query: |
avg_over_time(vllm_num_requests_waiting{model="qwen2-72b-instruct"}[1m])
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_gpu_cache_usage_perc
threshold: "80"
query: |
avg_over_time(vllm_gpu_cache_usage_perc{model="qwen2-72b-instruct"}[2m])
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_e2e_request_latency_seconds
threshold: "5"
query: |
histogram_quantile(0.95, sum(rate(vllm_e2e_request_latency_seconds_bucket{model="qwen2-72b-instruct"}[2m])) by (le))
多指标伸缩策略
┌──────────────────────────────────────────────────────────────┐
│ KEDA多指标伸缩决策流程 │
│ │
│ 指标1: 请求队列深度 │
│ vllm_num_requests_waiting > 10 → 扩容 │
│ │
│ 指标2: GPU KV Cache利用率 │
│ vllm_gpu_cache_usage_perc > 80% → 扩容 │
│ │
│ 指标3: P95端到端延迟 │
│ vllm_e2e_request_latency_seconds > 5s → 扩容 │
│ │
│ 决策: 任一指标触发即扩容，所有指标正常才缩容 │
│ 冷却: 扩容30s / 缩容600s (避免抖动) │
│ 步长: 扩容max(3 Pod, 50%) / 缩容1 Pod/2min │
└──────────────────────────────────────────────────────────────┘
GPU资源优化与多租户隔离
NVIDIA MIG分区
在A100/H100上使用MIG（Multi-Instance GPU）将单张GPU划分为多个实例，实现多租户推理隔离。
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
name: mig-instance
namespace: llm-inference
spec:
devices:
- name: gpu
driver: nvidia.com/gpu
requests:
- name: mig-instance
deviceClassName: mig.nvidia.com/1g.10gb
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-small-model
namespace: llm-inference
spec:
replicas: 4
selector:
matchLabels:
app: vllm-small
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.0
args:
- --model
- /models/Qwen2.5-7B-Instruct
- --gpu-memory-utilization
- "0.90"
- --max-model-len
- "8192"
resources:
claims:
- name: mig-instance
resourceClaims:
- name: mig-instance
claimName: mig-instance
GPU时间片共享
对于低优先级推理任务，使用GPU时间片共享，多个Pod共享同一张GPU。
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-batch-inference
namespace: llm-inference
spec:
replicas: 2
selector:
matchLabels:
app: vllm-batch
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.0
args:
- --model
- /models/Qwen2.5-7B-Instruct
- --gpu-memory-utilization
- "0.45"
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "0"
- name: CUDA_MPS_PIPE_DIRECTORY
value: "/tmp/nvidia-mps"
- name: CUDA_MPS_LOG_DIRECTORY
value: "/tmp/nvidia-log"
多租户GPU资源配额
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-quota-team-a
namespace: team-a-inference
spec:
hard:
requests.nvidia.com/gpu: "8"
limits.nvidia.com/gpu: "8"
requests.cpu: "32"
limits.cpu: "64"
requests.memory: 128Gi
limits.memory: 256Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-quota-team-b
namespace: team-b-inference
spec:
hard:
requests.nvidia.com/gpu: "4"
limits.nvidia.com/gpu: "4"
3层弹性策略与故障自愈
3层弹性架构
┌──────────────────────────────────────────────────────────────┐
│ 3层弹性策略架构 │
│ │
│ Layer 1: Pod级 (KEDA HPA) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 基于自定义指标的Pod副本数伸缩 │ │
│ │ 响应时间: 30-60s │ │
│ │ 粒度: 单个Deployment │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ 仍不足 │
│ Layer 2: Deployment级 (滚动更新) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 模型版本切换、参数调整的滚动更新 │ │
│ │ 响应时间: 5-15min │ │
│ │ 粒度: 单个模型服务 │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ 集群级 │
│ Layer 3: Cluster级 (Cluster Autoscaler) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 节点池扩缩容，新增GPU节点 │ │
│ │ 响应时间: 3-10min (含GPU驱动初始化) │ │
│ │ 粒度: 整个集群 │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
故障自愈：Pod异常检测与恢复
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen2-72b
namespace: llm-inference
spec:
replicas: 2
selector:
matchLabels:
app: vllm
template:
spec:
containers:
- name: vllm-server
image: vllm/vllm-openai:v0.6.0
startupProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
failureThreshold: 30
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180
periodSeconds: 15
failureThreshold: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: vllm-pdb
namespace: llm-inference
spec:
minAvailable: 1
selector:
matchLabels:
app: vllm
model: qwen2-72b-instruct
推理服务预热策略
import httpx
import asyncio
import logging
logger = logging.getLogger(__name__)
class InferenceWarmer:
def __init__(self, base_url: str, model: str, warmup_prompts: list[str]):
self.base_url = base_url
self.model = model
self.warmup_prompts = warmup_prompts
async def warmup(self, rounds: int = 3):
for round_num in range(rounds):
tasks = [self._send_request(prompt) for prompt in self.warmup_prompts]
results = await asyncio.gather(*tasks, return_exceptions=True)
success_count = sum(1 for r in results if not isinstance(r, Exception))
logger.info(f"Warmup round {round_num + 1}: {success_count}/{len(tasks)} succeeded")
async def _send_request(self, prompt: str) -> dict:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.post(
f"{self.base_url}/v1/completions",
json={
"model": self.model,
"prompt": prompt,
"max_tokens": 32,
"temperature": 0.0,
},
)
return response.json()
async def warmup_new_pod(pod_ip: str, model: str):
warmer = InferenceWarmer(
base_url=f"http://{pod_ip}:8000",
model=model,
warmup_prompts=[
"Hello, how are you?",
"What is the capital of France?",
"Explain quantum computing briefly.",
],
)
await warmer.warmup(rounds=3)
生产级推理集群可观测性
vLLM Prometheus指标
apiVersion: v1
kind: ConfigMap
metadata:
name: vllm-alerts
namespace: llm-inference
data:
alerts.yml: |
groups:
- name: vllm.rules
rules:
- alert: VLLMHighQueueDepth
expr: vllm_num_requests_waiting > 20
for: 2m
labels:
severity: warning
annotations:
summary: "vLLM request queue depth is high"
description: "Queue depth {{ $value }} exceeds threshold 20"
- alert: VLLMHighTTFT
expr: histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "vLLM TTFT P95 is high"
description: "TTFT P95 {{ $value }}s exceeds 2s threshold"
- alert: VLLMGPUOOM
expr: vllm_gpu_cache_usage_perc > 0.95
for: 1m
labels:
severity: critical
annotations:
summary: "vLLM GPU cache near OOM"
description: "GPU cache usage {{ $value }}% exceeds 95%"
- alert: VLLMHighErrorRate
expr: rate(vllm_request_errors_total[5m]) / rate(vllm_request_total[5m]) > 0.05
for: 3m
labels:
severity: critical
annotations:
summary: "vLLM error rate is high"
description: "Error rate {{ $value | humanizePercentage }} exceeds 5%"
推理服务Grafana Dashboard关键面板
| 面板 | 指标 | 说明 |
|---|---|---|
| 请求吞吐量 | rate(vllm_request_total[1m]) |
每秒完成请求数 |
| 队列深度 | vllm_num_requests_waiting |
等待中的请求数 |
| TTFT P95 | histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) |
首Token延迟 |
| TPOT P95 | histogram_quantile(0.95, rate(vllm_time_per_output_token_seconds_bucket[5m])) |
每Token延迟 |
| GPU Cache利用率 | vllm_gpu_cache_usage_perc |
KV Cache显存占用 |
| 并发序列数 | vllm_num_requests_running |
正在执行的序列数 |
| 错误率 | rate(vllm_request_errors_total[5m]) / rate(vllm_request_total[5m]) |
请求失败率 |
推理服务性能基准（Qwen2.5-72B, 2xA100）
| 场景 | 并发数 | TTFT P50 | TTFT P95 | TPOT | 吞吐量(tokens/s) |
|---|---|---|---|---|---|
| 短文本(128 in, 256 out) | 16 | 0.3s | 0.8s | 25ms | 4096 |
| 中等文本(512 in, 512 out) | 8 | 0.5s | 1.2s | 30ms | 2048 |
| 长文本(2048 in, 1024 out) | 4 | 1.2s | 2.5s | 35ms | 1024 |
| 超长文本(8192 in, 2048 out) | 2 | 3.5s | 6.0s | 40ms | 512 |
总结与引流
K8s 1.30+为大模型推理服务提供了更精细的资源控制能力。vLLM的PagedAttention+ContinuousBatching是LLM推理的事实标准，KEDA自定义指标伸缩比HPA CPU指标更精准匹配推理负载。3层弹性策略（Pod级HPA→Deployment级滚动更新→Cluster级集群伸缩）确保推理服务在不同负载场景下的稳定运行。
开发要点回顾：
- vLLM部署：启用prefix-caching和chunked-prefill，GPU显存利用率设为0.92，max-num-seqs根据GPU规格调整
- KEDA伸缩：基于请求队列深度、GPU Cache利用率、TTFT延迟三指标触发，扩容30s冷却/缩容600s冷却
- GPU优化：大模型用独占GPU，小模型用MIG分区，批量推理用时间片共享
- 故障自愈：startupProbe→readinessProbe→livenessProbe三级探测，PDB保证最小可用副本
- 可观测性：Prometheus+Grafana全链路监控，关键告警：队列深度>20、TTFT>2s、GPU Cache>95%
相关阅读：
- AI Agent多智能体编排实战：从单Agent到生产级多Agent协作系统 — Agent编排中的推理服务调度
- K8s GPU调度实战：MIG分区与GPU共享 — GPU资源管理与多租户隔离
- 大模型RAG系统从零到生产级全链路实战 — RAG系统中的推理服务集成
权威参考：
本站提供浏览器本地工具，免注册即可试用 →
#K8s大模型推理#LLM弹性调度#vLLM K8s部署#推理服务自动伸缩#GPU推理优化#2026