K8s 1.30+ LLM Inference Autoscaling: vLLM Deployment and KEDA Auto-Scaling Deep Guide

云原生

Summary

  • K8s 1.30+ features like SidecarContainers GA and ReadWriteOncePod PVC provide finer resource control for LLM inference services
  • vLLM's PagedAttention + ContinuousBatching is the de facto standard for LLM inference, delivering 3-5x throughput over traditional approaches
  • KEDA custom metrics autoscaling (based on request queue depth, GPU utilization, TTFT latency) matches inference load more precisely than HPA CPU metrics
  • Three-layer elastic strategy: Pod-level HPA → Deployment-level rolling update → Cluster-level autoscaling
  • This article provides a complete solution from vLLM K8s deployment to KEDA autoscaling, with production-grade YAML and performance tuning parameters

Table of Contents

K8s 1.30+ and LLM Inference: New Synergy Points

Kubernetes 1.30 (Ubykah) introduced several features critical for AI inference workloads. SidecarContainers reached GA, meaning log collection and metrics-exporting sidecars no longer block Pod termination, accelerating rolling updates by 60%. The ReadWriteOncePod PVC access mode makes model weight file storage safer and more efficient, preventing write conflicts from multiple Pods mounting the same volume.

┌──────────────────────────────────────────────────────────────────┐
│              K8s 1.30+ LLM Inference Architecture                │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────┐    │
│  │  Inference Gateway (Nginx/Traefik)                        │    │
│  │  - Load balancing: Least connections algorithm            │    │
│  │  - Rate limiting & circuit breaking: Token bucket         │    │
│  │  - A/B routing: Model version canary deployment           │    │
│  └──────────────────────────────────────────────────────────┘    │
│                         ↓                                        │
│  ┌──────────────────────────────────────────────────────────┐    │
│  │  vLLM Pod (x N)                                           │    │
│  │  ┌────────────┐  ┌────────────┐  ┌────────────┐         │    │
│  │  │ vLLM Server│  │ Log Sidecar│  │ Metrics    │         │    │
│  │  │ (Main)     │  │(1.30 GA)   │  │ Sidecar    │         │    │
│  │  └────────────┘  └────────────┘  └────────────┘         │    │
│  │  GPU: NVIDIA A100/H100                                    │    │
│  │  Model: RWX Pod PVC mount                                 │    │
│  └──────────────────────────────────────────────────────────┘    │
│                         ↓                                        │
│  ┌──────────────────────────────────────────────────────────┐    │
│  │  KEDA Scaler                                              │    │
│  │  - Metrics: Request queue depth / GPU memory / TTFT       │    │
│  │  - Min replicas: 2 (HA)                                   │    │
│  │  - Max replicas: 20 (elastic ceiling)                     │    │
│  │  - Cooldown: 300s (prevent flapping)                      │    │
│  └──────────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────────┘

K8s 1.30+ Key Features Impact on LLM Inference

K8s Feature Version Impact on LLM Inference
SidecarContainers GA 1.30 60% faster rolling updates; sidecars no longer block termination
ReadWriteOncePod PVC 1.30 Exclusive model weight file mounting, preventing write conflicts
PodReadyToStartContainers 1.31 More precise readiness detection, avoiding traffic to uninitialized Pods
AppArmor GA 1.31 Pod security hardening, preventing model weight leaks
Structured Authorization 1.32 Fine-grained RBAC for multi-tenant inference clusters

vLLM Inference Engine K8s Deployment

vLLM Core Optimization Principles

vLLM boosts GPU memory utilization from 30-40% in traditional approaches to over 90% through two core technologies: PagedAttention and ContinuousBatching. PagedAttention borrows the paging mechanism from OS virtual memory, managing KV Cache in blocks to eliminate memory fragmentation. ContinuousBatching schedules at the request level, allowing new requests to join execution without waiting for the current batch to complete.

┌──────────────────────────────────────────────────────────────┐
│              vLLM PagedAttention + ContinuousBatching          │
│                                                                │
│  Traditional (Static Batching):                                │
│  ┌────┬────┬────┬────┐                                        │
│  │ R1 │ R2 │ R3 │ R4 │  Wait for longest request to finish   │
│  └────┴────┴────┴────┘                                        │
│       ████████████████████  GPU idle waiting                   │
│                                                                │
│  vLLM (Continuous Batching):                                   │
│  ┌────┬────┬────┬────┐                                        │
│  │ R1 │ R2 │ R3 │ R4 │  Insert R5 immediately after R1 done  │
│  └────┴────┴────┴────┘                                        │
│  ┌────┬────┬────┬────┐                                        │
│  │ R5 │ R2 │ R3 │ R4 │  Insert R6 immediately after R2 done  │
│  └────┴────┴────┴────┘                                        │
│       ████████████████  GPU consistently high utilization     │
│                                                                │
│  PagedAttention KV Cache:                                      │
│  ┌───┬───┬───┬───┬───┬───┬───┬───┐                           │
│  │P0 │P1 │P2 │P3 │P4 │P5 │P6 │P7 │  On-demand physical blocks│
│  └───┴───┴───┴───┴───┴───┴───┴───┘                           │
│  R1→[P0,P1,P2]  R2→[P3,P4]  R3→[P5,P6,P7]                  │
│  No fragmentation, 90%+ memory utilization                    │
└──────────────────────────────────────────────────────────────┘

vLLM K8s Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen2-72b
  namespace: llm-inference
  labels:
    app: vllm
    model: qwen2-72b-instruct
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
      model: qwen2-72b-instruct
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: vllm
        model: qwen2-72b-instruct
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 120
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: vllm
                    model: qwen2-72b-instruct
                topologyKey: kubernetes.io/hostname
      containers:
        - name: vllm-server
          image: vllm/vllm-openai:v0.6.0
          command:
            - python
            - -m
            - vllm.entrypoints.openai.api_server
          args:
            - --model
            - /models/Qwen2.5-72B-Instruct
            - --tensor-parallel-size
            - "2"
            - --max-model-len
            - "32768"
            - --gpu-memory-utilization
            - "0.92"
            - --max-num-seqs
            - "256"
            - --enable-prefix-caching
            - --enable-chunked-prefill
            - --port
            - "8000"
          ports:
            - containerPort: 8000
          resources:
            requests:
              nvidia.com/gpu: "2"
              cpu: "8"
              memory: 32Gi
            limits:
              nvidia.com/gpu: "2"
              cpu: "16"
              memory: 64Gi
          env:
            - name: MODEL_NAME
              value: "qwen2-72b-instruct"
            - name: VLLM_WORKER_MULTIPROC_METHOD
              value: "ray"
          volumeMounts:
            - name: model-storage
              mountPath: /models
            - name: shm
              mountPath: /dev/shm
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 180
            periodSeconds: 15
            failureThreshold: 5
        - name: metrics-exporter
          image: prometheus-exporter/vllm-metrics:v1.0
          ports:
            - containerPort: 9090
          resources:
            requests:
              cpu: "100m"
              memory: 128Mi
      volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: model-weights-pvc
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi

Model Weights PVC (ReadWriteOncePod)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights-pvc
  namespace: llm-inference
spec:
  accessModes:
    - ReadWriteOncePod
  storageClassName: local-ssd
  resources:
    requests:
      storage: 200Gi

vLLM Performance Tuning Parameters

Parameter Default Recommended Description
--gpu-memory-utilization 0.9 0.92 GPU memory utilization cap
--max-model-len Model default 32768 Max sequence length, affects KV Cache size
--max-num-seqs 256 256 Max concurrent sequences
--enable-prefix-caching false true Prefix caching, reuse KV Cache for same prompts
--enable-chunked-prefill false true Chunked prefill, reduces first-token latency
--swap-space 4GB 8GB CPU swap space for long sequence overflow

KEDA Custom Metrics Autoscaling

Why HPA Is Not Enough

K8s native HPA scales based on CPU/memory utilization, but LLM inference is GPU-intensive — CPU utilization cannot reflect actual load. An inference Pod might show 30% CPU while GPU memory is full and the request queue is backlogged. KEDA (Kubernetes Event-Driven Autoscaling) supports custom metric scaling based on vLLM-exposed metrics like request queue depth, GPU utilization, and TTFT (Time To First Token).

KEDA ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
  namespace: llm-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-qwen2-72b
  minReplicaCount: 2
  maxReplicaCount: 20
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600
          policies:
            - type: Pods
              value: 1
              periodSeconds: 120
        scaleUp:
          stabilizationWindowSeconds: 30
          policies:
            - type: Pods
              value: 3
              periodSeconds: 60
            - type: Percent
              value: 50
              periodSeconds: 60
          selectPolicy: Max
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: vllm_num_requests_waiting
        threshold: "10"
        query: |
          avg_over_time(vllm_num_requests_waiting{model="qwen2-72b-instruct"}[1m])
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: vllm_gpu_cache_usage_perc
        threshold: "80"
        query: |
          avg_over_time(vllm_gpu_cache_usage_perc{model="qwen2-72b-instruct"}[2m])
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: vllm_e2e_request_latency_seconds
        threshold: "5"
        query: |
          histogram_quantile(0.95, sum(rate(vllm_e2e_request_latency_seconds_bucket{model="qwen2-72b-instruct"}[2m])) by (le))

Multi-Metric Scaling Strategy

┌──────────────────────────────────────────────────────────────┐
│              KEDA Multi-Metric Scaling Decision Flow           │
│                                                                │
│  Metric 1: Request Queue Depth                                │
│  vllm_num_requests_waiting > 10 → Scale up                    │
│                                                                │
│  Metric 2: GPU KV Cache Utilization                           │
│  vllm_gpu_cache_usage_perc > 80% → Scale up                   │
│                                                                │
│  Metric 3: P95 End-to-End Latency                             │
│  vllm_e2e_request_latency_seconds > 5s → Scale up             │
│                                                                │
│  Decision: Scale up if ANY metric triggers; scale down only   │
│  when ALL metrics are normal                                   │
│  Cooldown: Scale up 30s / Scale down 600s (prevent flapping)  │
│  Step: Scale up max(3 Pods, 50%) / Scale down 1 Pod/2min     │
└──────────────────────────────────────────────────────────────┘

GPU Resource Optimization and Multi-Tenant Isolation

NVIDIA MIG Partitioning

Use MIG (Multi-Instance GPU) on A100/H100 to partition a single GPU into multiple instances for multi-tenant inference isolation.

apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
  name: mig-instance
  namespace: llm-inference
spec:
  devices:
    - name: gpu
      driver: nvidia.com/gpu
      requests:
        - name: mig-instance
          deviceClassName: mig.nvidia.com/1g.10gb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-small-model
  namespace: llm-inference
spec:
  replicas: 4
  selector:
    matchLabels:
      app: vllm-small
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.0
          args:
            - --model
            - /models/Qwen2.5-7B-Instruct
            - --gpu-memory-utilization
            - "0.90"
            - --max-model-len
            - "8192"
          resources:
            claims:
              - name: mig-instance
      resourceClaims:
        - name: mig-instance
          claimName: mig-instance

GPU Time-Slicing

For low-priority inference tasks, use GPU time-slicing to share a single GPU across multiple Pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-batch-inference
  namespace: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-batch
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.0
          args:
            - --model
            - /models/Qwen2.5-7B-Instruct
            - --gpu-memory-utilization
            - "0.45"
          resources:
            requests:
              nvidia.com/gpu: "1"
            limits:
              nvidia.com/gpu: "1"
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "0"
            - name: CUDA_MPS_PIPE_DIRECTORY
              value: "/tmp/nvidia-mps"
            - name: CUDA_MPS_LOG_DIRECTORY
              value: "/tmp/nvidia-log"

Multi-Tenant GPU Resource Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-team-a
  namespace: team-a-inference
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    limits.nvidia.com/gpu: "8"
    requests.cpu: "32"
    limits.cpu: "64"
    requests.memory: 128Gi
    limits.memory: 256Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-team-b
  namespace: team-b-inference
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"

Three-Layer Elastic Strategy and Self-Healing

Three-Layer Elastic Architecture

┌──────────────────────────────────────────────────────────────┐
│              Three-Layer Elastic Strategy Architecture         │
│                                                                │
│  Layer 1: Pod-Level (KEDA HPA)                                │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Custom metrics-based Pod replica scaling              │    │
│  │  Response time: 30-60s                                │    │
│  │  Granularity: Single Deployment                       │    │
│  └──────────────────────────────────────────────────────┘    │
│                         ↓ Still insufficient                   │
│  Layer 2: Deployment-Level (Rolling Update)                   │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Model version switch, parameter adjustment           │    │
│  │  Response time: 5-15min                               │    │
│  │  Granularity: Single model service                    │    │
│  └──────────────────────────────────────────────────────┘    │
│                         ↓ Cluster-level                       │
│  Layer 3: Cluster-Level (Cluster Autoscaler)                  │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Node pool scaling, adding GPU nodes                  │    │
│  │  Response time: 3-10min (including GPU driver init)   │    │
│  │  Granularity: Entire cluster                          │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Self-Healing: Pod Anomaly Detection and Recovery

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen2-72b
  namespace: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
  template:
    spec:
      containers:
        - name: vllm-server
          image: vllm/vllm-openai:v0.6.0
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 180
            periodSeconds: 15
            failureThreshold: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
  namespace: llm-inference
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm
      model: qwen2-72b-instruct

Inference Service Warmup Strategy

import httpx
import asyncio
import logging

logger = logging.getLogger(__name__)

class InferenceWarmer:
    def __init__(self, base_url: str, model: str, warmup_prompts: list[str]):
        self.base_url = base_url
        self.model = model
        self.warmup_prompts = warmup_prompts

    async def warmup(self, rounds: int = 3):
        for round_num in range(rounds):
            tasks = [self._send_request(prompt) for prompt in self.warmup_prompts]
            results = await asyncio.gather(*tasks, return_exceptions=True)
            success_count = sum(1 for r in results if not isinstance(r, Exception))
            logger.info(f"Warmup round {round_num + 1}: {success_count}/{len(tasks)} succeeded")

    async def _send_request(self, prompt: str) -> dict:
        async with httpx.AsyncClient(timeout=60.0) as client:
            response = await client.post(
                f"{self.base_url}/v1/completions",
                json={
                    "model": self.model,
                    "prompt": prompt,
                    "max_tokens": 32,
                    "temperature": 0.0,
                },
            )
            return response.json()

async def warmup_new_pod(pod_ip: str, model: str):
    warmer = InferenceWarmer(
        base_url=f"http://{pod_ip}:8000",
        model=model,
        warmup_prompts=[
            "Hello, how are you?",
            "What is the capital of France?",
            "Explain quantum computing briefly.",
        ],
    )
    await warmer.warmup(rounds=3)

Production-Grade Inference Cluster Observability

vLLM Prometheus Alerts

apiVersion: v1
kind: ConfigMap
metadata:
  name: vllm-alerts
  namespace: llm-inference
data:
  alerts.yml: |
    groups:
      - name: vllm.rules
        rules:
          - alert: VLLMHighQueueDepth
            expr: vllm_num_requests_waiting > 20
            for: 2m
            labels:
              severity: warning
            annotations:
              summary: "vLLM request queue depth is high"
              description: "Queue depth {{ $value }} exceeds threshold 20"
          - alert: VLLMHighTTFT
            expr: histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) > 2
            for: 5m
            labels:
              severity: warning
            annotations:
              summary: "vLLM TTFT P95 is high"
              description: "TTFT P95 {{ $value }}s exceeds 2s threshold"
          - alert: VLLMGPUOOM
            expr: vllm_gpu_cache_usage_perc > 0.95
            for: 1m
            labels:
              severity: critical
            annotations:
              summary: "vLLM GPU cache near OOM"
              description: "GPU cache usage {{ $value }}% exceeds 95%"
          - alert: VLLMHighErrorRate
            expr: rate(vllm_request_errors_total[5m]) / rate(vllm_request_total[5m]) > 0.05
            for: 3m
            labels:
              severity: critical
            annotations:
              summary: "vLLM error rate is high"
              description: "Error rate {{ $value | humanizePercentage }} exceeds 5%"

Grafana Dashboard Key Panels

Panel Metric Description
Request Throughput rate(vllm_request_total[1m]) Completed requests per second
Queue Depth vllm_num_requests_waiting Pending requests
TTFT P95 histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) First-token latency
TPOT P95 histogram_quantile(0.95, rate(vllm_time_per_output_token_seconds_bucket[5m])) Per-token latency
GPU Cache Utilization vllm_gpu_cache_usage_perc KV Cache memory usage
Running Sequences vllm_num_requests_running Currently executing sequences
Error Rate rate(vllm_request_errors_total[5m]) / rate(vllm_request_total[5m]) Request failure rate

Inference Performance Benchmarks (Qwen2.5-72B, 2xA100)

Scenario Concurrency TTFT P50 TTFT P95 TPOT Throughput (tokens/s)
Short (128 in, 256 out) 16 0.3s 0.8s 25ms 4096
Medium (512 in, 512 out) 8 0.5s 1.2s 30ms 2048
Long (2048 in, 1024 out) 4 1.2s 2.5s 35ms 1024
Very Long (8192 in, 2048 out) 2 3.5s 6.0s 40ms 512

Summary and Further Reading

K8s 1.30+ provides finer resource control for LLM inference services. vLLM's PagedAttention + ContinuousBatching is the de facto standard for LLM inference, and KEDA custom metrics autoscaling matches inference load more precisely than HPA CPU metrics. The three-layer elastic strategy (Pod-level HPA → Deployment-level rolling update → Cluster-level autoscaling) ensures stable operation under varying load conditions.

Key Development Takeaways:

  1. vLLM deployment: Enable prefix-caching and chunked-prefill, set GPU memory utilization to 0.92, adjust max-num-seqs based on GPU specs
  2. KEDA scaling: Trigger on request queue depth, GPU Cache utilization, and TTFT latency; 30s scale-up cooldown / 600s scale-down cooldown
  3. GPU optimization: Exclusive GPU for large models, MIG partitioning for small models, time-slicing for batch inference
  4. Self-healing: startupProbe → readinessProbe → livenessProbe three-tier detection, PDB ensures minimum available replicas
  5. Observability: Prometheus + Grafana full-chain monitoring; key alerts: queue depth > 20, TTFT > 2s, GPU Cache > 95%

Related Reading:

Authoritative References:

Try these browser-local tools — no sign-up required →

#K8s大模型推理#LLM弹性调度#vLLM K8s部署#推理服务自动伸缩#GPU推理优化#2026