K8s AI GPU調度實戰：MIG分區、GPU共享與多租戶工作負載編排

云原生

摘要

  • GPU是AI工作負載的核心資源：K8s集群中GPU利用率平均僅30-40%，調度優化可提升至80%+
  • NVIDIA MIG分區：1張A100可切分為7個實例，支持不同AI任務並行，資源利用率提升3×
  • GPU共享3種模式：時間分片(TS)、MPS、MIG，分別適合推理、訓練、混合場景
  • 多租戶調度策略：GPU配額管理、優先級搶佔、彈性伸縮，確保SLA達標
  • 本文提供K8s GPU調度全棧方案，含Device Plugin配置與調度器擴展實戰

目錄

GPU調度：AI集群的核心挑戰

GPU資源浪費現狀

問題 原因 浪費比例
推理低利用率 單推理請求僅用10-20% GPU 40-60%
訓練碎片化 小模型訓練佔用整卡 20-30%
空閒等待 任務排隊等GPU釋放 15-25%
配置不當 資源請求與實際不匹配 10-15%

GPU調度演進路線

階段 時間 方案 特點
獨佔模式 2020前 1Pod=1GPU 簡單但浪費
時間分片 2021 GPU時間片共享 適合推理
MIG分區 2022 A100硬件級分區 隔離性好
MPS共享 2023 多進程共享GPU 適合訓練
彈性調度 2024-2026 動態MIG+優先級 智能化

2026年主流GPU規格

GPU 顯存 MIG實例 適合場景 價格($/h)
A100 80GB 80GB 7×10GB或2×40GB 通用訓練推理 3.5
H100 80GB 80GB 7×10GB或2×40GB 大模型訓練 4.5
H200 141GB 141GB 7×20GB或2×70GB 超大模型 6.0
L40S 48GB 48GB 不支持MIG 推理/微調 1.5
RTX 4090 24GB 不支持MIG 開發測試 0.8

NVIDIA MIG分區實戰

MIG架構解析

┌──────────────────────────────────────────────────────────────┐
│              A100 80GB MIG分區方案                              │
│                                                                │
│  方案1: 7× MIG 1g.10gb (最大並行度)                           │
│  ┌──────┐┌──────┐┌──────┐┌──────┐┌──────┐┌──────┐┌──────┐   │
│  │ GI 0 ││ GI 1 ││ GI 2 ││ GI 3 ││ GI 4 ││ GI 5 ││ GI 6 │   │
│  │10GB  ││10GB  ││10GB  ││10GB  ││10GB  ││10GB  ││10GB  │   │
│  │14SM  ││14SM  ││14SM  ││14SM  ││14SM  ││14SM  ││14SM  │   │
│  └──────┘└──────┘└──────┘└──────┘└──────┘└──────┘└──────┘   │
│  適合: 7個輕量推理服務並行                                      │
│                                                                │
│  方案2: 2× MIG 3g.40gb (大模型推理)                           │
│  ┌─────────────────────────────┐┌─────────────────────────────┐│
│  │          GI 0               ││          GI 1               ││
│  │          40GB               ││          40GB               ││
│  │          42SM               ││          42SM               ││
│  └─────────────────────────────┘└─────────────────────────────┘│
│  適合: 2個70B模型推理(量化後)                                    │
│                                                                │
│  方案3: 1× MIG 4g.40gb + 2× MIG 1g.10gb (混合)               │
│  ┌─────────────────────────────┐┌──────┐┌──────┐              │
│  │          GI 0               ││ GI 1 ││ GI 2 │              │
│  │          40GB               ││10GB  ││10GB  │              │
│  │          56SM               ││14SM  ││14SM  │              │
│  └─────────────────────────────┘└──────┘└──────┘              │
│  適合: 1個大模型推理 + 2個輕量推理                               │
└──────────────────────────────────────────────────────────────┘

MIG配置實戰

# nvidia-mig-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.10gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      all-2g.20gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "2g.20gb": 3
      all-3g.40gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "3g.40gb": 2
      mixed:
        - devices: [0]
          mig-enabled: true
          mig-devices:
            "3g.40gb": 2
        - devices: [1]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
        - devices: [2, 3]
          mig-enabled: false
---
apiVersion: nvidia.com/v1alpha1
kind: MigManager
metadata:
  name: mig-manager
spec:
  config: nvidia-mig-config
  gpuClientsConfig:
    version: v1
    gpuClients:
      - namespace: "ai-inference"
        podSelector:
          matchLabels:
            workload: "llm-inference"
        migDevice: "3g.40gb"
      - namespace: "ai-inference"
        podSelector:
          matchLabels:
            workload: "light-inference"
        migDevice: "1g.10gb"

MIG Pod調度

# 大模型推理 - 使用MIG 3g.40gb
apiVersion: v1
kind: Pod
metadata:
  name: llm-inference-70b
  namespace: ai-inference
  labels:
    workload: llm-inference
spec:
  containers:
    - name: inference
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/mig-3g.40gb: 1
      env:
        - name: MODEL_NAME
          value: "Qwen/Qwen2.5-72B-Instruct-AWQ"
        - name: GPU_MEMORY_UTILIZATION
          value: "0.95"
        - name: MAX_MODEL_LEN
          value: "8192"
---
# 輕量推理 - 使用MIG 1g.10gb
apiVersion: v1
kind: Pod
metadata:
  name: embedding-service
  namespace: ai-inference
  labels:
    workload: light-inference
spec:
  containers:
    - name: embedding
      image: huggingface/tei:latest
      resources:
        limits:
          nvidia.com/mig-1g.10gb: 1
      env:
        - name: MODEL_NAME
          value: "BAAI/bge-large-zh-v1.5"
        - name: MAX_BATCH_SIZE
          value: "256"

GPU共享3種模式

模式對比

維度 時間分片(TS) MPS MIG
隔離級別 軟件 硬件(部分) 硬件(完全)
顯存隔離 否(共享) 否(共享) 是(獨立)
性能隔離 差 中 好
並行度 高 中 中
故障隔離 差 差 好
適用場景 推理 訓練 混合
GPU要求 通用 Volta+ A100+

時間分片配置

# gpu-time-slicing.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-time-slicing-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4
            devices: all
---
# 使用時間分片的Pod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-pool
  namespace: ai-inference
spec:
  replicas: 8
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      containers:
        - name: inference
          image: vllm/vllm-openai:latest
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: MODEL_NAME
              value: "Qwen/Qwen2.5-7B-Instruct"

MPS配置

# gpu-mps-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-mps-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    sharing:
      mps:
        resources:
          - name: nvidia.com/gpu
            replicas: 2
            devices: all
---
apiVersion: v1
kind: Pod
metadata:
  name: training-job-mps
  namespace: ai-training
spec:
  containers:
    - name: training
      image: pytorch/pytorch:2.4-cuda12.4
      resources:
        limits:
          nvidia.com/gpu: 1
      command:
        - python
        - -m
        - torch.distributed.launch
        - --nproc_per_node=2
        - train.py

共享模式選型決策

是否有A100/H100?
├── 否 → 時間分片(推理) / MPS(訓練)
└── 是 → 需要完全隔離?
    ├── 是 → MIG分區
    └── 否 → 需要訓練?
        ├── 是 → MPS
        └── 否 → 時間分片

K8s GPU Device Plugin配置

NVIDIA Device Plugin部署

# nvidia-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      priorityClassName: system-node-critical
      containers:
        - name: nvidia-device-plugin
          image: nvcr.io/nvidia/k8s-device-plugin:v0.16.0
          args:
            - --config=default
            - --mig-strategy=mixed
            - --pass-device-specs=true
            - --device-list-strategy=configmap
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

GPU資源監控

# gpu-monitor.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-monitor-config
  namespace: monitoring
data:
  gpu-metrics.json: |
    {
      "metrics": [
        "gpu_utilization",
        "gpu_memory_utilization",
        "gpu_memory_used_bytes",
        "gpu_memory_total_bytes",
        "gpu_power_usage_watts",
        "gpu_temperature_celsius",
        "gpu_sm_clock_mhz",
        "gpu_mem_clock_mhz"
      ],
      "scrape_interval": "15s",
      "labels": {
        "cluster": "production",
        "region": "cn-east"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    metadata:
      labels:
        app: dcgm-exporter
    spec:
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.7
          ports:
            - containerPort: 9400
              name: metrics
          env:
            - name: DCGM_EXPORTER_COLLECTORS
              value: "/etc/dcgm-exporter/dcp-metrics-inventory.csv"
          resources:
            limits:
              nvidia.com/gpu: 1

多租戶GPU調度策略

GPU配額管理

# gpu-resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-team-a
  namespace: team-a
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    limits.nvidia.com/gpu: "8"
    requests.nvidia.com/mig-3g.40gb: "4"
    limits.nvidia.com/mig-3g.40gb: "4"
    requests.nvidia.com/mig-1g.10gb: "14"
    limits.nvidia.com/mig-1g.10gb: "14"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-team-b
  namespace: team-b
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"
    requests.nvidia.com/mig-3g.40gb: "2"
    limits.nvidia.com/mig-3g.40gb: "2"

優先級搶佔調度

# gpu-priority-class.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-critical
value: 1000000
globalDefault: false
description: "Critical GPU workloads - can preempt others"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-high
value: 900000
globalDefault: false
description: "High priority GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-low
value: 100000
globalDefault: false
description: "Low priority GPU workloads - preemptible"
---
# 在線推理 - 高優先級
apiVersion: apps/v1
kind: Deployment
metadata:
  name: online-inference
spec:
  template:
    spec:
      priorityClassName: gpu-critical
      containers:
        - name: inference
          resources:
            limits:
              nvidia.com/mig-3g.40gb: 1
---
# 離線訓練 - 低優先級
apiVersion: batch/v1
kind: Job
metadata:
  name: offline-training
spec:
  template:
    spec:
      priorityClassName: gpu-low
      containers:
        - name: training
          resources:
            limits:
              nvidia.com/gpu: 4

GPU彈性伸縮

# gpu-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-hpa
  namespace: ai-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-pool
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_utilization
        target:
          type: AverageValue
          averageValue: "70"
    - type: Pods
      pods:
        metric:
          name: request_latency_ms
        target:
          type: AverageValue
          averageValue: "200"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 300

GPU集群生產管理

GPU節點標籤與調度

# gpu-node-labels.yaml
# 標記GPU類型
kubectl label nodes gpu-node-1 nvidia.com/gpu.product=A100-SXM4-80GB
kubectl label nodes gpu-node-2 nvidia.com/gpu.product=H100-SXM5-80GB
kubectl label nodes gpu-node-3 nvidia.com/gpu.product=L40S-48GB

# 標記MIG配置
kubectl label nodes gpu-node-1 nvidia.com/mig.config=mixed
kubectl label nodes gpu-node-2 nvidia.com/mig.config=all-3g.40gb

# 調度到特定GPU
apiVersion: v1
kind: Pod
metadata:
  name: h100-training
spec:
  nodeSelector:
    nvidia.com/gpu.product: H100-SXM5-80GB
  containers:
    - name: training
      resources:
        limits:
          nvidia.com/gpu: 8

GPU故障自癒

# gpu-health-check.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-health-monitor
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: gpu-health-monitor
  template:
    metadata:
      labels:
        app: gpu-health-monitor
    spec:
      serviceAccountName: gpu-health-sa
      containers:
        - name: monitor
          image: custom/gpu-health-monitor:latest
          env:
            - name: CHECK_INTERVAL
              value: "60"
            - name: MEMORY_ERROR_THRESHOLD
              value: "10"
            - name: TEMPERATURE_THRESHOLD
              value: "90"
            - name: AUTO_CORDON
              value: "true"
          volumeMounts:
            - name: nvidia
              mountPath: /usr/local/nvidia
      volumes:
        - name: nvidia
          hostPath:
            path: /usr/local/nvidia

GPU利用率優化效果

優化措施 利用率提升 成本節省
MIG分區 +35% 30%
時間分片 +25% 20%
優先級調度 +15% 15%
彈性伸縮 +10% 10%
綜合優化 +50% 50%

總結與引流

關鍵要點回顧

  1. MIG分區：A100/H100的殺手鐧，1卡變7卡，推理場景首選
  2. GPU共享：時間分片適合推理、MPS適合訓練、MIG適合混合
  3. 多租戶調度：配額+優先級+彈性伸縮三件套確保SLA
  4. 綜合優化：MIG+調度+伸縮組合可實現50%成本節省

GPU調度方案推薦

集群規模 推薦方案 預期利用率
<8卡 時間分片 60%
8-32卡 MIG+時間分片 75%
32-128卡 MIG+優先級調度 80%
>128卡 全套方案 85%+

需要快速生成cURL命令測試GPU推理API？試試我們的cURL轉代碼工具和JSON格式化。

延伸閱讀

本站提供瀏覽器本地工具，免註冊即可試用 →

#K8s GPU调度#AI推理调度#GPU共享#MIG分区#K8s AI工作负载#2026