K8s AI GPU调度实战：MIG分区、GPU共享与多租户工作负载编排

云原生

摘要

  • GPU是AI工作负载的核心资源：K8s集群中GPU利用率平均仅30-40%，调度优化可提升至80%+
  • NVIDIA MIG分区：1张A100可切分为7个实例，支持不同AI任务并行，资源利用率提升3×
  • GPU共享3种模式：时间分片(TS)、MPS、MIG，分别适合推理、训练、混合场景
  • 多租户调度策略：GPU配额管理、优先级抢占、弹性伸缩，确保SLA达标
  • 本文提供K8s GPU调度全栈方案，含Device Plugin配置与调度器扩展实战

目录

GPU调度：AI集群的核心挑战

GPU资源浪费现状

问题 原因 浪费比例
推理低利用率 单推理请求仅用10-20% GPU 40-60%
训练碎片化 小模型训练占用整卡 20-30%
空闲等待 任务排队等GPU释放 15-25%
配置不当 资源请求与实际不匹配 10-15%

GPU调度演进路线

阶段 时间 方案 特点
独占模式 2020前 1Pod=1GPU 简单但浪费
时间分片 2021 GPU时间片共享 适合推理
MIG分区 2022 A100硬件级分区 隔离性好
MPS共享 2023 多进程共享GPU 适合训练
弹性调度 2024-2026 动态MIG+优先级 智能化

2026年主流GPU规格

GPU 显存 MIG实例 适合场景 价格($/h)
A100 80GB 80GB 7×10GB或2×40GB 通用训练推理 3.5
H100 80GB 80GB 7×10GB或2×40GB 大模型训练 4.5
H200 141GB 141GB 7×20GB或2×70GB 超大模型 6.0
L40S 48GB 48GB 不支持MIG 推理/微调 1.5
RTX 4090 24GB 不支持MIG 开发测试 0.8

NVIDIA MIG分区实战

MIG架构解析

┌──────────────────────────────────────────────────────────────┐
│              A100 80GB MIG分区方案                              │
│                                                                │
│  方案1: 7× MIG 1g.10gb (最大并行度)                           │
│  ┌──────┐┌──────┐┌──────┐┌──────┐┌──────┐┌──────┐┌──────┐   │
│  │ GI 0 ││ GI 1 ││ GI 2 ││ GI 3 ││ GI 4 ││ GI 5 ││ GI 6 │   │
│  │10GB  ││10GB  ││10GB  ││10GB  ││10GB  ││10GB  ││10GB  │   │
│  │14SM  ││14SM  ││14SM  ││14SM  ││14SM  ││14SM  ││14SM  │   │
│  └──────┘└──────┘└──────┘└──────┘└──────┘└──────┘└──────┘   │
│  适合: 7个轻量推理服务并行                                      │
│                                                                │
│  方案2: 2× MIG 3g.40gb (大模型推理)                           │
│  ┌─────────────────────────────┐┌─────────────────────────────┐│
│  │          GI 0               ││          GI 1               ││
│  │          40GB               ││          40GB               ││
│  │          42SM               ││          42SM               ││
│  └─────────────────────────────┘└─────────────────────────────┘│
│  适合: 2个70B模型推理(量化后)                                    │
│                                                                │
│  方案3: 1× MIG 4g.40gb + 2× MIG 1g.10gb (混合)               │
│  ┌─────────────────────────────┐┌──────┐┌──────┐              │
│  │          GI 0               ││ GI 1 ││ GI 2 │              │
│  │          40GB               ││10GB  ││10GB  │              │
│  │          56SM               ││14SM  ││14SM  │              │
│  └─────────────────────────────┘└──────┘└──────┘              │
│  适合: 1个大模型推理 + 2个轻量推理                               │
└──────────────────────────────────────────────────────────────┘

MIG配置实战

# nvidia-mig-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.10gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      all-2g.20gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "2g.20gb": 3
      all-3g.40gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "3g.40gb": 2
      mixed:
        - devices: [0]
          mig-enabled: true
          mig-devices:
            "3g.40gb": 2
        - devices: [1]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
        - devices: [2, 3]
          mig-enabled: false
---
apiVersion: nvidia.com/v1alpha1
kind: MigManager
metadata:
  name: mig-manager
spec:
  config: nvidia-mig-config
  gpuClientsConfig:
    version: v1
    gpuClients:
      - namespace: "ai-inference"
        podSelector:
          matchLabels:
            workload: "llm-inference"
        migDevice: "3g.40gb"
      - namespace: "ai-inference"
        podSelector:
          matchLabels:
            workload: "light-inference"
        migDevice: "1g.10gb"

MIG Pod调度

# 大模型推理 - 使用MIG 3g.40gb
apiVersion: v1
kind: Pod
metadata:
  name: llm-inference-70b
  namespace: ai-inference
  labels:
    workload: llm-inference
spec:
  containers:
    - name: inference
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/mig-3g.40gb: 1
      env:
        - name: MODEL_NAME
          value: "Qwen/Qwen2.5-72B-Instruct-AWQ"
        - name: GPU_MEMORY_UTILIZATION
          value: "0.95"
        - name: MAX_MODEL_LEN
          value: "8192"
---
# 轻量推理 - 使用MIG 1g.10gb
apiVersion: v1
kind: Pod
metadata:
  name: embedding-service
  namespace: ai-inference
  labels:
    workload: light-inference
spec:
  containers:
    - name: embedding
      image: huggingface/tei:latest
      resources:
        limits:
          nvidia.com/mig-1g.10gb: 1
      env:
        - name: MODEL_NAME
          value: "BAAI/bge-large-zh-v1.5"
        - name: MAX_BATCH_SIZE
          value: "256"

GPU共享3种模式

模式对比

维度 时间分片(TS) MPS MIG
隔离级别 软件 硬件(部分) 硬件(完全)
显存隔离 否(共享) 否(共享) 是(独立)
性能隔离 差 中 好
并行度 高 中 中
故障隔离 差 差 好
适用场景 推理 训练 混合
GPU要求 通用 Volta+ A100+

时间分片配置

# gpu-time-slicing.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-time-slicing-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4
            devices: all
---
# 使用时间分片的Pod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-pool
  namespace: ai-inference
spec:
  replicas: 8
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      containers:
        - name: inference
          image: vllm/vllm-openai:latest
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: MODEL_NAME
              value: "Qwen/Qwen2.5-7B-Instruct"

MPS配置

# gpu-mps-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-mps-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    sharing:
      mps:
        resources:
          - name: nvidia.com/gpu
            replicas: 2
            devices: all
---
apiVersion: v1
kind: Pod
metadata:
  name: training-job-mps
  namespace: ai-training
spec:
  containers:
    - name: training
      image: pytorch/pytorch:2.4-cuda12.4
      resources:
        limits:
          nvidia.com/gpu: 1
      command:
        - python
        - -m
        - torch.distributed.launch
        - --nproc_per_node=2
        - train.py

共享模式选型决策

是否有A100/H100?
├── 否 → 时间分片(推理) / MPS(训练)
└── 是 → 需要完全隔离?
    ├── 是 → MIG分区
    └── 否 → 需要训练?
        ├── 是 → MPS
        └── 否 → 时间分片

K8s GPU Device Plugin配置

NVIDIA Device Plugin部署

# nvidia-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      priorityClassName: system-node-critical
      containers:
        - name: nvidia-device-plugin
          image: nvcr.io/nvidia/k8s-device-plugin:v0.16.0
          args:
            - --config=default
            - --mig-strategy=mixed
            - --pass-device-specs=true
            - --device-list-strategy=configmap
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

GPU资源监控

# gpu-monitor.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-monitor-config
  namespace: monitoring
data:
  gpu-metrics.json: |
    {
      "metrics": [
        "gpu_utilization",
        "gpu_memory_utilization",
        "gpu_memory_used_bytes",
        "gpu_memory_total_bytes",
        "gpu_power_usage_watts",
        "gpu_temperature_celsius",
        "gpu_sm_clock_mhz",
        "gpu_mem_clock_mhz"
      ],
      "scrape_interval": "15s",
      "labels": {
        "cluster": "production",
        "region": "cn-east"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    metadata:
      labels:
        app: dcgm-exporter
    spec:
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.7
          ports:
            - containerPort: 9400
              name: metrics
          env:
            - name: DCGM_EXPORTER_COLLECTORS
              value: "/etc/dcgm-exporter/dcp-metrics-inventory.csv"
          resources:
            limits:
              nvidia.com/gpu: 1

多租户GPU调度策略

GPU配额管理

# gpu-resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-team-a
  namespace: team-a
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    limits.nvidia.com/gpu: "8"
    requests.nvidia.com/mig-3g.40gb: "4"
    limits.nvidia.com/mig-3g.40gb: "4"
    requests.nvidia.com/mig-1g.10gb: "14"
    limits.nvidia.com/mig-1g.10gb: "14"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-team-b
  namespace: team-b
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"
    requests.nvidia.com/mig-3g.40gb: "2"
    limits.nvidia.com/mig-3g.40gb: "2"

优先级抢占调度

# gpu-priority-class.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-critical
value: 1000000
globalDefault: false
description: "Critical GPU workloads - can preempt others"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-high
value: 900000
globalDefault: false
description: "High priority GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-low
value: 100000
globalDefault: false
description: "Low priority GPU workloads - preemptible"
---
# 在线推理 - 高优先级
apiVersion: apps/v1
kind: Deployment
metadata:
  name: online-inference
spec:
  template:
    spec:
      priorityClassName: gpu-critical
      containers:
        - name: inference
          resources:
            limits:
              nvidia.com/mig-3g.40gb: 1
---
# 离线训练 - 低优先级
apiVersion: batch/v1
kind: Job
metadata:
  name: offline-training
spec:
  template:
    spec:
      priorityClassName: gpu-low
      containers:
        - name: training
          resources:
            limits:
              nvidia.com/gpu: 4

GPU弹性伸缩

# gpu-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-hpa
  namespace: ai-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-pool
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_utilization
        target:
          type: AverageValue
          averageValue: "70"
    - type: Pods
      pods:
        metric:
          name: request_latency_ms
        target:
          type: AverageValue
          averageValue: "200"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 300

GPU集群生产管理

GPU节点标签与调度

# gpu-node-labels.yaml
# 标记GPU类型
kubectl label nodes gpu-node-1 nvidia.com/gpu.product=A100-SXM4-80GB
kubectl label nodes gpu-node-2 nvidia.com/gpu.product=H100-SXM5-80GB
kubectl label nodes gpu-node-3 nvidia.com/gpu.product=L40S-48GB

# 标记MIG配置
kubectl label nodes gpu-node-1 nvidia.com/mig.config=mixed
kubectl label nodes gpu-node-2 nvidia.com/mig.config=all-3g.40gb

# 调度到特定GPU
apiVersion: v1
kind: Pod
metadata:
  name: h100-training
spec:
  nodeSelector:
    nvidia.com/gpu.product: H100-SXM5-80GB
  containers:
    - name: training
      resources:
        limits:
          nvidia.com/gpu: 8

GPU故障自愈

# gpu-health-check.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-health-monitor
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: gpu-health-monitor
  template:
    metadata:
      labels:
        app: gpu-health-monitor
    spec:
      serviceAccountName: gpu-health-sa
      containers:
        - name: monitor
          image: custom/gpu-health-monitor:latest
          env:
            - name: CHECK_INTERVAL
              value: "60"
            - name: MEMORY_ERROR_THRESHOLD
              value: "10"
            - name: TEMPERATURE_THRESHOLD
              value: "90"
            - name: AUTO_CORDON
              value: "true"
          volumeMounts:
            - name: nvidia
              mountPath: /usr/local/nvidia
      volumes:
        - name: nvidia
          hostPath:
            path: /usr/local/nvidia

GPU利用率优化效果

优化措施 利用率提升 成本节省
MIG分区 +35% 30%
时间分片 +25% 20%
优先级调度 +15% 15%
弹性伸缩 +10% 10%
综合优化 +50% 50%

总结与引流

关键要点回顾

  1. MIG分区：A100/H100的杀手锏，1卡变7卡，推理场景首选
  2. GPU共享：时间分片适合推理、MPS适合训练、MIG适合混合
  3. 多租户调度：配额+优先级+弹性伸缩三件套确保SLA
  4. 综合优化：MIG+调度+伸缩组合可实现50%成本节省

GPU调度方案推荐

集群规模 推荐方案 预期利用率
<8卡 时间分片 60%
8-32卡 MIG+时间分片 75%
32-128卡 MIG+优先级调度 80%
>128卡 全套方案 85%+

需要快速生成cURL命令测试GPU推理API？试试我们的cURL转代码工具和JSON格式化。

延伸阅读

本站提供浏览器本地工具，免注册即可试用 →

#K8s GPU调度#AI推理调度#GPU共享#MIG分区#K8s AI工作负载#2026