Go K8s PDB & HPA Production: 6 Key Configurations for Zero-Downtime Auto-Scaling

云原生

When Auto-Scaling Becomes a Disaster: K8s Elasticity's Darkest Hour

3 AM, a traffic surge triggers HPA scale-up. But new Pods need 15 seconds of cold start, and existing Pods are already being OOM Killed. Worse, PDB is not configured — 3 Pods are evicted simultaneously during scale-down, causing an immediate 503. The outage lasts 40 minutes, impacting 100K users.

This isn't an isolated case. Scaling-induced service disruptions, improper HPA metric selection, missing PDB configuration, and severe resource waste have become the four major pain points of K8s auto-scaling. PDB (PodDisruptionBudget) guarantees minimum available instances, while HPA (HorizontalPodAutoscaler) enables on-demand scaling. Together, they achieve true zero-downtime auto-scaling. This article walks you through 6 key configurations to build a production-grade K8s resilience system.

Core Concepts Reference

Concept Full Name Purpose Key Parameters
PDB PodDisruptionBudget Limit minimum available Pods during voluntary disruptions minAvailable / maxUnavailable
HPA HorizontalPodAutoscaler Auto-scale Pod replicas based on metrics Target CPU/memory, custom metrics
VPA VerticalPodAutoscaler Auto-adjust Pod resource requests/limits minAllowed / maxAllowed
minAvailable — Minimum Pods that must remain available in PDB Absolute value or percentage
maxUnavailable — Maximum Pods allowed unavailable in PDB Absolute value or percentage
Target CPU — CPU utilization threshold that triggers HPA scale-up Typically 60%-80%
Custom Metrics — HPA scaling based on business metrics QPS, queue depth, etc.
Scaling Policy — HPA scale-up/down behavior control scaleUp/scaleDown policies
Cold Start — Time from Pod start to ready Impacts scale-up response speed

Problem Analysis: 5 Challenges of K8s Auto-Scaling

Challenge 1: Scale-Up Delay Causes Overload. HPA detects a CPU spike and triggers scale-up, but new Pods take 10-30 seconds from scheduling to ready. During this time, traffic keeps pouring in and existing Pods may be overwhelmed.

Challenge 2: Scale-Down Causes Service Disruption. HPA randomly selects Pods to terminate during scale-down. Without PDB, too many Pods may be terminated simultaneously, causing a sudden drop in service capacity or even unavailability.

Challenge 3: Improper Metric Selection. Scaling based solely on CPU doesn't reflect real load. When a Go service has low CPU but high goroutine accumulation, HPA won't scale up, causing latency spikes.

Challenge 4: Cold Start Impact. Go applications need time to initialize connection pools and load configs. If readinessProbe is misconfigured, traffic hits new Pods before they're ready, causing request failures.

Challenge 5: Resource Fragmentation. After HPA scales up, Pods may be unevenly distributed. Scale-down may concentrate deletions on one node, causing unbalanced resource utilization and potentially triggering node-level cascading failures.

Configuration 1: PDB Minimum Availability Guarantee

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-service-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api-service
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: gateway-pdb
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: gateway

PDB ensures at least 2 Pods remain available via minAvailable, or limits to 1 Pod unavailable via maxUnavailable. Key principle: minAvailable works for services with fixed replica counts, maxUnavailable for services with dynamic replica counts. PDB only protects against voluntary disruptions (node maintenance, scale-down), not involuntary disruptions (Pod crashes).

Configuration 2: HPA CPU/Memory Auto-Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

HPA triggers scaling at 70% CPU and 80% memory. scaleUp policy allows doubling replicas within 60 seconds, scaleDown policy reduces at most 10% every 60 seconds, and stabilizationWindowSeconds prevents scale-down flapping. Key: Set scale-down cooldown to 300 seconds to avoid frequent scaling from traffic fluctuations.

Configuration 3: HPA Custom Metric Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service-custom-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: 1000
    - type: Pods
      pods:
        metric:
          name: goroutine_count
        target:
          type: AverageValue
          averageValue: "5000"
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Custom metrics are exposed to HPA via Prometheus Adapter. http_requests_per_second scales based on QPS, goroutine_count scales based on Go runtime goroutine count. Key: When Go services have low CPU but high goroutine accumulation, CPU-only metrics won't trigger scale-up — custom goroutine metrics are essential for production.

Go code for exposing custom metrics:

package main

import (
    "net/http"
    "runtime"
    "sync/atomic"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestCounter atomic.Int64
    goroutineGauge = prometheus.NewGaugeFunc(
        prometheus.GaugeOpts{
            Name: "goroutine_count",
            Help: "Current number of goroutines",
        },
        func() float64 {
            return float64(runtime.NumGoroutine())
        },
    )
    httpRequestsPerSecond = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "http_requests_per_second",
            Help: "HTTP requests per second",
        },
        []string{"method", "path"},
    )
)

func init() {
    prometheus.MustRegister(goroutineGauge)
    prometheus.MustRegister(httpRequestsPerSecond)
}

func metricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        httpRequestsPerSecond.WithLabelValues(r.Method, r.URL.Path).Inc()
        next.ServeHTTP(w, r)
    })
}

Configuration 4: Scaling Policies and Cooldown Periods

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service-behavior-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 4
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      selectPolicy: Min
      policies:
        - type: Percent
          value: 5
          periodSeconds: 120
        - type: Pods
          value: 1
          periodSeconds: 120

Scale-up policy selectPolicy: Max chooses the most aggressive policy, ensuring fast response to load growth. Scale-down policy selectPolicy: Min chooses the most conservative policy, with a 600-second cooldown window preventing premature scale-down. Production rule: Scale up fast, scale down slow — better to spend extra resources than risk service disruption.

Configuration 5: Go Application Startup Optimization and Readiness Probes

package main

import (
    "context"
    "database/sql"
    "fmt"
    "net/http"
    "time"

    _ "github.com/go-sql-driver/mysql"
    "github.com/redis/go-redis/v9"
)

type App struct {
    db    *sql.DB
    redis *redis.Client
    ready bool
}

func (a *App) Init(ctx context.Context) error {
    var err error
    a.db, err = sql.Open("mysql", "user:pass@tcp(mysql:3306)/db")
    if err != nil {
        return fmt.Errorf("open mysql: %w", err)
    }
    a.db.SetMaxOpenConns(50)
    a.db.SetMaxIdleConns(10)
    a.db.SetConnMaxLifetime(5 * time.Minute)

    for i := 0; i < 10; i++ {
        if err = a.db.PingContext(ctx); err == nil {
            break
        }
        time.Sleep(time.Second)
    }
    if err != nil {
        return fmt.Errorf("ping mysql after retries: %w", err)
    }

    a.redis = redis.NewClient(&redis.Options{
        Addr:         "redis:6379",
        PoolSize:     50,
        MinIdleConns: 10,
    })
    if err = a.redis.Ping(ctx).Err(); err != nil {
        return fmt.Errorf("ping redis: %w", err)
    }

    a.ready = true
    return nil
}

func (a *App) ReadinessHandler(w http.ResponseWriter, r *http.Request) {
    if !a.ready {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    if err := a.db.PingContext(r.Context()); err != nil {
        http.Error(w, "db unhealthy", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: api-service
          image: api-service:latest
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi

Key design: startupProbe gives slow-starting Pods enough initialization time (up to 60 seconds), readinessProbe checks dependency health, preStop hook gives Pods 10 seconds for graceful shutdown, terminationGracePeriodSeconds ensures in-flight requests complete after SIGTERM.

Configuration 6: End-to-End Resilience Testing

package main

import (
    "context"
    "fmt"
    "os"
    "time"

    autoscalingv2 "k8s.io/api/autoscaling/v2"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

type ResilienceTester struct {
    clientset *kubernetes.Clientset
    namespace string
}

func NewResilienceTester(kubeconfig, namespace string) (*ResilienceTester, error) {
    config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        return nil, fmt.Errorf("build kubeconfig: %w", err)
    }
    cs, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }
    return &ResilienceTester{clientset: cs, namespace: namespace}, nil
}

func (t *ResilienceTester) TestScaleUp(ctx context.Context, deployName string) error {
    hpa, err := t.clientset.AutoscalingV2().HorizontalPodAutoscalers(t.namespace).Get(ctx, deployName+"-hpa", metav1.GetOptions{})
    if err != nil {
        return fmt.Errorf("get hpa: %w", err)
    }
    fmt.Printf("HPA %s: min=%d max=%d current=%d\n",
        hpa.Name, *hpa.Spec.MinReplicas, hpa.Spec.MaxReplicas, hpa.Status.CurrentReplicas)

    deploy, err := t.clientset.AppsV1().Deployments(t.namespace).Get(ctx, deployName, metav1.GetOptions{})
    if err != nil {
        return fmt.Errorf("get deploy: %w", err)
    }
    fmt.Printf("Deployment %s: replicas=%d ready=%d available=%d\n",
        deploy.Name, deploy.Status.Replicas, deploy.Status.ReadyReplicas, deploy.Status.AvailableReplicas)
    return nil
}

func (t *ResilienceTester) TestPDB(ctx context.Context, pdbName string) error {
    pdb, err := t.clientset.PolicyV1().PodDisruptionBudgets(t.namespace).Get(ctx, pdbName, metav1.GetOptions{})
    if err != nil {
        return fmt.Errorf("get pdb: %w", err)
    }
    fmt.Printf("PDB %s: disruptionsAllowed=%d currentHealthy=%d desiredHealthy=%d\n",
        pdb.Name, pdb.Status.DisruptionsAllowed, pdb.Status.CurrentHealthy, pdb.Status.DesiredHealthy)
    return nil
}

func (t *ResilienceTester) RunFullTest(ctx context.Context) error {
    fmt.Println("=== PDB Test ===")
    if err := t.TestPDB(ctx, "api-service-pdb"); err != nil {
        fmt.Fprintf(os.Stderr, "PDB test failed: %v\n", err)
    }

    fmt.Println("=== HPA Test ===")
    if err := t.TestScaleUp(ctx, "api-service"); err != nil {
        fmt.Fprintf(os.Stderr, "HPA test failed: %v\n", err)
    }

    fmt.Println("=== Scale Up Simulation ===")
    scale, err := t.clientset.AppsV1().Deployments(t.namespace).GetScale(ctx, "api-service", metav1.GetOptions{})
    if err != nil {
        return fmt.Errorf("get scale: %w", err)
    }
    newScale := scale.DeepCopy()
    newScale.Spec.Replicas = scale.Spec.Replicas * 2
    _, err = t.clientset.AppsV1().Deployments(t.namespace).UpdateScale(ctx, "api-service", newScale, metav1.UpdateOptions{})
    if err != nil {
        return fmt.Errorf("update scale: %w", err)
    }
    fmt.Printf("Scaled from %d to %d replicas\n", scale.Spec.Replicas, newScale.Spec.Replicas)

    time.Sleep(30 * time.Second)
    return t.TestScaleUp(ctx, "api-service")
}

func main() {
    kubeconfig := os.Getenv("KUBECONFIG")
    if kubeconfig == "" {
        kubeconfig = clientcmd.RecommendedHomeFile
    }
    tester, err := NewResilienceTester(kubeconfig, "production")
    if err != nil {
        fmt.Fprintf(os.Stderr, "init tester: %v\n", err)
        os.Exit(1)
    }
    if err := tester.RunFullTest(context.Background()); err != nil {
        fmt.Fprintf(os.Stderr, "test failed: %v\n", err)
        os.Exit(1)
    }
}

End-to-end testing verifies PDB protection, HPA scaling, and Deployment status. In production, run during low-traffic periods and observe whether scaling is smooth, PDB is effective, and Pods become healthy and ready.

5 Common Pitfalls

❌ Pitfall 1: Setting PDB minAvailable to 100% ✅ 100% means no voluntary disruptions are allowed — node maintenance becomes impossible. Set to 50%-66% to ensure at least half the Pods remain available.

❌ Pitfall 2: Setting HPA target CPU to 90% ✅ A 90% threshold means Pods are near their limit before scaling triggers — request latency will inevitably spike. Set to 60%-75% to leave a scaling buffer.

❌ Pitfall 3: Only configuring CPU metrics, ignoring memory and custom metrics ✅ Go services may have low CPU but high memory/goroutine usage. You must combine CPU + memory + business metrics to accurately reflect load.

❌ Pitfall 4: Using the same endpoint for readinessProbe and livenessProbe ✅ Readiness probe should check dependencies (DB/Redis), liveness probe should only check the process. Sharing an endpoint causes dependency flaps to restart Pods, worsening the failure.

❌ Pitfall 5: Ignoring preStop hook ✅ Without preStop, Pods are immediately removed from Service endpoints after SIGTERM — in-flight requests may be lost. sleep 10 gives Pods enough time to complete requests.

10 Error Troubleshooting

Error Symptom Possible Cause Debug Command Solution
HPA can't get CPU metrics Metrics Server not installed kubectl get pods -n kube-system | grep metrics Install Metrics Server
PDB DisruptionsAllowed=0 minAvailable equals current replicas kubectl get pdb -o yaml Lower minAvailable or increase replicas
HPA scale-up not triggering Metrics below threshold kubectl get hpa -o yaml Check current metric values and thresholds
Pod Pending after scale-up Insufficient node resources kubectl describe pod <pending-pod> Add nodes or reduce resource requests
Service 503 after scale-down PDB not configured or too low kubectl get pdb Configure PDB to guarantee minimum availability
Pod CrashLoopBackOff immediately after start readinessProbe failing kubectl logs <pod> Check dependency initialization and probe config
HPA scaling frequently Scale-down cooldown too short kubectl describe hpa Increase stabilizationWindowSeconds
Custom metrics unavailable Prometheus Adapter not configured kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 Deploy Prometheus Adapter
Pods force-evicted during node maintenance PDB not created kubectl get pdb -A Create PDB for critical services
Scale-up too slow scaleUp policy too conservative kubectl describe hpa Adjust scaleUp policy to Percent:100

Advanced Optimization

1. Predictive Scaling. Based on historical traffic patterns, scale up before peak hours arrive. Combine KEDA's Cron trigger or a custom predictive controller for "resources ready before traffic."

2. Pod Topology Spread Constraints. Use topologySpreadConstraints to ensure scaled-up Pods are evenly distributed across availability zones, preventing single-AZ failures from causing service unavailability.

3. Priority and Preemption. Set high priority classes for critical services — when resources are scarce, critical services are prioritized while low-priority services can be preempted.

4. VPA and HPA Coordination. VPA adjusts resource requests, HPA adjusts replica counts. Recommend VPA in recommendation-only mode (mode: Off) to avoid conflicts with HPA.

5. FinOps Cost Optimization. Combine Spot instances with Cluster Autoscaler — non-critical services use Spot instances for cost savings, critical services use On-Demand instances for stability.

Comparison: HPA vs VPA vs KEDA vs Cluster Autoscaler

Feature HPA VPA KEDA Cluster Autoscaler
Scaling Dimension Horizontal (replicas) Vertical (resources) Horizontal (replicas) Node count
Trigger Method CPU/memory/custom metrics Historical resource usage Event-driven (multi-source) Pod scheduling failure
Use Case High load variance Misconfigured resources Event-driven/batch Insufficient node resources
PDB Compatibility ✅ Required ⚠️ May conflict ✅ Required ✅ Required
Cold Start Impact ⚠️ Affected ✅ Not affected ⚠️ Affected ⚠️ Affected
Go Service Fit ⚠️ Needs custom metrics ✅ Auto-adjusts ✅ Rich triggers ✅ Transparent
Production Maturity ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐
Recommended Combo HPA+PDB VPA recommendation mode KEDA+PDB CA+HPA+PDB

Summary

K8s auto-scaling isn't just about configuring an HPA — it's a four-part system: PDB for availability, HPA for elasticity, probes for readiness, and policies for pacing. The 6 key configurations — PDB minimum availability, HPA CPU/memory scaling, custom metric scaling, scaling policies and cooldown, Go startup optimization and probes, and end-to-end resilience testing — cover the complete production resilience chain. Remember: scale up fast, scale down slow, PDB is mandatory, probes must be separate — that's how you achieve true zero-downtime auto-scaling. In the future, AI-based predictive scaling and serverless elasticity will further reduce operational complexity.

  • JSON Formatter — Format HPA/PDB YAML/JSON configs, quickly debug resource definition issues
  • Hash Calculator — Calculate ConfigMap and Secret checksums, ensure scaling config data integrity
  • cURL to Code — Convert cURL test commands to Go code, accelerate K8s API client development

Further Reading

Try these browser-local tools — no sign-up required →

#K8s PDB#K8s HPA#PodDisruptionBudget#自动伸缩#Go微服务#2026#云原生