Go K8s Gateway API Production: 6 Critical Steps to Migrate from Ingress

云原生

When Ingress Becomes the Bottleneck: K8s Traffic Management's Darkest Hour

2 AM, ops team urgently rolls out a canary deployment. The Ingress nginx annotation is modified 3 times, but nginx.ingress.kubernetes.io/canary never takes effect. Worse, 3 teams share one Ingress controller — any annotation change risks overwriting another team's config. The canary is rolled back, deployment delayed by 6 hours.

This isn't an isolated case. Ingress annotation hell, cross-namespace limitations, and multi-team collaboration conflicts have become the biggest pain points in K8s traffic management. Kubernetes Gateway API, as Ingress's successor, solves these problems through role-based design, extensible routing, and native traffic splitting. This article walks you through 6 critical steps for a production-grade migration from Ingress to Gateway API.

Core Concepts Reference

Resource Role Responsibility Analogy
GatewayClass Infrastructure Provider Define controller type and parameters IngressClass
Gateway Cluster Operator Define entry listeners and infrastructure Ingress controller itself
HTTPRoute Application Developer Define HTTP routing rules Ingress rules
TCPRoute Application Developer Define TCP routing rules No native Ingress support
GRPCRoute Application Developer Define gRPC routing rules No native Ingress support
ReferenceGrant Cluster Operator Allow cross-namespace references No equivalent
ParentRef Application Developer Bind Route to Gateway Ingress's ingressClassName

Table of Contents

  1. Problem Analysis: 5 Challenges with Ingress
  2. Step 1: GatewayClass and Infrastructure Configuration
  3. Step 2: Gateway Resource Definition
  4. Step 3: HTTPRoute Routing Rules Configuration
  5. Step 4: Ingress to Gateway API Migration Script (Go)
  6. Step 5: Canary Release and Traffic Splitting
  7. Step 6: Multi-Cluster Gateway Configuration
  8. 5 Common Pitfalls
  9. 10 Error Troubleshooting
  10. Advanced Optimization Tips
  11. Comparison: Ingress vs Gateway API vs Custom Gateway
  12. Recommended Tools
  13. Summary and Further Reading

Problem Analysis: 5 Challenges with Ingress

Challenge 1: Annotation Inconsistency. Nginx Ingress uses nginx.ingress.kubernetes.io/rewrite-target, Traefik uses traefik.ingress.kubernetes.io/rewrite-target. Incompatible annotations make controller migration extremely costly.

Challenge 2: Cross-Namespace Routing. Ingress backend can only reference Services in the same namespace. Cross-namespace traffic requires ExternalName Service hacks — insecure and uncontrollable.

Challenge 3: Multi-Cluster Gateway. Ingress was designed for single clusters only. Multi-cluster scenarios require Federation or custom control planes.

Challenge 4: Complex Canary Releases. Nginx Ingress canary annotations require creating two Ingress objects. Weight configuration is unintuitive and mixed with regular routing.

Challenge 5: Limited Protocol Support. Ingress only supports HTTP/HTTPS. TCP/UDP requires extra configuration, gRPC needs annotation hacks, and WebSocket support varies by controller.

Step 1: GatewayClass and Infrastructure Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: production-gateway-class
  annotations:
    gateway.networking.k8s.io/controller: nginx
spec:
  controllerName: k8s.io/nginx-gateway-controller
  parametersRef:
    group: ""
    kind: ConfigMap
    name: gateway-params
    namespace: gateway-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: gateway-params
  namespace: gateway-system
data:
  workerProcesses: "auto"
  maxConnections: "10240"
  proxyBufferSize: "16k"

GatewayClass is managed by the infrastructure team, defining the controller type and global parameters. Different environments can use different GatewayClasses to isolate dev/staging/production.

Step 2: Gateway Resource Definition

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: gateway-system
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
spec:
  gatewayClassName: production-gateway-class
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: "true"
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
            namespace: gateway-system
      allowedRoutes:
        namespaces:
          from: All
  addresses:
    - type: IPAddress
      value: 10.0.1.100

Gateway is managed by the ops team, defining listeners, TLS termination, and allowed route namespaces. allowedRoutes implements fine-grained namespace access control, replacing Ingress's implicit global access.

Step 3: HTTPRoute Routing Rules Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-service-route
  namespace: app-team-a
spec:
  parentRefs:
    - name: production-gateway
      namespace: gateway-system
      sectionName: https
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v2
        - headers:
            - type: Exact
              name: X-API-Version
              value: "2"
      backendRefs:
        - name: api-v2-service
          port: 8080
          weight: 90
        - name: api-v2-canary-service
          port: 8080
          weight: 10
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: api-v1-service
          port: 8080

HTTPRoute is managed by application teams, supporting path matching, header matching, and weight-based traffic splitting. parentRefs binds to a specific Gateway listener, and weight enables native traffic splitting.

Step 4: Ingress to Gateway API Migration Script (Go)

package main

import (
    "context"
    "fmt"
    "os"

    networkingv1 "k8s.io/api/networking/v1"
    gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

type IngressMigrator struct {
    clientset    *kubernetes.Clientset
    gatewayClass string
    gatewayName  string
    gatewayNS    string
}

func NewIngressMigrator(kubeconfig, gatewayClass, gatewayName, gatewayNS string) (*IngressMigrator, error) {
    config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        return nil, fmt.Errorf("build kubeconfig: %w", err)
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }
    return &IngressMigrator{
        clientset:    clientset,
        gatewayClass: gatewayClass,
        gatewayName:  gatewayName,
        gatewayNS:    gatewayNS,
    }, nil
}

func (m *IngressMigrator) Migrate(ctx context.Context, namespace string) ([]*gatewayv1.HTTPRoute, error) {
    ingresses, err := m.clientset.NetworkingV1().Ingresses(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list ingresses: %w", err)
    }

    var routes []*gatewayv1.HTTPRoute
    for _, ing := range ingresses.Items {
        route, err := m.convertIngressToHTTPRoute(&ing)
        if err != nil {
            fmt.Fprintf(os.Stderr, "skip ingress %s: %v\n", ing.Name, err)
            continue
        }
        routes = append(routes, route)
    }
    return routes, nil
}

func (m *IngressMigrator) convertIngressToHTTPRoute(ing *networkingv1.Ingress) (*gatewayv1.HTTPRoute, error) {
    route := &gatewayv1.HTTPRoute{
        TypeMeta: metav1.TypeMeta{
            APIVersion: "gateway.networking.k8s.io/v1",
            Kind:       "HTTPRoute",
        },
        ObjectMeta: metav1.ObjectMeta{
            Name:        ing.Name + "-route",
            Namespace:   ing.Namespace,
            Annotations: filterGatewayAnnotations(ing.Annotations),
        },
        Spec: gatewayv1.HTTPRouteSpec{
            ParentRefs: []gatewayv1.ParentReference{
                {
                    Name:      gatewayv1.ObjectName(m.gatewayName),
                    Namespace: (*gatewayv1.Namespace)(&m.gatewayNS),
                },
            },
        },
    }

    for _, rule := range ing.Spec.Rules {
        if rule.Host != "" {
            hostname := gatewayv1.PreciseHostname(rule.Host)
            route.Spec.Hostnames = append(route.Spec.Hostnames, hostname)
        }
        for _, path := range rule.HTTP.Paths {
            matchType := gatewayv1.PathMatchPathPrefix
            if path.PathType != nil && *path.PathType == networkingv1.PathTypeExact {
                matchType = gatewayv1.PathMatchExact
            }
            route.Spec.Rules = append(route.Spec.Rules, gatewayv1.HTTPRouteRule{
                Matches: []gatewayv1.HTTPRouteMatch{
                    {
                        Path: &gatewayv1.HTTPPathMatch{
                            Type:  &matchType,
                            Value: &path.Path,
                        },
                    },
                },
                BackendRefs: []gatewayv1.HTTPBackendRef{
                    {
                        BackendRef: gatewayv1.BackendRef{
                            BackendObjectReference: gatewayv1.BackendObjectReference{
                                Name: gatewayv1.ObjectName(path.Backend.Service.Name),
                                Port: (*gatewayv1.PortNumber)(&path.Backend.Service.Port.Number),
                            },
                        },
                    },
                },
            })
        }
    }
    return route, nil
}

func filterGatewayAnnotations(annotations map[string]string) map[string]string {
    filtered := make(map[string]string)
    for k, v := range annotations {
        if k == "kubernetes.io/ingress.class" || k == "nginx.ingress.kubernetes.io/rewrite-target" {
            continue
        }
        filtered[k] = v
    }
    return filtered
}

func main() {
    kubeconfig := os.Getenv("KUBECONFIG")
    if kubeconfig == "" {
        kubeconfig = clientcmd.RecommendedHomeFile
    }

    migrator, err := NewIngressMigrator(kubeconfig, "production-gateway-class", "production-gateway", "gateway-system")
    if err != nil {
        fmt.Fprintf(os.Stderr, "init migrator: %v\n", err)
        os.Exit(1)
    }

    routes, err := migrator.Migrate(context.Background(), "default")
    if err != nil {
        fmt.Fprintf(os.Stderr, "migrate: %v\n", err)
        os.Exit(1)
    }

    for _, route := range routes {
        fmt.Printf("Generated HTTPRoute: %s/%s\n", route.Namespace, route.Name)
    }
    fmt.Printf("Total: %d routes migrated\n", len(routes))
}

Step 5: Canary Release and Traffic Splitting

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-canary-route
  namespace: app-team-a
spec:
  parentRefs:
    - name: production-gateway
      namespace: gateway-system
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - headers:
            - type: Exact
              name: X-Canary
              value: "true"
      backendRefs:
        - name: api-v2-service
          port: 8080
          weight: 100
    - backendRefs:
        - name: api-v1-service
          port: 8080
          weight: 95
        - name: api-v2-service
          port: 8080
          weight: 5

Gateway API natively supports header-based and weight-based traffic splitting — no Nginx canary annotations needed. Start with 5% traffic and gradually increase to 100% to complete the canary release.

Step 6: Multi-Cluster Gateway Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: multi-cluster-gateway-class
spec:
  controllerName: k8s.io/nginx-gateway-controller
  parametersRef:
    group: multicluster.x-k8s.io
    kind: ServiceImport
    name: multi-cluster-params
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: cross-cluster-route
  namespace: app-team-a
spec:
  parentRefs:
    - name: production-gateway
      namespace: gateway-system
  hostnames:
    - "api.example.com"
  rules:
    - backendRefs:
        - name: api-service-cluster-east
          port: 8080
          weight: 70
        - name: api-service-cluster-west
          port: 8080
          weight: 30

Multi-cluster gateways use ServiceImport and the Multi-Cluster Service API for cross-cluster traffic distribution, with weight-based controls for regional affinity and failover.

5 Common Pitfalls

❌ Pitfall 1: Placing Gateway and Route in the same namespace ✅ Put Gateway in gateway-system, Route in application namespaces, and control access via allowedRoutes. Role separation is Gateway API's core design principle.

❌ Pitfall 2: Ignoring ReferenceGrant for cross-namespace references ✅ When referencing Services or Secrets across namespaces, you must create a ReferenceGrant in the target namespace — otherwise routes won't take effect.

❌ Pitfall 3: Deleting Ingress before creating Gateway ✅ Create Gateway and HTTPRoute first, verify routes are working, then delete Ingress for a smooth migration.

❌ Pitfall 4: HTTPRoute weight totals not summing to 100 ✅ Weights are relative and don't need to sum to 100, but keeping the total at 100 makes calculation and understanding easier.

❌ Pitfall 5: GatewayClass without parametersRef ✅ In production, always configure controller parameters via parametersRef to avoid performance bottlenecks from defaults.

10 Error Troubleshooting

Error Symptom Possible Cause Debug Command Solution
HTTPRoute status is empty parentRef points to non-existent Gateway kubectl get gateway -A Check Gateway name and namespace
Route not taking effect allowedRoutes doesn't permit Route namespace kubectl describe gateway Add namespace label or change to All
TLS certificate not loaded certificateRefs Secret doesn't exist kubectl get secret -n gateway-system Create TLS Secret
Cross-namespace route fails Missing ReferenceGrant kubectl get referencegrant -A Create ReferenceGrant to allow reference
Canary traffic not splitting weight is 0 or not set kubectl describe httproute Check backendRefs weight values
GatewayClass unavailable Controller Pod not running kubectl get pods -n gateway-system Start Gateway controller
TCPRoute not working Gateway not listening on TCP port kubectl describe gateway Add TCP protocol listener
GRPCRoute routing fails Backend Service doesn't support gRPC kubectl describe svc Verify Service supports gRPC protocol
Migration script returns 404 client-go version incompatible with cluster kubectl version Match client-go with K8s version
Multi-cluster route unbalanced ServiceImport misconfigured kubectl get serviceimport -A Check Multi-Cluster Service config

Advanced Optimization Tips

1. Gateway Infrastructure Separation. Create different GatewayClasses for services with different SLAs. Core services use dedicated Gateway instances to avoid shared infrastructure interference.

2. Route Policy Composition. HTTPRoute supports nested matches — combine path, header, and query parameters for fine-grained routing. For example, match both /api/v2 path and X-Feature-Flag: enabled header.

3. Timeout and Retry Policies. Use backendReqTimeout and retry policy annotations to implement timeout control and automatic retries at the routing layer, reducing application-level code.

4. Observability Integration. Gateway API supports OpenTelemetry integration via PolicyAttachment for route-level Trace and Metrics collection.

5. Progressive Migration. Use kubectl apply --dry-run=server to validate YAML and kubectl diff to preview changes, ensuring zero-risk migration.

Comparison: Ingress vs Gateway API vs Custom Gateway

Feature Ingress Gateway API Custom Gateway
Role Separation ❌ All mixed together ✅ Infra/Ops/Dev separated ⚠️ Depends on implementation
Cross-Namespace ❌ Not supported ✅ ReferenceGrant ✅ Free configuration
Traffic Splitting ⚠️ Annotation hacks ✅ Native weight ✅ Free configuration
TCP/gRPC ❌ Annotation hacks ✅ Native support ✅ Free configuration
Multi-Cluster ❌ Not supported ⚠️ Requires MCS API ✅ Free configuration
Portability ⚠️ Incompatible annotations ✅ Standardized API ❌ Fully coupled
Learning Curve Low Medium High
Ops Cost Medium Low High
Community Support ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐
Production Readiness ⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐
  • JSON Formatter — Format Gateway API YAML/JSON configs, quickly debug resource definition issues
  • Hash Calculator — Calculate TLS certificate and ConfigMap checksums, ensure gateway config data integrity
  • cURL to Code — Convert cURL test commands to Go code, accelerate Gateway API client development

Summary

Gateway API isn't a simple upgrade to Ingress — it's a paradigm shift in K8s traffic management. From "one Ingress rules them all" to "role separation, clear responsibilities"; from "annotation hell" to "standardized API"; from "single cluster" to "native multi-cluster support". The 6 migration steps — GatewayClass definition, Gateway configuration, HTTPRoute routing, Go migration script, canary traffic splitting, and multi-cluster gateway — cover the complete production migration chain. Remember: create first, delete later; migrate progressively; validate every step — that's how you achieve zero-downtime migration.

Further Reading

Try these browser-local tools — no sign-up required →

#Kubernetes Gateway API#K8s网关#Ingress迁移#Go#流量管理#2026#云原生