AI Inference Gateway Patterns: LLM API Gateway, Model Routing, and Rate Limiting
技术架构
Summary
- The AI inference gateway is the "front door" of LLM services: routing, rate limiting, degradation, and observability are the 4 essential capabilities
- 3 model routing strategies: cost-first, latency-first, and quality-first — choose based on business scenario
- Rate limiting is not just about preventing abuse: Token Rate Limiting is more precise than request frequency limiting, preventing a single user from monopolizing the context window
- Fallback mechanisms are the SLA baseline: primary model timeout → secondary model takeover → cache fallback → graceful degradation
- This article provides a complete solution from gateway architecture to Go implementation, including K8s deployment and Prometheus monitoring
Table of Contents
- Why LLM Services Need a Dedicated Gateway
- AI Inference Gateway Architecture Design
- Model Routing: 3 Strategies and Implementation
- Token Rate Limiting: Precise Throttling
- Fallback and Degradation: The SLA Baseline
- Go Gateway Implementation and K8s Deployment
- Summary and Further Reading
Why LLM Services Need a Dedicated Gateway
3 Shortcomings of Traditional API Gateways
| Shortcoming | Description | Impact |
|---|---|---|
| No Token Awareness | Rate limits by request frequency, unaware of Token consumption | A single user with long-context requests can monopolize GPU |
| No Model Routing | Cannot route to different models based on request characteristics | Small questions use large models, wasting cost |
| No Streaming Adaptation | SSE streaming response has different rate limiting/degradation logic | Streaming requests cannot gracefully degrade after timeout |
AI Inference Gateway vs Traditional API Gateway
| Dimension | Traditional API Gateway | AI Inference Gateway |
|---|---|---|
| Rate Limiting Dimension | Request frequency | Token consumption + request frequency |
| Routing Strategy | URL path | Model capability + cost + latency |
| Response Mode | Request-response | SSE streaming + request-response |
| Degradation Strategy | Return error | Fallback to secondary model |
| Observability | QPS/latency | Token throughput/first-token latency/context length |
AI Inference Gateway Architecture Design
┌──────────────────────────────────────────────────────────────┐
│ AI Inference Gateway Architecture │
│ │
│ ┌──────────┐ │
│ │ Client │ │
│ └────┬─────┘ │
│ │ │
│ ┌────▼──────────────────────────────────────────────────┐ │
│ │ AI Inference Gateway │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐ │ │
│ │ │ Auth │ │ Rate │ │ Routing │ │ Degrade │ │ │
│ │ │ API Key │ │ Token RL │ │ Model │ │Fallback │ │ │
│ │ │ OAuth2.0 │ │ Priority │ │ Load Bal │ │ Cache │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └─────────┘ │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ Observability │ │ │
│ │ │ Prometheus + OpenTelemetry + Structured Logging │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────┘ │
│ │ │ │ │
│ ┌────▼─────┐ ┌───▼──────┐ ┌─▼────────┐ │
│ │ vLLM │ │ TGI │ │ SGLang │ │
│ │ Qwen-7B │ │ Llama-70B│ │ Qwen-7B │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────────────────────────────────────────┘
Model Routing: 3 Strategies and Implementation
Routing Strategy Comparison
| Strategy | Routing Logic | Cost | Latency | Quality | Use Case |
|---|---|---|---|---|---|
| Cost-First | Route to smallest model first | Lowest | Low | Medium | Internal tools |
| Latency-First | Route to fastest model first | Medium | Lowest | Medium | Real-time chat |
| Quality-First | Route to strongest model first | Highest | High | Highest | Professional scenarios |
Go Routing Implementation
package gateway
import (
"context"
"math"
"sync"
"time"
)
type ModelEndpoint struct {
Name string
URL string
ModelID string
MaxTokens int
CostPerToken float64
AvgLatency time.Duration
CurrentLoad float64
Capabilities []string
}
type RoutingStrategy string
const (
CostFirst RoutingStrategy = "cost_first"
LatencyFirst RoutingStrategy = "latency_first"
QualityFirst RoutingStrategy = "quality_first"
)
type ModelRouter struct {
endpoints []*ModelEndpoint
strategy RoutingStrategy
mu sync.RWMutex
}
func NewModelRouter(strategy RoutingStrategy) *ModelRouter {
return &ModelRouter{strategy: strategy}
}
func (r *ModelRouter) Register(endpoint *ModelEndpoint) {
r.mu.Lock()
defer r.mu.Unlock()
r.endpoints = append(r.endpoints, endpoint)
}
func (r *ModelRouter) Route(ctx context.Context, req *InferenceRequest) (*ModelEndpoint, error) {
r.mu.RLock()
defer r.mu.RUnlock()
candidates := r.filterByCapability(req)
if len(candidates) == 0 {
return nil, ErrNoAvailableModel
}
switch r.strategy {
case CostFirst:
return r.routeByCost(candidates), nil
case LatencyFirst:
return r.routeByLatency(candidates), nil
case QualityFirst:
return r.routeByQuality(candidates), nil
default:
return r.routeByLatency(candidates), nil
}
}
func (r *ModelRouter) routeByCost(candidates []*ModelEndpoint) *ModelEndpoint {
best := candidates[0]
for _, ep := range candidates[1:] {
if ep.CostPerToken < best.CostPerToken && ep.CurrentLoad < 0.9 {
best = ep
}
}
return best
}
func (r *ModelRouter) routeByLatency(candidates []*ModelEndpoint) *ModelEndpoint {
best := candidates[0]
for _, ep := range candidates[1:] {
effectiveLatency := float64(ep.AvgLatency) / (1.0 - ep.CurrentLoad + 0.01)
bestLatency := float64(best.AvgLatency) / (1.0 - best.CurrentLoad + 0.01)
if effectiveLatency < bestLatency {
best = ep
}
}
return best
}
func (r *ModelRouter) routeByQuality(candidates []*ModelEndpoint) *ModelEndpoint {
best := candidates[0]
for _, ep := range candidates[1:] {
if ep.MaxTokens > best.MaxTokens && ep.CurrentLoad < 0.9 {
best = ep
}
}
return best
}
func (r *ModelRouter) filterByCapability(req *InferenceRequest) []*ModelEndpoint {
var filtered []*ModelEndpoint
for _, ep := range r.endpoints {
if ep.CurrentLoad >= 0.95 {
continue
}
if req.MaxTokens > 0 && ep.MaxTokens < req.MaxTokens {
continue
}
filtered = append(filtered, ep)
}
return filtered
}
Token Rate Limiting: Precise Throttling
Token Rate Limiting vs Request Rate Limiting
| Rate Limiting Dimension | Advantages | Disadvantages | Use Case |
|---|---|---|---|
| Request Frequency | Simple implementation | Unaware of Token consumption | Low-precision rate limiting |
| Tokens/Minute | Precise GPU consumption control | Requires Token estimation | Production recommended |
| Concurrent Requests | Controls GPU concurrency | Unaware of context length | Simple scenarios |
Go Token Rate Limiting Implementation
package gateway
import (
"context"
"sync"
"time"
)
type TokenBucket struct {
mu sync.Mutex
tokens float64
maxTokens float64
refillRate float64
lastRefill time.Time
}
func NewTokenBucket(maxTokens, refillRate float64) *TokenBucket {
return &TokenBucket{
tokens: maxTokens,
maxTokens: maxTokens,
refillRate: refillRate,
lastRefill: time.Now(),
}
}
func (tb *TokenBucket) Allow(estimatedTokens int) bool {
tb.mu.Lock()
defer tb.mu.Unlock()
now := time.Now()
elapsed := now.Sub(tb.lastRefill).Seconds()
tb.tokens = math.Min(tb.maxTokens, tb.tokens+elapsed*tb.refillRate)
tb.lastRefill = now
if tb.tokens >= float64(estimatedTokens) {
tb.tokens -= float64(estimatedTokens)
return true
}
return false
}
type TokenRateLimiter struct {
buckets map[string]*TokenBucket
mu sync.RWMutex
}
func NewTokenRateLimiter() *TokenRateLimiter {
return &TokenRateLimiter{buckets: make(map[string]*TokenBucket)}
}
func (rl *TokenRateLimiter) RegisterUser(userID string, maxTokens, refillRate float64) {
rl.mu.Lock()
defer rl.mu.Unlock()
rl.buckets[userID] = NewTokenBucket(maxTokens, refillRate)
}
func (rl *TokenRateLimiter) Allow(userID string, estimatedTokens int) bool {
rl.mu.RLock()
bucket, ok := rl.buckets[userID]
rl.mu.RUnlock()
if !ok {
return false
}
return bucket.Allow(estimatedTokens)
}
Fallback and Degradation: The SLA Baseline
4-Level Degradation Strategy
┌──────────────────────────────────────────────────────────┐
│ 4-Level Degradation Strategy │
│ │
│ Level 0: Normal Service │
│ ┌──────────────────────────────────────────┐ │
│ │ Primary Model (Qwen-72B) → 50ms, Best Q │ │
│ └──────────────────────────────────────────┘ │
│ ↓ Timeout/Overload │
│ Level 1: Secondary Model Takeover │
│ ┌──────────────────────────────────────────┐ │
│ │ Secondary Model (Qwen-7B) → 20ms, Lower │ │
│ └──────────────────────────────────────────┘ │
│ ↓ Secondary Also Overloaded │
│ Level 2: Cache Fallback │
│ ┌──────────────────────────────────────────┐ │
│ │ Semantic Cache Hit → 5ms, Cache Quality │ │
│ └──────────────────────────────────────────┘ │
│ ↓ Cache Miss │
│ Level 3: Graceful Degradation │
│ ┌──────────────────────────────────────────┐ │
│ │ Return preset reply + retry prompt │ │
│ └──────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
Go Fallback Implementation
type FallbackChain struct {
primary *ModelEndpoint
secondary *ModelEndpoint
cache SemanticCache
timeout time.Duration
}
func (fc *FallbackChain) Generate(ctx context.Context, req *InferenceRequest) (string, error) {
ctx, cancel := context.WithTimeout(ctx, fc.timeout)
defer cancel()
result, err := fc.callModel(ctx, fc.primary, req)
if err == nil {
return result, nil
}
log.Warn("primary model failed, falling back", "error", err)
if fc.secondary != nil {
result, err = fc.callModel(ctx, fc.secondary, req)
if err == nil {
return result, nil
}
log.Warn("secondary model failed", "error", err)
}
if fc.cache != nil {
cached, ok := fc.cache.Get(req.Prompt)
if ok {
log.Info("cache hit on fallback")
return cached, nil
}
}
return "Service temporarily unavailable, please retry later.", ErrServiceUnavailable
}
Go Gateway Implementation and K8s Deployment
K8s Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-inference-gateway
namespace: ai-inference
spec:
replicas: 3
selector:
matchLabels:
app: ai-inference-gateway
template:
spec:
containers:
- name: gateway
image: myregistry/ai-inference-gateway:v1.0
ports:
- containerPort: 8080
resources:
requests:
cpu: "2"
memory: 2Gi
limits:
cpu: "4"
memory: 4Gi
env:
- name: PRIMARY_MODEL_URL
value: "http://vllm-qwen72b:8000/v1"
- name: SECONDARY_MODEL_URL
value: "http://vllm-qwen7b:8000/v1"
- name: REDIS_URL
value: "redis://redis:6379"
- name: ROUTING_STRATEGY
value: "latency_first"
- name: TOKEN_RATE_LIMIT
value: "100000"
---
apiVersion: v1
kind: Service
metadata:
name: ai-gateway-svc
namespace: ai-inference
spec:
selector:
app: ai-inference-gateway
ports:
- port: 8080
targetPort: 8080
type: ClusterIP
Prometheus Monitoring
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: ai-gateway-alerts
namespace: ai-inference
spec:
groups:
- name: ai-gateway
rules:
- alert: HighFallbackRate
expr: rate(gateway_fallback_total[5m]) / rate(gateway_requests_total[5m]) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "AI gateway fallback rate too high"
- alert: TokenRateLimitExceeded
expr: rate(gateway_rate_limit_exceeded_total[5m]) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "Token rate limit triggered frequently"
Summary and Further Reading
The AI inference gateway is the "front door" of LLM services, and the 4 core capabilities (routing, rate limiting, degradation, observability) are all indispensable. Token Rate Limiting is more precise than request frequency limiting, and the 4-level Fallback ensures the SLA baseline.
Key Design Takeaways:
- AI inference gateways must be Token-aware; traditional API gateways are insufficient
- 3 routing strategies: cost-first / latency-first / quality-first — choose by scenario
- Token Rate Limiting is the standard for production rate limiting
- 4-level Fallback: primary model → secondary model → cache → graceful degradation
- Go implementation + K8s deployment + Prometheus monitoring is the production standard
Related Reading:
- Cloud-Native AI Deployment Guide — K8s deployment for AI inference services
- LLM Inference Acceleration Benchmarks — Backend inference engine selection for gateways
- K8s Gateway API Migration Guide — K8s traffic management evolution
Authoritative References:
Try these browser-local tools — no sign-up required →
#AI推理网关#LLM API网关#模型路由策略#推理服务治理#AI网关限流#2026