Go遙測可觀測性實戰：用結構化日誌和指標構建生產級可觀測性的5個核心模式

2026年，Go語言的遙測生態已經從「可選」變成了「必選」。隨著微服務架構的深度普及，一個線上問題如果無法在5分鐘內定位，就意味著用戶流失和收入損失。Go 1.22引入的slog標準庫、OpenTelemetry Go SDK的成熟、以及各大雲廠商對OTLP協議的全面支援，讓Go服務的可觀測性終於有了統一的答案。但現實是：很多團隊仍然在用fmt.Println調試線上問題，日誌和指標割裂，追蹤鏈路斷裂，告警風暴不斷。本文將從5個核心模式出發，帶你構建真正能用的生產級可觀測性體系。

核心概念

概念	說明	關鍵包/工具
slog結構化日誌	Go 1.22+標準庫，支持鍵值對結構化輸出	log/slog
OpenTelemetry Metrics	統一指標採集標準，支持Counter/Gauge/Histogram	go.opentelemetry.io/otel/metric
分散式追蹤	跨服務請求鏈路追蹤，W3C TraceContext傳播	go.opentelemetry.io/otel/trace
Instrumentation中介軟體	自動化的HTTP/gRPC攔截埋點	go.opentelemetry.io/contrib/instrumentation
可觀測性儀表盤	Grafana/Prometheus整合的統一監控視圖	Grafana, Prometheus, Loki

問題分析：Go可觀測性的5大痛點

痛點1：日誌無結構，排查如大海撈針

傳統log.Printf輸出純文本，無法被機器解析，日誌平台無法建立索引，排查問題時只能靠肉眼搜索。

痛點2：指標與日誌割裂，無法關聯分析

指標顯示CPU飆升，但無法直接跳轉到對應時間段的日誌，兩個系統各自為政，問題定位效率極低。

痛點3：分散式追蹤鏈路斷裂

服務A調用服務B，但追蹤ID沒有正確傳播，導致鏈路在邊界處斷裂，無法看到完整的請求路徑。

痛點4：手動埋點代碼侵入嚴重

每個HTTP handler都要手動寫一堆埋點代碼，業務邏輯被可觀測性代碼淹沒，維護成本極高。

痛點5：生產環境告警風暴

缺乏合理的指標聚合和告警策略，一個服務抖動觸發幾十條告警，反而掩蓋了真正的問題。

核心模式1：slog結構化日誌與上下文傳遞

slog是Go 1.22引入的結構化日誌標準庫，它不僅支持鍵值對輸出，更重要的是支持Context傳遞，讓日誌自動攜帶請求級別的上下文信息。

package main

import (
	"context"
	"log/slog"
	"os"
	"time"
)

// RequestIDKey 用於從context中提取請求ID的鍵
type RequestIDKey struct{}

// Logger 封裝slog.Logger，提供上下文感知的日誌方法
type Logger struct {
	inner *slog.Logger
}

// NewLogger 創建帶有默認字段的Logger
func NewLogger(serviceName string) *Logger {
	handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: slog.LevelInfo,
	})
	logger := slog.New(handler).With(
		"service", serviceName,
		"pid", os.Getpid(),
	)
	return &Logger{inner: logger}
}

// WithContext 從context中提取請求信息並附加到日誌
func (l *Logger) WithContext(ctx context.Context) *slog.Logger {
	logger := l.inner
	if reqID, ok := ctx.Value(RequestIDKey{}).(string); ok {
		logger = logger.With("request_id", reqID)
	}
	if traceID := getTraceIDFromContext(ctx); traceID != "" {
		logger = logger.With("trace_id", traceID)
	}
	return logger
}

// InfoContext 記錄Info級別日誌，自動攜帶上下文
func (l *Logger) InfoContext(ctx context.Context, msg string, args ...any) {
	l.WithContext(ctx).InfoContext(ctx, msg, args...)
}

// ErrorContext 記錄Error級別日誌，自動攜帶上下文
func (l *Logger) ErrorContext(ctx context.Context, msg string, args ...any) {
	l.WithContext(ctx).ErrorContext(ctx, msg, args...)
}

// getTraceIDFromContext 從OpenTelemetry context中提取traceID
func getTraceIDFromContext(ctx context.Context) string {
	span := trace.SpanFromContext(ctx)
	if span.SpanContext().IsValid() {
		return span.SpanContext().TraceID().String()
	}
	return ""
}

// --- 使用示例 ---

// middlewareRequestID HTTP中介軟體：注入請求ID到context
func middlewareRequestID(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		reqID := r.Header.Get("X-Request-ID")
		if reqID == "" {
			reqID = generateUUID()
		}
		ctx := context.WithValue(r.Context(), RequestIDKey{}, reqID)
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

// handleGetUser 業務handler：使用上下文感知日誌
func handleGetUser(logger *Logger) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		ctx := r.Context()
		userID := r.PathValue("id")

		logger.InfoContext(ctx, "fetching user",
			"user_id", userID,
			"method", r.Method,
			"path", r.URL.Path,
		)

		user, err := fetchUserFromDB(ctx, userID)
		if err != nil {
			logger.ErrorContext(ctx, "failed to fetch user",
				"user_id", userID,
				"error", err.Error(),
				"duration_ms", time.Since(time.Now()).Milliseconds(),
			)
			http.Error(w, "internal error", http.StatusInternalServerError)
			return
		}

		logger.InfoContext(ctx, "user fetched successfully",
			"user_id", userID,
			"user_name", user.Name,
		)
		json.NewEncoder(w).Encode(user)
	}
}

關鍵要點：

• 使用JSONHandler輸出結構化JSON，便於日誌平台解析和索引
• 透過With()方法添加服務級默認字段，避免每條日誌重複寫
• 從Context中提取請求ID和TraceID，實現日誌與追蹤的自動關聯
• WithContext模式讓業務代碼無需關心日誌上下文傳遞細節

核心模式2：OpenTelemetry Metrics指標採集

OpenTelemetry Metrics提供了統一的指標採集標準，支持Counter、Gauge、Histogram三種指標類型，配合Prometheus匯出器，可以無縫對接現有監控體系。

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"go.opentelemetry.io/otel/exporters/prometheus"
	"go.opentelemetry.io/otel/metric"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

// MetricsProvider 封裝OpenTelemetry MeterProvider
type MetricsProvider struct {
	provider *sdkmetric.MeterProvider
	meter    metric.Meter
}

// NewMetricsProvider 創建MetricsProvider並註冊Prometheus匯出器
func NewMetricsProvider(serviceName string) (*MetricsProvider, error) {
	exporter, err := prometheus.New()
	if err != nil {
		return nil, fmt.Errorf("create prometheus exporter: %w", err)
	}

	provider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(exporter),
		sdkmetric.WithView(
			sdkmetric.NewView(
				sdkmetric.Instrument{Name: "http.server.duration"},
				sdkmetric.Stream{
					Aggregation: sdkmetric.AggregationExplicitBucketHistogram{
						Boundaries: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
					},
				},
			),
		),
	)

	meter := provider.Meter(
		serviceName,
		metric.WithInstrumentationVersion("1.0.0"),
	)

	return &MetricsProvider{
		provider: provider,
		meter:    meter,
	}, nil
}

// AppMetrics 應用級指標集合
type AppMetrics struct {
	httpRequestsTotal    metric.Int64Counter
	httpRequestDuration  metric.Float64Histogram
	activeConnections    metric.Int64UpDownCounter
	dbQueryDuration      metric.Float64Histogram
	businessOpsTotal     metric.Int64Counter
}

// NewAppMetrics 初始化所有應用指標
func NewAppMetrics(mp *MetricsProvider) (*AppMetrics, error) {
	m := mp.meter
	am := &AppMetrics{}

	var err error

	am.httpRequestsTotal, err = m.Int64Counter(
		"http.server.requests.total",
		metric.WithDescription("Total number of HTTP requests"),
		metric.WithUnit("{request}"),
	)
	if err != nil {
		return nil, err
	}

	am.httpRequestDuration, err = m.Float64Histogram(
		"http.server.duration",
		metric.WithDescription("HTTP request duration"),
		metric.WithUnit("s"),
	)
	if err != nil {
		return nil, err
	}

	am.activeConnections, err = m.Int64UpDownCounter(
		"http.server.connections.active",
		metric.WithDescription("Number of active connections"),
		metric.WithUnit("{connection}"),
	)
	if err != nil {
		return nil, err
	}

	am.dbQueryDuration, err = m.Float64Histogram(
		"db.query.duration",
		metric.WithDescription("Database query duration"),
		metric.WithUnit("s"),
	)
	if err != nil {
		return nil, err
	}

	am.businessOpsTotal, err = m.Int64Counter(
		"business.operations.total",
		metric.WithDescription("Total number of business operations"),
		metric.WithUnit("{operation}"),
	)
	if err != nil {
		return nil, err
	}

	return am, nil
}

// RecordHTTPRequest 記錄HTTP請求指標
func (am *AppMetrics) RecordHTTPRequest(ctx context.Context, method, path, status string, duration time.Duration) {
	attrs := metric.WithAttributes(
		attribute.String("http.method", method),
		attribute.String("http.route", path),
		attribute.Int("http.status_code", statusCodeToInt(status)),
	)

	am.httpRequestsTotal.Add(ctx, 1, attrs)
	am.httpRequestDuration.Record(ctx, duration.Seconds(), attrs)
}

// RecordDBQuery 記錄資料庫查詢指標
func (am *AppMetrics) RecordDBQuery(ctx context.Context, query string, duration time.Duration, err error) {
	attrs := metric.WithAttributes(
		attribute.String("db.query.name", query),
		attribute.Bool("db.query.error", err != nil),
	)
	am.dbQueryDuration.Record(ctx, duration.Seconds(), attrs)
}

// RecordBusinessOp 記錄業務操作指標
func (am *AppMetrics) RecordBusinessOp(ctx context.Context, op string, success bool) {
	attrs := metric.WithAttributes(
		attribute.String("operation.type", op),
		attribute.Bool("operation.success", success),
	)
	am.businessOpsTotal.Add(ctx, 1, attrs)
}

// --- 使用示例 ---

// metricsMiddleware HTTP中介軟體：自動採集請求指標
func metricsMiddleware(metrics *AppMetrics, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		metrics.activeConnections.Add(r.Context(), 1)

		rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
		next.ServeHTTP(rw, r)

		duration := time.Since(start)
		metrics.RecordHTTPRequest(
			r.Context(),
			r.Method,
			r.URL.Path,
			fmt.Sprintf("%d", rw.statusCode),
			duration,
		)
		metrics.activeConnections.Add(r.Context(), -1)
	})
}

// responseWriter 包裝http.ResponseWriter以捕獲狀態碼
type responseWriter struct {
	http.ResponseWriter
	statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
	rw.statusCode = code
	rw.ResponseWriter.WriteHeader(code)
}

關鍵要點：

• 使用ExplicitBucketHistogram自定義直方圖桶邊界，適配HTTP請求延遲分佈
• UpDownCounter適合追蹤活躍連接數等可增可減的指標
• 中介軟體自動採集指標，業務代碼零侵入
• 透過WithAttributes添加維度標籤，支持多維度的指標聚合和篩選

核心模式3：分散式追蹤與上下文傳播

分散式追蹤是可觀測性的第三根支柱。透過W3C TraceContext標準，Go服務可以自動在HTTP/gRPC調用間傳播追蹤上下文，實現跨服務的請求鏈路追蹤。

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

// TracingProvider 封裝OpenTelemetry TracerProvider
type TracingProvider struct {
	provider *sdktrace.TracerProvider
	tracer   trace.Tracer
}

// NewTracingProvider 創建TracingProvider並配置OTLP匯出
func NewTracingProvider(ctx context.Context, serviceName, otlpEndpoint string) (*TracingProvider, error) {
	exporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint(otlpEndpoint),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, fmt.Errorf("create OTLP exporter: %w", err)
	}

	provider := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceNameKey.String(serviceName),
			semconv.ServiceVersionKey.String("1.0.0"),
		)),
		sdktrace.WithSampler(sdktrace.ParentBased(
			sdktrace.TraceIDRatioBased(0.1), // 生產環境採樣10%
		)),
	)

	// 設置全局TracerProvider和傳播器
	otel.SetTracerProvider(provider)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	))

	tracer := provider.Tracer(
		serviceName,
		trace.WithInstrumentationVersion("1.0.0"),
	)

	return &TracingProvider{
		provider: provider,
		tracer:   tracer,
	}, nil
}

// Shutdown 優雅關閉TracerProvider
func (tp *TracingProvider) Shutdown(ctx context.Context) error {
	return tp.provider.Shutdown(ctx)
}

// SpanBuilder 構建Span的流暢API
type SpanBuilder struct {
	tracer  trace.Tracer
	name    string
	attrs   []attribute.KeyValue
	options []trace.SpanStartOption
}

// NewSpanBuilder 創建SpanBuilder
func (tp *TracingProvider) NewSpanBuilder(name string) *SpanBuilder {
	return &SpanBuilder{
		tracer: tp.tracer,
		name:   name,
	}
}

// WithAttr 添加屬性
func (sb *SpanBuilder) WithAttr(key string, value any) *SpanBuilder {
	switch v := value.(type) {
	case string:
		sb.attrs = append(sb.attrs, attribute.String(key, v))
	case int:
		sb.attrs = append(sb.attrs, attribute.Int(key, v))
	case bool:
		sb.attrs = append(sb.attrs, attribute.Bool(key, v))
	}
	return sb
}

// WithOption 添加SpanStartOption
func (sb *SpanBuilder) WithOption(opt trace.SpanStartOption) *SpanBuilder {
	sb.options = append(sb.options, opt)
	return sb
}

// Do 執行帶追蹤的函數
func (sb *SpanBuilder) Do(ctx context.Context, fn func(ctx context.Context) error) error {
	if len(sb.attrs) > 0 {
		sb.options = append(sb.options, trace.WithAttributes(sb.attrs...))
	}
	ctx, span := sb.tracer.Start(ctx, sb.name, sb.options...)
	defer span.End()

	if err := fn(ctx); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return err
	}

	span.SetStatus(codes.Ok, "")
	return nil
}

// --- 使用示例 ---

// tracingMiddleware HTTP中介軟體：自動創建根Span
func tracingMiddleware(tp *TracingProvider, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		propagator := otel.GetTextMapPropagator()
		ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))

		spanName := fmt.Sprintf("%s %s", r.Method, r.URL.Path)
		ctx, span := tp.tracer.Start(ctx, spanName,
			trace.WithAttributes(
				semconv.HTTPRequestMethodKey.String(r.Method),
				semconv.URLPathKey.String(r.URL.Path),
				semconv.UserAgentOriginalKey.String(r.UserAgent()),
			),
			trace.WithSpanKind(trace.SpanKindServer),
		)
		defer span.End()

		rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
		next.ServeHTTP(rw, r.WithContext(ctx))

		span.SetAttributes(semconv.HTTPResponseStatusCodeKey.Int(rw.statusCode))
		if rw.statusCode >= 400 {
			span.SetStatus(codes.Error, fmt.Sprintf("HTTP %d", rw.statusCode))
		}
	})
}

// callUserService 調用用戶服務的HTTP客戶端，自動傳播追蹤上下文
func callUserService(ctx context.Context, tp *TracingProvider, userID string) (*User, error) {
	var user User
	err := tp.NewSpanBuilder("call-user-service").
		WithAttr("user.id", userID).
		WithOption(trace.WithSpanKind(trace.SpanKindClient)).
		Do(ctx, func(ctx context.Context) error {
			req, err := http.NewRequestWithContext(ctx, http.MethodGet,
				fmt.Sprintf("http://user-service:8080/users/%s", userID), nil)
			if err != nil {
				return fmt.Errorf("create request: %w", err)
			}

			// 自動注入追蹤上下文到HTTP頭
			otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

			resp, err := http.DefaultClient.Do(req)
			if err != nil {
				return fmt.Errorf("do request: %w", err)
			}
			defer resp.Body.Close()

			if resp.StatusCode != http.StatusOK {
				return fmt.Errorf("unexpected status: %d", resp.StatusCode)
			}

			return json.NewDecoder(resp.Body).Decode(&user)
		})

	return &user, err
}

關鍵要點：

• 使用ParentBased採樣策略，根請求採樣10%，子請求跟隨父決策，確保鏈路完整
• TextMapPropagator自動在HTTP頭中注入/提取TraceContext，實現跨服務傳播
• SpanBuilder流暢API簡化Span創建，減少樣板代碼
• 客戶端調用時必須調用Inject，服務端中介軟體自動調用Extract

核心模式4：自定義Instrumentation中介軟體

OpenTelemetry Contrib提供了豐富的Instrumentation包，但生產環境往往需要自定義中介軟體來滿足特定需求，如業務指標採集、敏感信息過濾、自定義Span屬性等。

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
	"go.opentelemetry.io/otel/trace"
)

// InstrumentationMiddleware 統一的可觀測性中介軟體
type InstrumentationMiddleware struct {
	logger  *Logger
	metrics *AppMetrics
	tracer  trace.Tracer
}

// NewInstrumentationMiddleware 創建InstrumentationMiddleware
func NewInstrumentationMiddleware(
	logger *Logger,
	metrics *AppMetrics,
	tp *TracingProvider,
) *InstrumentationMiddleware {
	return &InstrumentationMiddleware{
		logger:  logger,
		metrics: metrics,
		tracer:  tp.tracer,
	}
}

// InstrumentHTTP 完整的HTTP可觀測性中介軟體
func (im *InstrumentationMiddleware) InstrumentHTTP(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		ctx := r.Context()

		// 1. 追蹤：創建服務端Span
		propagator := otel.GetTextMapPropagator()
		ctx = propagator.Extract(ctx, propagation.HeaderCarrier(r.Header))
		spanName := fmt.Sprintf("HTTP %s %s", r.Method, r.URL.Path)
		ctx, span := im.tracer.Start(ctx, spanName,
			trace.WithSpanKind(trace.SpanKindServer),
			trace.WithAttributes(
				attribute.String("http.method", r.Method),
				attribute.String("http.url", r.URL.String()),
				attribute.String("http.user_agent", r.UserAgent()),
				attribute.String("net.peer.ip", r.RemoteAddr),
			),
		)
		defer span.End()

		// 2. 指標：追蹤活躍連接
		im.metrics.activeConnections.Add(ctx, 1)
		defer im.metrics.activeConnections.Add(ctx, -1)

		// 3. 日誌：記錄請求開始
		im.logger.InfoContext(ctx, "request started",
			"method", r.Method,
			"path", r.URL.Path,
			"remote_addr", r.RemoteAddr,
		)

		// 4. 執行業務handler
		rw := &instrumentedWriter{
			ResponseWriter: w,
			statusCode:     http.StatusOK,
			bytesWritten:   0,
		}
		next.ServeHTTP(rw, r.WithContext(ctx))

		// 5. 記錄響應指標
		duration := time.Since(start)
		im.metrics.RecordHTTPRequest(ctx, r.Method, r.URL.Path,
			fmt.Sprintf("%d", rw.statusCode), duration)

		// 6. 追蹤：記錄響應信息
		span.SetAttributes(
			attribute.Int("http.status_code", rw.statusCode),
			attribute.Int("http.response_size", rw.bytesWritten),
			attribute.Float64("http.duration_ms", float64(duration.Milliseconds())),
		)
		if rw.statusCode >= 500 {
			span.SetStatus(codes.Error, fmt.Sprintf("HTTP %d", rw.statusCode))
		}

		// 7. 日誌：記錄請求完成
		im.logger.InfoContext(ctx, "request completed",
			"status_code", rw.statusCode,
			"duration_ms", duration.Milliseconds(),
			"bytes_written", rw.bytesWritten,
		)
	})
}

// instrumentedWriter 包裝ResponseWriter以捕獲響應信息
type instrumentedWriter struct {
	http.ResponseWriter
	statusCode   int
	bytesWritten int
}

func (iw *instrumentedWriter) WriteHeader(code int) {
	iw.statusCode = code
	iw.ResponseWriter.WriteHeader(code)
}

func (iw *instrumentedWriter) Write(b []byte) (int, error) {
	n, err := iw.ResponseWriter.Write(b)
	iw.bytesWritten += n
	return n, err
}

// InstrumentDB 資料庫查詢可觀測性裝飾器
func (im *InstrumentationMiddleware) InstrumentDB(
	ctx context.Context,
	queryName string,
	fn func(ctx context.Context) error,
) error {
	start := time.Now()
	ctx, span := im.tracer.Start(ctx, fmt.Sprintf("DB %s", queryName),
		trace.WithSpanKind(trace.SpanKindClient),
		trace.WithAttributes(
			attribute.String("db.query.name", queryName),
		),
	)
	defer span.End()

	err := fn(ctx)
	duration := time.Since(start)

	im.metrics.RecordDBQuery(ctx, queryName, duration, err)

	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		im.logger.ErrorContext(ctx, "db query failed",
			"query", queryName,
			"error", err.Error(),
			"duration_ms", duration.Milliseconds(),
		)
	} else {
		im.logger.InfoContext(ctx, "db query completed",
			"query", queryName,
			"duration_ms", duration.Milliseconds(),
		)
	}

	return err
}

// InstrumentGRPC gRPC可觀測性攔截器（服務端）
func (im *InstrumentationMiddleware) InstrumentGRPC(
	ctx context.Context,
	req interface{},
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (interface{}, error) {
	start := time.Now()

	ctx, span := im.tracer.Start(ctx, info.FullMethod,
		trace.WithSpanKind(trace.SpanKindServer),
	)
	defer span.End()

	resp, err := handler(ctx, req)
	duration := time.Since(start)

	im.metrics.RecordHTTPRequest(ctx, "gRPC", info.FullMethod,
		statusFromError(err), duration)

	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		im.logger.ErrorContext(ctx, "grpc call failed",
			"method", info.FullMethod,
			"error", err.Error(),
			"duration_ms", duration.Milliseconds(),
		)
	} else {
		im.logger.InfoContext(ctx, "grpc call completed",
			"method", info.FullMethod,
			"duration_ms", duration.Milliseconds(),
		)
	}

	return resp, err
}

關鍵要點：

• 統一中介軟體同時處理日誌、指標、追蹤三大支柱，確保數據一致性
• instrumentedWriter捕獲響應狀態碼和位元組數，豐富追蹤和指標維度
• InstrumentDB裝飾器為資料庫查詢添加可觀測性，無需修改業務代碼
• gRPC攔截器與HTTP中介軟體共享同一套可觀測性基礎設施

核心模式5：生產級可觀測性儀表盤整合

將日誌、指標、追蹤三大支柱統一整合到Grafana儀表盤，實現從告警到日誌到追蹤的一鍵跳轉，是生產級可觀測性的最終目標。

package main

import (
	"context"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"go.opentelemetry.io/otel/exporters/prometheus"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// ObservabilityStack 生產級可觀測性棧
type ObservabilityStack struct {
	logger          *Logger
	metricsProvider *MetricsProvider
	tracingProvider *TracingProvider
	middleware      *InstrumentationMiddleware
	metrics         *AppMetrics
	shutdownFuncs   []func(ctx context.Context) error
}

// Config 可觀測性棧配置
type Config struct {
	ServiceName    string
	OTLPEndpoint   string
	MetricsPath    string
	LogLevel       string
	SamplingRatio  float64
}

// NewObservabilityStack 初始化完整的可觀測性棧
func NewObservabilityStack(ctx context.Context, cfg Config) (*ObservabilityStack, error) {
	stack := &ObservabilityStack{}

	// 1. 初始化結構化日誌
	stack.logger = NewLogger(cfg.ServiceName)

	// 2. 初始化指標採集
	promExporter, err := prometheus.New()
	if err != nil {
		return nil, fmt.Errorf("create prometheus exporter: %w", err)
	}

	metricProvider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(promExporter),
	)
	stack.metricsProvider = &MetricsProvider{
		provider: metricProvider,
		meter:    metricProvider.Meter(cfg.ServiceName),
	}
	stack.shutdownFuncs = append(stack.shutdownFuncs, metricProvider.Shutdown)

	// 3. 初始化分散式追蹤
	otlpExporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint(cfg.OTLPEndpoint),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, fmt.Errorf("create OTLP exporter: %w", err)
	}

	traceProvider := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(otlpExporter),
		sdktrace.WithSampler(sdktrace.ParentBased(
			sdktrace.TraceIDRatioBased(cfg.SamplingRatio),
		)),
	)
	stack.tracingProvider = &TracingProvider{
		provider: traceProvider,
		tracer:   traceProvider.Tracer(cfg.ServiceName),
	}
	stack.shutdownFuncs = append(stack.shutdownFuncs, traceProvider.Shutdown)

	// 4. 初始化應用指標
	stack.metrics, err = NewAppMetrics(stack.metricsProvider)
	if err != nil {
		return nil, fmt.Errorf("create app metrics: %w", err)
	}

	// 5. 初始化統一中介軟體
	stack.middleware = NewInstrumentationMiddleware(
		stack.logger,
		stack.metrics,
		stack.tracingProvider,
	)

	return stack, nil
}

// Shutdown 優雅關閉所有可觀測性組件
func (s *ObservabilityStack) Shutdown(ctx context.Context) error {
	var firstErr error
	for _, fn := range s.shutdownFuncs {
		if err := fn(ctx); err != nil && firstErr == nil {
			firstErr = err
		}
	}
	return firstErr
}

// --- 完整啟動示例 ---

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// 初始化可觀測性棧
	stack, err := NewObservabilityStack(ctx, Config{
		ServiceName:   "user-service",
		OTLPEndpoint:  "otel-collector:4317",
		MetricsPath:   "/metrics",
		LogLevel:      "info",
		SamplingRatio: 0.1,
	})
	if err != nil {
		fmt.Fprintf(os.Stderr, "init observability: %v\n", err)
		os.Exit(1)
	}
	defer stack.Shutdown(context.Background())

	// 創建HTTP路由
	mux := http.NewServeMux()

	// 註冊業務路由（帶可觀測性中介軟體）
	mux.HandleFunc("GET /users/{id}", handleGetUser(stack.logger))
	mux.HandleFunc("POST /users", handleCreateUser(stack.logger, stack.metrics))

	// 註冊Prometheus指標端點
	mux.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
		promHandler := prometheus.Handler()
		promHandler.ServeHTTP(w, r)
	})

	// 健康檢查
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ok"))
	})

	// 應用可觀測性中介軟體
	handler := stack.middleware.InstrumentHTTP(mux)
	handler = middlewareRequestID(handler)

	// 啟動HTTP伺服器
	server := &http.Server{
		Addr:         ":8080",
		Handler:      handler,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 30 * time.Second,
		IdleTimeout:  60 * time.Second,
	}

	// 優雅關閉
	go func() {
		sigCh := make(chan os.Signal, 1)
		signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
		<-sigCh

		shutdownCtx, shutdownCancel := context.WithTimeout(
			context.Background(), 10*time.Second,
		)
		defer shutdownCancel()

		stack.logger.InfoContext(ctx, "shutting down server...")
		server.Shutdown(shutdownCtx)
		stack.Shutdown(shutdownCtx)
	}()

	stack.logger.InfoContext(ctx, "server starting",
		"addr", server.Addr,
		"service", "user-service",
	)

	if err := server.ListenAndServe(); err != http.ErrServerClosed {
		stack.logger.ErrorContext(ctx, "server error", "error", err.Error())
		os.Exit(1)
	}
}

Grafana儀表盤配置要點：

# docker-compose.yaml - 完整可觀測性棧
version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

  prometheus:
    image: prom/prometheus:v2.50.0
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  loki:
    image: grafana/loki:2.9.4
    ports:
      - "3100:3100"

  tempo:
    image: grafana/tempo:2.3.1
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml
    ports:
      - "3200:3200"

  grafana:
    image: grafana/grafana:10.3.3
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
      - ./grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
    ports:
      - "3000:3000"

關鍵要點：

• ObservabilityStack統一管理三大支柱的生命週期，確保優雅關閉
• Prometheus + Loki + Tempo + Grafana構成完整的可觀測性後端
• Grafana資料來源配置中啟用TraceID跳轉，實現從日誌到追蹤的一鍵關聯
• 採樣率透過配置控制，生產環境建議10%採樣，異常鏈路自動100%採樣

常見陷阱

陷阱1：slog的With()方法不會複製屬性

// ❌ 錯誤：With()返回新Logger，原Logger不受影響
logger := slog.Default().With("request_id", "abc123")
logger.Info("message") // 有 request_id
slog.Info("message")   // 沒有 request_id！

// ✅ 正確：始終使用With()返回的新Logger
baseLogger := slog.Default()
requestLogger := baseLogger.With("request_id", "abc123")
requestLogger.Info("message") // 有 request_id

陷阱2：忘記在HTTP客戶端傳播追蹤上下文

// ❌ 錯誤：直接使用http.NewRequest，追蹤鏈路斷裂
req, _ := http.NewRequest("GET", "http://user-service/users/123", nil)
resp, _ := http.DefaultClient.Do(req)

// ✅ 正確：使用NewRequestWithContext並注入傳播器
req, _ := http.NewRequestWithContext(ctx, "GET", "http://user-service/users/123", nil)
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
resp, _ := http.DefaultClient.Do(req)

陷阱3：Histogram桶邊界設置不當

// ❌ 錯誤：使用默認桶邊界，無法區分正常和異常延遲
am.httpRequestDuration, _ = m.Float64Histogram("http.server.duration")

// ✅ 正確：自定義桶邊界，適配HTTP請求延遲分佈
am.httpRequestDuration, _ = m.Float64Histogram(
	"http.server.duration",
	metric.WithUnit("s"),
)
// 在MeterProvider中配置View自定義桶邊界
sdkmetric.WithView(
	sdkmetric.NewView(
		sdkmetric.Instrument{Name: "http.server.duration"},
		sdkmetric.Stream{
			Aggregation: sdkmetric.AggregationExplicitBucketHistogram{
				Boundaries: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
			},
		},
	),
)

陷阱4：在熱路徑中創建大量屬性

// ❌ 錯誤：每次請求都創建新的attribute slice
span.SetAttributes(
	attribute.String("user.id", userID),
	attribute.String("request.path", path),
	attribute.String("user.agent", userAgent),
)

// ✅ 正確：預創建常用屬性，減少GC壓力
var (
	attrUserID    = attribute.Key("user.id")
	attrReqPath   = attribute.Key("request.path")
	attrUserAgent = attribute.Key("user.agent")
)

span.SetAttributes(
	attrUserID.String(userID),
	attrReqPath.String(path),
	attrUserAgent.String(userAgent),
)

陷阱5：TracerProvider未優雅關閉導致Span丟失

// ❌ 錯誤：直接退出進程，BatchSpanProcessor中的Span丟失
func main() {
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	// ... 程式運行 ...
	// 進程退出，未Flush的Span丟失！
}

// ✅ 正確：在退出前調用Shutdown確保所有Span被匯出
func main() {
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	defer func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		tp.Shutdown(ctx) // 確保所有Span被Flush
	}()
	// ... 程式運行 ...
}

錯誤排查

錯誤現象	可能原因	排查方法	解決方案
日誌中無trace_id	未從context提取trace信息	檢查WithContext是否正確調用	確保中介軟體在日誌之前初始化
Prometheus無指標數據	MeterProvider未註冊Reader	檢查/metrics端點是否正常	確認prometheus.New()已傳入WithReader
追蹤鏈路在服務邊界斷裂	未傳播TraceContext	檢查HTTP客戶端是否調用Inject	客戶端請求前注入傳播器
Span數據延遲出現在Jaeger	BatchSpanProcessor批量發送	檢查Shutdown是否被調用	優雅關閉時調用tp.Shutdown(ctx)
指標值異常偏高	UpDownCounter未正確遞減	檢查Add(-1)是否在所有路徑執行	使用defer確保遞減
Grafana無法查詢Loki日誌	Loki資料來源配置錯誤	檢查Loki URL和Label配置	確認{service="xxx"}標籤匹配
採樣率設置後仍全量採集	採樣器配置被覆蓋	檢查是否有AlwaysSample覆蓋	使用ParentBased包裝採樣器
gRPC追蹤Span缺失	未註冊攔截器	檢查gRPC Server是否添加攔截器	使用otelgrpc.ServerInterceptor
日誌輸出為純文本非JSON	使用了默認的TextHandler	檢查slog.New的Handler參數	替換為JSONHandler
OTLP匯出連接超時	Collector未啟動或端口錯誤	檢查Collector狀態和端口	確認4317端口可達

進階優化

1. 自適應採樣策略

根據請求特徵動態調整採樣率：錯誤請求100%採樣，慢請求50%採樣，正常請求1%採樣。透過ShouldSample介面實現自定義採樣邏輯。

2. 日誌與追蹤自動關聯

透過slog.Handler自定義實現，在每條日誌中自動注入當前Span的TraceID和SpanID，實現日誌到追蹤的一鍵跳轉，無需手動傳遞。

3. 指標基數控制

高基數標籤（如user_id）會導致Prometheus記憶體爆炸。使用attribute.KeyValueFilter或CardinalityLimit配置限制標籤基數，防止指標爆炸。

4. Exemplar關聯指標與追蹤

在Prometheus指標中嵌入Exemplar（包含TraceID的樣本），實現從指標圖表直接跳轉到追蹤詳情。OpenTelemetry SDK已原生支持Exemplar。

5. 多集群聯邦監控

對於多集群部署，使用Prometheus Federation + Thanos實現全局指標視圖，Grafana配置多資料來源實現跨集群可觀測性。

對比

維度	slog + OTel	Zap + Prometheus	Logrus + Jaeger
日誌結構化	✅ 標準庫原生支持	✅ 第三方庫	⚠️ 需要Hook
指標採集	✅ OTel統一SDK	✅ 原生Prometheus	❌ 需額外整合
分散式追蹤	✅ OTel原生支持	❌ 需額外整合	✅ Jaeger客戶端
上下文傳遞	✅ Context原生整合	⚠️ 需手動傳遞	❌ 不支持
採樣控制	✅ 內置多種採樣器	❌ 不適用	⚠️ 有限支持
標準兼容性	✅ W3C/CNCF標準	⚠️ Prometheus專屬	⚠️ Jaeger專屬
維護成本	✅ 標準庫+CNCF維護	⚠️ 社群維護	❌ 已停止維護
學習曲線	⚠️ OTel概念較多	✅ 簡單直接	✅ 簡單

總結

2026年的Go可觀測性，slog + OpenTelemetry已經成為事實標準。不要再用fmt.Println調試生產問題，不要讓日誌和指標割裂，不要讓追蹤鏈路在服務邊界斷裂。從結構化日誌開始，逐步接入指標和追蹤，最終構建三大支柱統一的可觀測性體系——這才是Go服務生產化的正確姿勢。

線上工具推薦

• JSON格式化 — 格式化slog輸出的JSON日誌，快速定位問題字段
• cURL轉代碼 — 將cURL命令轉為Go HTTP客戶端代碼，快速驗證追蹤上下文傳播
• 哈希計算 — 計算請求ID的哈希值，用於日誌脫敏和採樣分桶