Go Telemetry Observability in Practice: 5 Core Patterns for Building Production-Grade Observability with Structured Logging and Metrics

技术架构

In 2026, Go's telemetry ecosystem has evolved from "optional" to "essential." As microservice architectures become deeply entrenched, an online issue that can't be diagnosed within 5 minutes means user churn and revenue loss. Go 1.22's slog standard library, the maturity of the OpenTelemetry Go SDK, and full OTLP protocol support from major cloud providers have finally given Go services a unified answer for observability. Yet reality tells a different story: many teams still debug production issues with fmt.Println, logs and metrics remain siloed, trace links break at service boundaries, and alert storms never cease. This article walks you through 5 core patterns to build a truly production-grade observability system.

Core Concepts

Concept Description Key Package/Tool
slog Structured Logging Go 1.22+ standard library with key-value structured output log/slog
OpenTelemetry Metrics Unified metric collection standard supporting Counter/Gauge/Histogram go.opentelemetry.io/otel/metric
Distributed Tracing Cross-service request tracing with W3C TraceContext propagation go.opentelemetry.io/otel/trace
Instrumentation Middleware Automated HTTP/gRPC interception and instrumentation go.opentelemetry.io/contrib/instrumentation
Observability Dashboard Unified monitoring view integrated with Grafana/Prometheus Grafana, Prometheus, Loki

Problem Analysis: 5 Pain Points of Go Observability

Pain Point 1: Unstructured Logs Make Debugging Like Finding a Needle in a Haystack

Traditional log.Printf outputs plain text that machines can't parse, log platforms can't index, and debugging requires manual visual searching.

Pain Point 2: Siloed Metrics and Logs Prevent Correlated Analysis

Metrics show CPU spikes, but you can't jump directly to the corresponding log timeframe. Two separate systems working in isolation make problem identification extremely inefficient.

Pain Point 3: Distributed Trace Links Break at Service Boundaries

Service A calls Service B, but the trace ID isn't properly propagated, causing the trace to break at the boundary. You can't see the complete request path.

Pain Point 4: Manual Instrumentation Causes Severe Code Intrusion

Every HTTP handler requires a pile of manual instrumentation code. Business logic gets buried under observability code, making maintenance extremely costly.

Pain Point 5: Production Alert Storms

Without proper metric aggregation and alerting strategies, a single service hiccup triggers dozens of alerts, actually masking the real problem.

Core Pattern 1: slog Structured Logging with Context Propagation

slog is Go 1.22's structured logging standard library. It not only supports key-value output but, more importantly, supports Context propagation, allowing logs to automatically carry request-level context information.

package main

import (
	"context"
	"log/slog"
	"os"
	"time"
)

// RequestIDKey is the key for extracting request ID from context
type RequestIDKey struct{}

// Logger wraps slog.Logger with context-aware logging methods
type Logger struct {
	inner *slog.Logger
}

// NewLogger creates a Logger with default fields
func NewLogger(serviceName string) *Logger {
	handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: slog.LevelInfo,
	})
	logger := slog.New(handler).With(
		"service", serviceName,
		"pid", os.Getpid(),
	)
	return &Logger{inner: logger}
}

// WithContext extracts request info from context and attaches it to the logger
func (l *Logger) WithContext(ctx context.Context) *slog.Logger {
	logger := l.inner
	if reqID, ok := ctx.Value(RequestIDKey{}).(string); ok {
		logger = logger.With("request_id", reqID)
	}
	if traceID := getTraceIDFromContext(ctx); traceID != "" {
		logger = logger.With("trace_id", traceID)
	}
	return logger
}

// InfoContext logs at Info level, automatically carrying context
func (l *Logger) InfoContext(ctx context.Context, msg string, args ...any) {
	l.WithContext(ctx).InfoContext(ctx, msg, args...)
}

// ErrorContext logs at Error level, automatically carrying context
func (l *Logger) ErrorContext(ctx context.Context, msg string, args ...any) {
	l.WithContext(ctx).ErrorContext(ctx, msg, args...)
}

// getTraceIDFromContext extracts traceID from OpenTelemetry context
func getTraceIDFromContext(ctx context.Context) string {
	span := trace.SpanFromContext(ctx)
	if span.SpanContext().IsValid() {
		return span.SpanContext().TraceID().String()
	}
	return ""
}

// --- Usage Example ---

// middlewareRequestID HTTP middleware: injects request ID into context
func middlewareRequestID(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		reqID := r.Header.Get("X-Request-ID")
		if reqID == "" {
			reqID = generateUUID()
		}
		ctx := context.WithValue(r.Context(), RequestIDKey{}, reqID)
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

// handleGetUser Business handler: uses context-aware logging
func handleGetUser(logger *Logger) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		ctx := r.Context()
		userID := r.PathValue("id")

		logger.InfoContext(ctx, "fetching user",
			"user_id", userID,
			"method", r.Method,
			"path", r.URL.Path,
		)

		user, err := fetchUserFromDB(ctx, userID)
		if err != nil {
			logger.ErrorContext(ctx, "failed to fetch user",
				"user_id", userID,
				"error", err.Error(),
				"duration_ms", time.Since(time.Now()).Milliseconds(),
			)
			http.Error(w, "internal error", http.StatusInternalServerError)
			return
		}

		logger.InfoContext(ctx, "user fetched successfully",
			"user_id", userID,
			"user_name", user.Name,
		)
		json.NewEncoder(w).Encode(user)
	}
}

Key Takeaways:

  • Use JSONHandler for structured JSON output, enabling log platform parsing and indexing
  • Add service-level default fields via With(), avoiding repetition in every log entry
  • Extract Request ID and Trace ID from Context for automatic log-trace correlation
  • The WithContext pattern lets business code ignore logging context propagation details

Core Pattern 2: OpenTelemetry Metrics Collection

OpenTelemetry Metrics provides a unified metric collection standard supporting Counter, Gauge, and Histogram instrument types. Combined with the Prometheus exporter, it seamlessly integrates with existing monitoring systems.

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"go.opentelemetry.io/otel/exporters/prometheus"
	"go.opentelemetry.io/otel/metric"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

// MetricsProvider wraps OpenTelemetry MeterProvider
type MetricsProvider struct {
	provider *sdkmetric.MeterProvider
	meter    metric.Meter
}

// NewMetricsProvider creates MetricsProvider and registers Prometheus exporter
func NewMetricsProvider(serviceName string) (*MetricsProvider, error) {
	exporter, err := prometheus.New()
	if err != nil {
		return nil, fmt.Errorf("create prometheus exporter: %w", err)
	}

	provider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(exporter),
		sdkmetric.WithView(
			sdkmetric.NewView(
				sdkmetric.Instrument{Name: "http.server.duration"},
				sdkmetric.Stream{
					Aggregation: sdkmetric.AggregationExplicitBucketHistogram{
						Boundaries: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
					},
				},
			),
		),
	)

	meter := provider.Meter(
		serviceName,
		metric.WithInstrumentationVersion("1.0.0"),
	)

	return &MetricsProvider{
		provider: provider,
		meter:    meter,
	}, nil
}

// AppMetrics application-level metrics collection
type AppMetrics struct {
	httpRequestsTotal    metric.Int64Counter
	httpRequestDuration  metric.Float64Histogram
	activeConnections    metric.Int64UpDownCounter
	dbQueryDuration      metric.Float64Histogram
	businessOpsTotal     metric.Int64Counter
}

// NewAppMetrics initializes all application metrics
func NewAppMetrics(mp *MetricsProvider) (*AppMetrics, error) {
	m := mp.meter
	am := &AppMetrics{}

	var err error

	am.httpRequestsTotal, err = m.Int64Counter(
		"http.server.requests.total",
		metric.WithDescription("Total number of HTTP requests"),
		metric.WithUnit("{request}"),
	)
	if err != nil {
		return nil, err
	}

	am.httpRequestDuration, err = m.Float64Histogram(
		"http.server.duration",
		metric.WithDescription("HTTP request duration"),
		metric.WithUnit("s"),
	)
	if err != nil {
		return nil, err
	}

	am.activeConnections, err = m.Int64UpDownCounter(
		"http.server.connections.active",
		metric.WithDescription("Number of active connections"),
		metric.WithUnit("{connection}"),
	)
	if err != nil {
		return nil, err
	}

	am.dbQueryDuration, err = m.Float64Histogram(
		"db.query.duration",
		metric.WithDescription("Database query duration"),
		metric.WithUnit("s"),
	)
	if err != nil {
		return nil, err
	}

	am.businessOpsTotal, err = m.Int64Counter(
		"business.operations.total",
		metric.WithDescription("Total number of business operations"),
		metric.WithUnit("{operation}"),
	)
	if err != nil {
		return nil, err
	}

	return am, nil
}

// RecordHTTPRequest records HTTP request metrics
func (am *AppMetrics) RecordHTTPRequest(ctx context.Context, method, path, status string, duration time.Duration) {
	attrs := metric.WithAttributes(
		attribute.String("http.method", method),
		attribute.String("http.route", path),
		attribute.Int("http.status_code", statusCodeToInt(status)),
	)

	am.httpRequestsTotal.Add(ctx, 1, attrs)
	am.httpRequestDuration.Record(ctx, duration.Seconds(), attrs)
}

// RecordDBQuery records database query metrics
func (am *AppMetrics) RecordDBQuery(ctx context.Context, query string, duration time.Duration, err error) {
	attrs := metric.WithAttributes(
		attribute.String("db.query.name", query),
		attribute.Bool("db.query.error", err != nil),
	)
	am.dbQueryDuration.Record(ctx, duration.Seconds(), attrs)
}

// RecordBusinessOp records business operation metrics
func (am *AppMetrics) RecordBusinessOp(ctx context.Context, op string, success bool) {
	attrs := metric.WithAttributes(
		attribute.String("operation.type", op),
		attribute.Bool("operation.success", success),
	)
	am.businessOpsTotal.Add(ctx, 1, attrs)
}

// --- Usage Example ---

// metricsMiddleware HTTP middleware: automatically collects request metrics
func metricsMiddleware(metrics *AppMetrics, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		metrics.activeConnections.Add(r.Context(), 1)

		rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
		next.ServeHTTP(rw, r)

		duration := time.Since(start)
		metrics.RecordHTTPRequest(
			r.Context(),
			r.Method,
			r.URL.Path,
			fmt.Sprintf("%d", rw.statusCode),
			duration,
		)
		metrics.activeConnections.Add(r.Context(), -1)
	})
}

// responseWriter wraps http.ResponseWriter to capture status code
type responseWriter struct {
	http.ResponseWriter
	statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
	rw.statusCode = code
	rw.ResponseWriter.WriteHeader(code)
}

Key Takeaways:

  • Use ExplicitBucketHistogram with custom bucket boundaries to match HTTP request latency distributions
  • UpDownCounter is ideal for tracking metrics that can increase and decrease, like active connections
  • Middleware automatically collects metrics with zero business code intrusion
  • Add dimensional labels via WithAttributes for multi-dimensional metric aggregation and filtering

Core Pattern 3: Distributed Tracing and Context Propagation

Distributed tracing is the third pillar of observability. Through the W3C TraceContext standard, Go services can automatically propagate tracing context across HTTP/gRPC calls, enabling cross-service request chain tracing.

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

// TracingProvider wraps OpenTelemetry TracerProvider
type TracingProvider struct {
	provider *sdktrace.TracerProvider
	tracer   trace.Tracer
}

// NewTracingProvider creates TracingProvider with OTLP export configuration
func NewTracingProvider(ctx context.Context, serviceName, otlpEndpoint string) (*TracingProvider, error) {
	exporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint(otlpEndpoint),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, fmt.Errorf("create OTLP exporter: %w", err)
	}

	provider := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceNameKey.String(serviceName),
			semconv.ServiceVersionKey.String("1.0.0"),
		)),
		sdktrace.WithSampler(sdktrace.ParentBased(
			sdktrace.TraceIDRatioBased(0.1), // 10% sampling in production
		)),
	)

	// Set global TracerProvider and propagator
	otel.SetTracerProvider(provider)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	))

	tracer := provider.Tracer(
		serviceName,
		trace.WithInstrumentationVersion("1.0.0"),
	)

	return &TracingProvider{
		provider: provider,
		tracer:   tracer,
	}, nil
}

// Shutdown gracefully shuts down the TracerProvider
func (tp *TracingProvider) Shutdown(ctx context.Context) error {
	return tp.provider.Shutdown(ctx)
}

// SpanBuilder fluent API for building Spans
type SpanBuilder struct {
	tracer  trace.Tracer
	name    string
	attrs   []attribute.KeyValue
	options []trace.SpanStartOption
}

// NewSpanBuilder creates a SpanBuilder
func (tp *TracingProvider) NewSpanBuilder(name string) *SpanBuilder {
	return &SpanBuilder{
		tracer: tp.tracer,
		name:   name,
	}
}

// WithAttr adds an attribute
func (sb *SpanBuilder) WithAttr(key string, value any) *SpanBuilder {
	switch v := value.(type) {
	case string:
		sb.attrs = append(sb.attrs, attribute.String(key, v))
	case int:
		sb.attrs = append(sb.attrs, attribute.Int(key, v))
	case bool:
		sb.attrs = append(sb.attrs, attribute.Bool(key, v))
	}
	return sb
}

// WithOption adds a SpanStartOption
func (sb *SpanBuilder) WithOption(opt trace.SpanStartOption) *SpanBuilder {
	sb.options = append(sb.options, opt)
	return sb
}

// Do executes a function with tracing
func (sb *SpanBuilder) Do(ctx context.Context, fn func(ctx context.Context) error) error {
	if len(sb.attrs) > 0 {
		sb.options = append(sb.options, trace.WithAttributes(sb.attrs...))
	}
	ctx, span := sb.tracer.Start(ctx, sb.name, sb.options...)
	defer span.End()

	if err := fn(ctx); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return err
	}

	span.SetStatus(codes.Ok, "")
	return nil
}

// --- Usage Example ---

// tracingMiddleware HTTP middleware: automatically creates root Span
func tracingMiddleware(tp *TracingProvider, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		propagator := otel.GetTextMapPropagator()
		ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))

		spanName := fmt.Sprintf("%s %s", r.Method, r.URL.Path)
		ctx, span := tp.tracer.Start(ctx, spanName,
			trace.WithAttributes(
				semconv.HTTPRequestMethodKey.String(r.Method),
				semconv.URLPathKey.String(r.URL.Path),
				semconv.UserAgentOriginalKey.String(r.UserAgent()),
			),
			trace.WithSpanKind(trace.SpanKindServer),
		)
		defer span.End()

		rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
		next.ServeHTTP(rw, r.WithContext(ctx))

		span.SetAttributes(semconv.HTTPResponseStatusCodeKey.Int(rw.statusCode))
		if rw.statusCode >= 400 {
			span.SetStatus(codes.Error, fmt.Sprintf("HTTP %d", rw.statusCode))
		}
	})
}

// callUserService HTTP client calling user service, automatically propagating trace context
func callUserService(ctx context.Context, tp *TracingProvider, userID string) (*User, error) {
	var user User
	err := tp.NewSpanBuilder("call-user-service").
		WithAttr("user.id", userID).
		WithOption(trace.WithSpanKind(trace.SpanKindClient)).
		Do(ctx, func(ctx context.Context) error {
			req, err := http.NewRequestWithContext(ctx, http.MethodGet,
				fmt.Sprintf("http://user-service:8080/users/%s", userID), nil)
			if err != nil {
				return fmt.Errorf("create request: %w", err)
			}

			// Automatically inject trace context into HTTP headers
			otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

			resp, err := http.DefaultClient.Do(req)
			if err != nil {
				return fmt.Errorf("do request: %w", err)
			}
			defer resp.Body.Close()

			if resp.StatusCode != http.StatusOK {
				return fmt.Errorf("unexpected status: %d", resp.StatusCode)
			}

			return json.NewDecoder(resp.Body).Decode(&user)
		})

	return &user, err
}

Key Takeaways:

  • Use ParentBased sampling strategy: 10% root request sampling with child requests following parent decisions, ensuring complete trace chains
  • TextMapPropagator automatically injects/extracts TraceContext in HTTP headers for cross-service propagation
  • SpanBuilder fluent API simplifies Span creation and reduces boilerplate
  • Client calls must invoke Inject; server middleware automatically calls Extract

Core Pattern 4: Custom Instrumentation Middleware

OpenTelemetry Contrib provides rich instrumentation packages, but production environments often require custom middleware to meet specific needs such as business metric collection, sensitive information filtering, and custom Span attributes.

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
	"go.opentelemetry.io/otel/trace"
)

// InstrumentationMiddleware unified observability middleware
type InstrumentationMiddleware struct {
	logger  *Logger
	metrics *AppMetrics
	tracer  trace.Tracer
}

// NewInstrumentationMiddleware creates InstrumentationMiddleware
func NewInstrumentationMiddleware(
	logger *Logger,
	metrics *AppMetrics,
	tp *TracingProvider,
) *InstrumentationMiddleware {
	return &InstrumentationMiddleware{
		logger:  logger,
		metrics: metrics,
		tracer:  tp.tracer,
	}
}

// InstrumentHTTP complete HTTP observability middleware
func (im *InstrumentationMiddleware) InstrumentHTTP(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		ctx := r.Context()

		// 1. Tracing: create server-side Span
		propagator := otel.GetTextMapPropagator()
		ctx = propagator.Extract(ctx, propagation.HeaderCarrier(r.Header))
		spanName := fmt.Sprintf("HTTP %s %s", r.Method, r.URL.Path)
		ctx, span := im.tracer.Start(ctx, spanName,
			trace.WithSpanKind(trace.SpanKindServer),
			trace.WithAttributes(
				attribute.String("http.method", r.Method),
				attribute.String("http.url", r.URL.String()),
				attribute.String("http.user_agent", r.UserAgent()),
				attribute.String("net.peer.ip", r.RemoteAddr),
			),
		)
		defer span.End()

		// 2. Metrics: track active connections
		im.metrics.activeConnections.Add(ctx, 1)
		defer im.metrics.activeConnections.Add(ctx, -1)

		// 3. Logging: record request start
		im.logger.InfoContext(ctx, "request started",
			"method", r.Method,
			"path", r.URL.Path,
			"remote_addr", r.RemoteAddr,
		)

		// 4. Execute business handler
		rw := &instrumentedWriter{
			ResponseWriter: w,
			statusCode:     http.StatusOK,
			bytesWritten:   0,
		}
		next.ServeHTTP(rw, r.WithContext(ctx))

		// 5. Record response metrics
		duration := time.Since(start)
		im.metrics.RecordHTTPRequest(ctx, r.Method, r.URL.Path,
			fmt.Sprintf("%d", rw.statusCode), duration)

		// 6. Tracing: record response info
		span.SetAttributes(
			attribute.Int("http.status_code", rw.statusCode),
			attribute.Int("http.response_size", rw.bytesWritten),
			attribute.Float64("http.duration_ms", float64(duration.Milliseconds())),
		)
		if rw.statusCode >= 500 {
			span.SetStatus(codes.Error, fmt.Sprintf("HTTP %d", rw.statusCode))
		}

		// 7. Logging: record request completion
		im.logger.InfoContext(ctx, "request completed",
			"status_code", rw.statusCode,
			"duration_ms", duration.Milliseconds(),
			"bytes_written", rw.bytesWritten,
		)
	})
}

// instrumentedWriter wraps ResponseWriter to capture response info
type instrumentedWriter struct {
	http.ResponseWriter
	statusCode   int
	bytesWritten int
}

func (iw *instrumentedWriter) WriteHeader(code int) {
	iw.statusCode = code
	iw.ResponseWriter.WriteHeader(code)
}

func (iw *instrumentedWriter) Write(b []byte) (int, error) {
	n, err := iw.ResponseWriter.Write(b)
	iw.bytesWritten += n
	return n, err
}

// InstrumentDB database query observability decorator
func (im *InstrumentationMiddleware) InstrumentDB(
	ctx context.Context,
	queryName string,
	fn func(ctx context.Context) error,
) error {
	start := time.Now()
	ctx, span := im.tracer.Start(ctx, fmt.Sprintf("DB %s", queryName),
		trace.WithSpanKind(trace.SpanKindClient),
		trace.WithAttributes(
			attribute.String("db.query.name", queryName),
		),
	)
	defer span.End()

	err := fn(ctx)
	duration := time.Since(start)

	im.metrics.RecordDBQuery(ctx, queryName, duration, err)

	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		im.logger.ErrorContext(ctx, "db query failed",
			"query", queryName,
			"error", err.Error(),
			"duration_ms", duration.Milliseconds(),
		)
	} else {
		im.logger.InfoContext(ctx, "db query completed",
			"query", queryName,
			"duration_ms", duration.Milliseconds(),
		)
	}

	return err
}

// InstrumentGRPC gRPC observability interceptor (server-side)
func (im *InstrumentationMiddleware) InstrumentGRPC(
	ctx context.Context,
	req interface{},
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (interface{}, error) {
	start := time.Now()

	ctx, span := im.tracer.Start(ctx, info.FullMethod,
		trace.WithSpanKind(trace.SpanKindServer),
	)
	defer span.End()

	resp, err := handler(ctx, req)
	duration := time.Since(start)

	im.metrics.RecordHTTPRequest(ctx, "gRPC", info.FullMethod,
		statusFromError(err), duration)

	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		im.logger.ErrorContext(ctx, "grpc call failed",
			"method", info.FullMethod,
			"error", err.Error(),
			"duration_ms", duration.Milliseconds(),
		)
	} else {
		im.logger.InfoContext(ctx, "grpc call completed",
			"method", info.FullMethod,
			"duration_ms", duration.Milliseconds(),
		)
	}

	return resp, err
}

Key Takeaways:

  • Unified middleware handles all three pillars (logs, metrics, traces) simultaneously, ensuring data consistency
  • instrumentedWriter captures response status code and byte count, enriching trace and metric dimensions
  • InstrumentDB decorator adds observability to database queries without modifying business code
  • gRPC interceptors share the same observability infrastructure as HTTP middleware

Core Pattern 5: Production-Grade Observability Dashboard Integration

Integrating the three pillars — logs, metrics, and traces — into a unified Grafana dashboard with one-click navigation from alerts to logs to traces is the ultimate goal of production-grade observability.

package main

import (
	"context"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"go.opentelemetry.io/otel/exporters/prometheus"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// ObservabilityStack production-grade observability stack
type ObservabilityStack struct {
	logger          *Logger
	metricsProvider *MetricsProvider
	tracingProvider *TracingProvider
	middleware      *InstrumentationMiddleware
	metrics         *AppMetrics
	shutdownFuncs   []func(ctx context.Context) error
}

// Config observability stack configuration
type Config struct {
	ServiceName    string
	OTLPEndpoint   string
	MetricsPath    string
	LogLevel       string
	SamplingRatio  float64
}

// NewObservabilityStack initializes the complete observability stack
func NewObservabilityStack(ctx context.Context, cfg Config) (*ObservabilityStack, error) {
	stack := &ObservabilityStack{}

	// 1. Initialize structured logging
	stack.logger = NewLogger(cfg.ServiceName)

	// 2. Initialize metrics collection
	promExporter, err := prometheus.New()
	if err != nil {
		return nil, fmt.Errorf("create prometheus exporter: %w", err)
	}

	metricProvider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(promExporter),
	)
	stack.metricsProvider = &MetricsProvider{
		provider: metricProvider,
		meter:    metricProvider.Meter(cfg.ServiceName),
	}
	stack.shutdownFuncs = append(stack.shutdownFuncs, metricProvider.Shutdown)

	// 3. Initialize distributed tracing
	otlpExporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint(cfg.OTLPEndpoint),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, fmt.Errorf("create OTLP exporter: %w", err)
	}

	traceProvider := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(otlpExporter),
		sdktrace.WithSampler(sdktrace.ParentBased(
			sdktrace.TraceIDRatioBased(cfg.SamplingRatio),
		)),
	)
	stack.tracingProvider = &TracingProvider{
		provider: traceProvider,
		tracer:   traceProvider.Tracer(cfg.ServiceName),
	}
	stack.shutdownFuncs = append(stack.shutdownFuncs, traceProvider.Shutdown)

	// 4. Initialize application metrics
	stack.metrics, err = NewAppMetrics(stack.metricsProvider)
	if err != nil {
		return nil, fmt.Errorf("create app metrics: %w", err)
	}

	// 5. Initialize unified middleware
	stack.middleware = NewInstrumentationMiddleware(
		stack.logger,
		stack.metrics,
		stack.tracingProvider,
	)

	return stack, nil
}

// Shutdown gracefully shuts down all observability components
func (s *ObservabilityStack) Shutdown(ctx context.Context) error {
	var firstErr error
	for _, fn := range s.shutdownFuncs {
		if err := fn(ctx); err != nil && firstErr == nil {
			firstErr = err
		}
	}
	return firstErr
}

// --- Complete Startup Example ---

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Initialize observability stack
	stack, err := NewObservabilityStack(ctx, Config{
		ServiceName:   "user-service",
		OTLPEndpoint:  "otel-collector:4317",
		MetricsPath:   "/metrics",
		LogLevel:      "info",
		SamplingRatio: 0.1,
	})
	if err != nil {
		fmt.Fprintf(os.Stderr, "init observability: %v\n", err)
		os.Exit(1)
	}
	defer stack.Shutdown(context.Background())

	// Create HTTP router
	mux := http.NewServeMux()

	// Register business routes (with observability middleware)
	mux.HandleFunc("GET /users/{id}", handleGetUser(stack.logger))
	mux.HandleFunc("POST /users", handleCreateUser(stack.logger, stack.metrics))

	// Register Prometheus metrics endpoint
	mux.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
		promHandler := prometheus.Handler()
		promHandler.ServeHTTP(w, r)
	})

	// Health check
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ok"))
	})

	// Apply observability middleware
	handler := stack.middleware.InstrumentHTTP(mux)
	handler = middlewareRequestID(handler)

	// Start HTTP server
	server := &http.Server{
		Addr:         ":8080",
		Handler:      handler,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 30 * time.Second,
		IdleTimeout:  60 * time.Second,
	}

	// Graceful shutdown
	go func() {
		sigCh := make(chan os.Signal, 1)
		signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
		<-sigCh

		shutdownCtx, shutdownCancel := context.WithTimeout(
			context.Background(), 10*time.Second,
		)
		defer shutdownCancel()

		stack.logger.InfoContext(ctx, "shutting down server...")
		server.Shutdown(shutdownCtx)
		stack.Shutdown(shutdownCtx)
	}()

	stack.logger.InfoContext(ctx, "server starting",
		"addr", server.Addr,
		"service", "user-service",
	)

	if err := server.ListenAndServe(); err != http.ErrServerClosed {
		stack.logger.ErrorContext(ctx, "server error", "error", err.Error())
		os.Exit(1)
	}
}

Grafana Dashboard Configuration Highlights:

# docker-compose.yaml - Complete observability stack
version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

  prometheus:
    image: prom/prometheus:v2.50.0
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  loki:
    image: grafana/loki:2.9.4
    ports:
      - "3100:3100"

  tempo:
    image: grafana/tempo:2.3.1
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml
    ports:
      - "3200:3200"

  grafana:
    image: grafana/grafana:10.3.3
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
      - ./grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
    ports:
      - "3000:3000"

Key Takeaways:

  • ObservabilityStack manages the lifecycle of all three pillars, ensuring graceful shutdown
  • Prometheus + Loki + Tempo + Grafana form a complete observability backend
  • Enable TraceID navigation in Grafana data source configuration for one-click log-to-trace correlation
  • Control sampling rate via configuration; 10% sampling recommended for production with automatic 100% sampling for error traces

Common Pitfalls

Pitfall 1: slog's With() Method Doesn't Copy Attributes

// ❌ Wrong: With() returns a new Logger, the original is unaffected
logger := slog.Default().With("request_id", "abc123")
logger.Info("message") // has request_id
slog.Info("message")   // no request_id!

// ✅ Correct: Always use the new Logger returned by With()
baseLogger := slog.Default()
requestLogger := baseLogger.With("request_id", "abc123")
requestLogger.Info("message") // has request_id

Pitfall 2: Forgetting to Propagate Trace Context in HTTP Clients

// ❌ Wrong: Using http.NewRequest directly, trace chain breaks
req, _ := http.NewRequest("GET", "http://user-service/users/123", nil)
resp, _ := http.DefaultClient.Do(req)

// ✅ Correct: Use NewRequestWithContext and inject propagator
req, _ := http.NewRequestWithContext(ctx, "GET", "http://user-service/users/123", nil)
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
resp, _ := http.DefaultClient.Do(req)

Pitfall 3: Improper Histogram Bucket Boundaries

// ❌ Wrong: Using default bucket boundaries, can't distinguish normal from abnormal latency
am.httpRequestDuration, _ = m.Float64Histogram("http.server.duration")

// ✅ Correct: Custom bucket boundaries matching HTTP request latency distribution
am.httpRequestDuration, _ = m.Float64Histogram(
	"http.server.duration",
	metric.WithUnit("s"),
)
// Configure View in MeterProvider for custom bucket boundaries
sdkmetric.WithView(
	sdkmetric.NewView(
		sdkmetric.Instrument{Name: "http.server.duration"},
		sdkmetric.Stream{
			Aggregation: sdkmetric.AggregationExplicitBucketHistogram{
				Boundaries: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
			},
		},
	),
)

Pitfall 4: Creating Many Attributes in Hot Paths

// ❌ Wrong: Creating new attribute slices on every request
span.SetAttributes(
	attribute.String("user.id", userID),
	attribute.String("request.path", path),
	attribute.String("user.agent", userAgent),
)

// ✅ Correct: Pre-create common attributes to reduce GC pressure
var (
	attrUserID    = attribute.Key("user.id")
	attrReqPath   = attribute.Key("request.path")
	attrUserAgent = attribute.Key("user.agent")
)

span.SetAttributes(
	attrUserID.String(userID),
	attrReqPath.String(path),
	attrUserAgent.String(userAgent),
)

Pitfall 5: TracerProvider Not Gracefully Shut Down, Causing Span Loss

// ❌ Wrong: Exiting directly, Spans in BatchSpanProcessor are lost
func main() {
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	// ... program runs ...
	// Process exits, unflushed Spans are lost!
}

// ✅ Correct: Call Shutdown before exit to ensure all Spans are exported
func main() {
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	defer func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		tp.Shutdown(ctx) // Ensure all Spans are flushed
	}()
	// ... program runs ...
}

Error Troubleshooting

Error Symptom Possible Cause Troubleshooting Method Solution
No trace_id in logs Trace info not extracted from context Check if WithContext is called correctly Ensure middleware initializes before logging
No Prometheus metric data MeterProvider has no Reader registered Check if /metrics endpoint works Confirm prometheus.New() is passed to WithReader
Trace chain breaks at service boundary TraceContext not propagated Check if HTTP client calls Inject Inject propagator before client requests
Span data appears delayed in Jaeger BatchSpanProcessor batch sending Check if Shutdown is called Call tp.Shutdown(ctx) on graceful shutdown
Metric values abnormally high UpDownCounter not properly decremented Check if Add(-1) executes on all paths Use defer to ensure decrement
Grafana can't query Loki logs Loki data source misconfigured Check Loki URL and Label config Confirm {service="xxx"} label matches
Full sampling despite sampling rate set Sampler config overridden Check for AlwaysSample override Use ParentBased to wrap sampler
gRPC trace Spans missing Interceptor not registered Check if gRPC Server has interceptor Use otelgrpc.ServerInterceptor
Log output is plain text not JSON Using default TextHandler Check slog.New Handler parameter Replace with JSONHandler
OTLP export connection timeout Collector not running or wrong port Check Collector status and port Confirm port 4317 is reachable

Advanced Optimization

1. Adaptive Sampling Strategy

Dynamically adjust sampling rate based on request characteristics: 100% for error requests, 50% for slow requests, 1% for normal requests. Implement custom sampling logic via the ShouldSample interface.

2. Automatic Log-Trace Correlation

Implement a custom slog.Handler that automatically injects the current Span's TraceID and SpanID into every log entry, enabling one-click navigation from logs to traces without manual propagation.

3. Metric Cardinality Control

High-cardinality labels (like user_id) can cause Prometheus memory explosions. Use attribute.KeyValueFilter or CardinalityLimit configuration to limit label cardinality and prevent metric explosion.

4. Exemplar Correlation Between Metrics and Traces

Embed Exemplars (samples containing TraceID) in Prometheus metrics to enable direct navigation from metric charts to trace details. OpenTelemetry SDK natively supports Exemplars.

5. Multi-Cluster Federated Monitoring

For multi-cluster deployments, use Prometheus Federation + Thanos for a global metric view, and configure multiple Grafana data sources for cross-cluster observability.

Comparison

Dimension slog + OTel Zap + Prometheus Logrus + Jaeger
Structured Logging ✅ Standard library native ✅ Third-party library ⚠️ Requires Hook
Metrics Collection ✅ OTel unified SDK ✅ Native Prometheus ❌ Requires extra integration
Distributed Tracing ✅ OTel native support ❌ Requires extra integration ✅ Jaeger client
Context Propagation ✅ Context native integration ⚠️ Manual propagation ❌ Not supported
Sampling Control ✅ Built-in multiple samplers ❌ Not applicable ⚠️ Limited support
Standard Compatibility ✅ W3C/CNCF standards ⚠️ Prometheus-specific ⚠️ Jaeger-specific
Maintenance Cost ✅ Standard lib + CNCF maintained ⚠️ Community maintained ❌ No longer maintained
Learning Curve ⚠️ More OTel concepts ✅ Simple and direct ✅ Simple

Summary

In 2026, slog + OpenTelemetry has become the de facto standard for Go observability. Stop debugging production issues with fmt.Println. Stop letting logs and metrics remain siloed. Stop letting trace chains break at service boundaries. Start with structured logging, gradually integrate metrics and tracing, and ultimately build a unified three-pillar observability system — that's the right approach to productionizing Go services.

Online Tool Recommendations

  • JSON Formatter — Format slog's JSON log output to quickly locate problem fields
  • cURL to Code — Convert cURL commands to Go HTTP client code for quick trace context propagation verification
  • Hash Calculator — Calculate hash values for request IDs, useful for log anonymization and sampling bucketing

Try these browser-local tools — no sign-up required →

#Go遥测#可观测性#OpenTelemetry#2026#技术架构