OpenTelemetry Observability in Practice: 6 Core Components for Unified Microservice Monitoring

DevOps

Microservices' Darkest Hour: When Troubleshooting Is Pure Guesswork

2 AM, alerts are flashing wildly. Users report order failures, but logs are scattered across 12 services' Elasticsearch indices. Trace IDs break between services and can't be linked together. Key metrics are missing, making it impossible to pinpoint bottlenecks. The team spent 3 hours digging through logs before discovering the payment service's database connection pool was exhausted — a conclusion that should have taken 5 minutes.

This isn't an isolated case. Scattered logs, broken traces, missing metrics, lost context, and alert fatigue — these are the five pain points of microservice observability. When a system is split from a monolith into dozens of microservices, traditional log-based debugging completely fails. OpenTelemetry, as the CNCF observability standard, provides a unified collection, processing, and export solution. This article covers 6 core components to help you build a complete microservice observability system.

Core Concepts Reference

Concept Description Core Role
Observability Ability to infer internal system state from external outputs Evolve from "monitoring known issues" to "exploring unknown ones"
OpenTelemetry (OTel) CNCF observability framework, unified Traces/Metrics/Logs collection Vendor-neutral APIs, SDKs, and tooling
Distributed Tracing Cross-service request call chain tracking Locate cross-service latency bottlenecks and root causes
Metrics Quantified measurement data of the system Performance monitoring, capacity planning, trend analysis
Logs Discrete event records Error investigation, auditing, context enrichment
OTel Collector Data processing hub: receive → process → export Decouple collection from backends, unified data processing
Context Propagation Propagating TraceID and other context across services W3C Trace Context standard, linking call chains

Problem Analysis: 5 Challenges of Microservice Observability

Challenge 1: Data Silos. Traces in Jaeger, Metrics in Prometheus, Logs in Elasticsearch — three types of data exist in isolation, impossible to correlate. A single request's Trace, Metric, and Log are scattered across three systems, requiring manual switching during troubleshooting.

Challenge 2: Context Breakage. Lack of unified TraceID propagation between services. Context information in HTTP headers, message queues, and gRPC metadata is lost, causing broken call chains. Async messaging scenarios (like Kafka) make context propagation even harder.

Challenge 3: Sampling Strategy. Full Trace collection generates massive data volumes. A service with 100K QPS produces billions of Spans daily. Too aggressive sampling loses critical error information; too conservative sampling makes storage costs unsustainable.

Challenge 4: Storage Costs. Traces, Metrics, and Logs generate TB-level storage daily. Elasticsearch cluster costs are high, long-term retention policies conflict with compliance requirements, and hot-cold data separation is complex.

Challenge 5: Alert Fatigue. Complex microservice dependencies mean one underlying failure triggers alerts from dozens of upstream services. Hundreds of alerts daily, but fewer than 5 require actual action, leading to team desensitization.

Component 1: Distributed Tracing Integration

Distributed tracing is the core of observability. OpenTelemetry collects Trace data through both auto-instrumentation and manual instrumentation.

Go Microservice Auto-Instrumentation

package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(context.Background(),
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("order-service"),
            semconv.ServiceVersionKey.String("1.0.0"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

otlptracegrpc.New creates a gRPC exporter connecting to the OTel Collector on port 4317. sdktrace.WithBatcher enables batch export to reduce network overhead. resource.NewWithAttributes sets the service name and version, identifying call chain nodes in Jaeger. semconv uses OpenTelemetry semantic conventions to ensure attribute naming follows standards.

HTTP Middleware Auto-Tracing

import (
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "net/http"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/api/orders", handleOrders)

    wrappedHandler := otelhttp.NewHandler(mux, "order-service",
        otelhttp.WithMessageEvents(otelhttp.Read, otelhttp.Write),
    )

    http.ListenAndServe(":8080", wrappedHandler)
}

otelhttp.NewHandler wraps the HTTP Handler, automatically creating a Span for each request and recording HTTP method, status code, latency, etc. WithMessageEvents records request and response body read/write events for troubleshooting data transfer issues.

Component 2: Metrics Collection with Prometheus

OpenTelemetry Metrics provides Counter, Histogram, and Gauge instruments with seamless Prometheus integration.

OTel Metrics Export

import (
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/metric"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

func initMeter() (metric.MeterProvider, error) {
    exporter, err := prometheus.New()
    if err != nil {
        return nil, err
    }

    mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(exporter))
    otel.SetMeterProvider(mp)
    return mp, nil
}

func registerMetrics(mp metric.MeterProvider) {
    meter := mp.Meter("order-service")

    requestCounter, _ := meter.Int64Counter(
        "order.requests.total",
        metric.WithDescription("Total order requests"),
    )

    latencyHistogram, _ := meter.Float64Histogram(
        "order.request.duration",
        metric.WithDescription("Order request duration"),
        metric.WithUnit("ms"),
    )

    activeOrders, _ := meter.Int64ObservableGauge(
        "order.active.count",
        metric.WithDescription("Active order count"),
    )

    _ = activeOrders
    _ = requestCounter
    _ = latencyHistogram
}

prometheus.New() creates a Prometheus exporter, automatically exposing OTel metrics at the :8889/metrics endpoint. Int64Counter is for request counts, monotonically increasing. Float64Histogram is for latency distribution — Prometheus can compute P99 via histogram_quantile. Int64ObservableGauge is for current instantaneous values (like active connections) obtained via callbacks.

Business Metrics Instrumentation

func processOrder(ctx context.Context, req *OrderRequest) {
    startTime := time.Now()

    requestCounter.Add(ctx, 1,
        attribute.String("order.type", req.Type),
        attribute.String("order.channel", req.Channel),
    )

    result, err := doProcessOrder(ctx, req)

    duration := float64(time.Since(startTime).Milliseconds())
    latencyHistogram.Record(ctx, duration,
        attribute.String("order.type", req.Type),
        attribute.Bool("order.success", err == nil),
    )
}

Each metric carries business dimension labels (order.type, order.channel), enabling Prometheus aggregation by label. Latency metrics distinguish success and failure, helping detect anomalies like "successful requests are fast but failed ones are slow."

Component 3: Logs Correlation with Trace ID

Logs correlation is the key to connecting Traces, Metrics, and Logs. By injecting TraceID and SpanID into logs, you can jump from a Trace to related Logs with one click.

Structured Logging with TraceID Injection

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/log"
    "go.opentelemetry.io/otel/log/global"
)

func setupLogger() {
    logger := global.LoggerProvider().Logger("order-service")

    span := otel.Tracer("order-service").SpanFromContext(context.Background())
    spanCtx := span.SpanContext()

    logger.Emit(context.Background(), log.Record{
        Body: log.StringValue("Order created successfully"),
        Attributes: []log.KeyValue{
            log.String("trace_id", spanCtx.TraceID().String()),
            log.String("span_id", spanCtx.SpanID().String()),
            log.String("order.id", "ORD-20260618-001"),
            log.Int("order.items", 3),
            log.Float64("order.amount", 299.99),
        },
        Severity: log.SeverityInfo,
    })
}

Zap Logger Integration with OTel

import (
    "go.opentelemetry.io/contrib/bridges/otelzap"
    "go.uber.org/zap"
)

func initZapLogger() *zap.Logger {
    core := otelzap.NewCore("order-service",
        otelzap.WithLoggerProvider(global.GetLoggerProvider()),
    )

    zapCore := zapcore.NewTee(
        zapcore.NewCore(
            zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
            zapcore.AddSync(os.Stdout),
            zapcore.DebugLevel,
        ),
        core,
    )

    return zap.New(zapCore)
}

otelzap.NewCore bridges Zap logs to the OpenTelemetry Logs pipeline, automatically carrying TraceID and SpanID. zapcore.NewTee outputs to both stdout and the OTel pipeline, ensuring both local debugging and remote collection work. The JSON encoder outputs structured logs that Elasticsearch can index directly.

Correlating Logs with Traces

In Kibana/Grafana Loki, query by TraceID for correlation:

# Find the slow request's TraceID in Jaeger
TraceID: 4bf92f3577b34da6a3ce929d0e0e4736

# Query all logs for this Trace in Elasticsearch
GET otel-logs/_search
{
  "query": {
    "term": {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
    }
  },
  "sort": [{ "timestamp": "asc" }]
}

From Jaeger's Trace detail page, click "View Logs" to automatically jump to Elasticsearch and query all logs for that TraceID, achieving seamless Trace-Log correlation.

Component 4: OTel Collector Deployment

The OTel Collector is the data processing hub, supporting receiving, processing, and exporting all three signals — the core of the entire observability architecture.

Collector Pipeline Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  elasticsearch:
    endpoints:
      - http://elasticsearch:9200
    logs_index: otel-logs

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, tail_sampling]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch]

Three pipelines process Traces, Metrics, and Logs separately. The batch processor sends data in batches to reduce network requests. tail_sampling ensures errors and slow requests are 100% collected. The prometheus exporter exposes Metrics on port 8889 for Prometheus scraping. The elasticsearch exporter writes Logs to ES with index name otel-logs.

Kubernetes DaemonSet Deployment

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          ports:
            - containerPort: 4317
            - containerPort: 4318
            - containerPort: 8889
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol
          resources:
            limits:
              memory: "512Mi"
              cpu: "500m"
      volumes:
        - name: config
          configMap:
            name: otel-collector-config

DaemonSet mode deploys one Collector instance per Node. Applications send data via localhost, reducing network hops. Resource limits of 512Mi memory and 500m CPU prevent the Collector from consuming too many resources and affecting business Pods.

Component 5: Sampling Strategy and Cost Control

Sampling strategy is key to balancing observability and cost. Head-based sampling is simple and efficient but may lose critical information. Tail-based sampling ensures 100% collection of errors but requires waiting for the Trace to complete.

Tail-Based Sampling Configuration

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 100
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 1000
      - name: critical-service
        type: string_attribute
        string_attribute:
          key: service.name
          values:
            - payment-service
            - order-service
      - name: health-check-exclude
        type: and
        and:
          sub_policies:
            - name: not-health
              type: string_attribute
              string_attribute:
                key: http.route
                values:
                  - /health
                  - /ready
                invert_match: true
      - name: fallback
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Five sampling policies evaluated top to bottom: errors 100% → slow requests 100% → critical services 100% → exclude health checks → remaining 10% probability. decision_wait: 10s waits 10 seconds to collect the complete Trace before deciding, ensuring both root and child Spans are evaluated. num_traces: 100000 limits the number of Traces cached in memory to prevent OOM.

Cost Estimation

Sampling Strategy Data Volume Monthly Storage Cost Notes
Full collection 100% $7,000 100K QPS, 1B Spans/day
Head-based 10% 10% $700 May lose critical errors
Tail-based (errors+slow+10%) ~15% $1,050 Recommended: ensures key info visible
Custom (errors+critical+5%) ~8% $560 Fine-grained control for large scale

Component 6: Alerting and SLO Monitoring

SLO (Service Level Objective) is a quantified target for measuring service reliability. Combined with OpenTelemetry metrics and Prometheus alerting rules, it enables SLO-based intelligent alerting.

SLO Definition and Alerting Rules

groups:
  - name: order-service-slo
    rules:
      - record: order:slo:availability:ratio
        expr: |
          sum(rate(http_server_request_duration_seconds_count{service="order-service",status_code!~"5.."}[5m]))
          /
          sum(rate(http_server_request_duration_seconds_count{service="order-service"}[5m]))

      - record: order:slo:latency:p99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_server_request_duration_seconds_bucket{service="order-service"}[5m]))
            by (le)
          )

      - alert: OrderServiceSLOAvailabilityBurnRate
        expr: |
          (
            1 - order:slo:availability:ratio
          ) > (1 - 0.999) * 14.4
        for: 5m
        labels:
          severity: critical
          team: order
        annotations:
          summary: "Order service availability SLO burn rate too high"
          description: "Availability below 99.9% for the past 5 minutes, burn rate exceeds 14.4x, 30-day SLO budget will be exhausted in 2 days"

      - alert: OrderServiceSLOLatencyBurnRate
        expr: |
          order:slo:latency:p99 > 0.5
        for: 5m
        labels:
          severity: warning
          team: order
        annotations:
          summary: "Order service P99 latency exceeds 500ms SLO"
          description: "Current P99 latency is {{ $value }}s, exceeding SLO threshold of 0.5s"

order:slo:availability:ratio calculates the availability ratio (non-5xx request percentage). Burn Rate is the ratio of SLO consumption speed to budget — 14.4x means the 30-day SLO budget will be exhausted in 2 days. for: 5m avoids transient spikes triggering alerts; sustained violations for 5 minutes are required.

Multi-Level Alerting Strategy

Level Condition Notification Response Time
P0 Critical Availability <99%, sustained 5min Phone+SMS+Slack Within 5 minutes
P1 Warning P99 latency >1s, sustained 5min SMS+Slack Within 15 minutes
P2 Info Error rate >0.1%, sustained 15min Slack Within 1 hour
P3 Low Resource usage >80%, sustained 30min Email Next business day

Pitfall Guide: 5 Common Traps

❌ Trap 1: Only collecting Traces, ignoring Metrics and Logs ✅ All three observability pillars are essential. Traces identify "where it's slow," Metrics quantify "how slow," and Logs explain "why it's slow." Only correlating all three enables complete troubleshooting.

❌ Trap 2: Single-point Collector deployment ✅ Production environments should deploy at least 2 Collector instances (DaemonSet or Gateway mode) to prevent Collector failure from causing total data loss.

❌ Trap 3: Full sampling without cost control ✅ Use tail-based sampling: errors and slow requests at 100%, normal requests sampled proportionally. Otherwise storage costs scale linearly with traffic.

❌ Trap 4: Not injecting TraceID into logs ✅ Every log entry must include trace_id and span_id fields. Without them, logs can't be correlated with Traces, requiring manual time-window searches.

❌ Trap 5: Overly sensitive alerting rules ✅ Use burn rate instead of simple threshold alerts. Set for duration to avoid transient spikes. Multi-level alerting prevents P0 alert floods.

Error Troubleshooting: 10 Common Errors

Error Symptom Possible Cause Diagnostic Command Solution
Traces break between services Context propagation not configured Check if HTTP headers carry traceparent Enable otelhttp middleware for auto-propagation
Collector connection refused gRPC port not open or wrong address telnet collector 4317 Check Collector address and port config
Metrics not appearing in Prometheus Exporter port not exposed curl localhost:8889/metrics Check prometheus exporter config and port mapping
Logs missing TraceID Logger not integrated with OTel Check if log output contains trace_id field Use otelzap to bridge Zap to OTel
Tail sampling OOM num_traces set too large Check Collector memory usage Reduce num_traces or increase memory limit
High Span data latency Batch processor timeout too long Check Collector processing latency Reduce timeout to 1-2 seconds
Elasticsearch write failure Index template not created or insufficient permissions Check ES errors in Collector logs Create index template and configure correct credentials
Kafka messages lose Trace context Producer/Consumer not configured with propagator Check if message headers contain traceparent Use OTel Kafka auto-instrumentation
Alert storm Rules too sensitive, missing for duration Check Alertmanager silence rules Add burn rate alerts and duration filtering
Collector CPU usage too high Data volume exceeds processing capacity Check Collector's own Metrics Add instances or enable filter processor to remove useless data

Advanced Optimization Tips

1. Frontend RUM Integration. Use @opentelemetry/sdk-trace-web to collect frontend performance data. Fetch/XHR requests automatically carry traceparent headers, enabling end-to-end tracing from browser to backend.

2. Multi-Cluster Collector Federation. Deploy DaemonSet Collectors per cluster with a Gateway Collector aggregating at the upper layer for unified multi-cluster observability. The Gateway layer handles global sampling and routing.

3. Exemplar-Based Metrics-Traces Correlation. Prometheus Exemplars link Metric data points to specific Traces. In Grafana, click on a Metrics chart to jump directly to Jaeger and view the corresponding Trace.

4. Custom Processor Extensions. Use the Collector's transform processor to add business attributes (like environment, team) and the filter processor to remove useless Spans like health checks, reducing storage overhead.

5. SLO Burn Rate Multi-Window Alerting. Calculate burn rates for both 1-hour and 5-minute windows simultaneously. Short windows detect sudden failures; long windows detect chronic degradation, avoiding false positives and missed alerts from a single window.

Comparison: OpenTelemetry vs Jaeger vs Zipkin vs SkyWalking

Feature OpenTelemetry Jaeger Zipkin SkyWalking
Positioning Collection standard + SDK Tracing backend storage Tracing backend storage Full-stack APM solution
Signal Types Traces+Metrics+Logs Traces Traces Traces+Metrics+Logs
Vendor Neutral ✅ CNCF standard ❌ Tied to Jaeger storage ❌ Tied to Zipkin storage ❌ Tied to SkyWalking
Auto-Instrumentation Multi-language SDK + Java Agent Java Agent Limited Java Agent
Collector OTel Collector (feature-rich) Jaeger Collector No standalone Collector OAP Server
Sampling Strategy Head + Tail sampling Head + Adaptive sampling Head sampling Head sampling
Metrics Integration Native support Requires Prometheus Not supported Native support
Logs Correlation Native support Requires ELK Not supported Native support
Alerting Requires Prometheus Requires Alertmanager Not supported Built-in
Community Activity ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐
Production Readiness ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐
  • JSON Formatter — Format OTel Collector configs and Kubernetes YAML, quickly troubleshoot Pipeline definition issues
  • Hash Calculator — Calculate TraceID and SpanID hashes, verify context propagation correctness
  • cURL to Code Converter — Convert OTLP API debug commands to Go/Python code, accelerate Collector integration development

Summary and Outlook

The core of microservice observability isn't tool stacking, but the implementation of three principles: unified Traces/Metrics/Logs signal collection, TraceID as the linking thread throughout the chain, and precise sampling for cost control. The 6 core components — distributed tracing integration, metrics collection with Prometheus, logs correlation with TraceID, OTel Collector deployment, sampling strategy and cost control, alerting and SLO monitoring — cover the complete pipeline from data collection to processing/export to intelligent alerting. Remember: all three signals are indispensable, TraceID is the correlation bond, sampling is the cost valve — only then can you build a truly effective microservice observability system. In 2026, with OpenTelemetry Logs signal officially stable, the three pillars are truly unified, and observability enters a new era of "one SDK solves everything."

Further Reading

Try these browser-local tools — no sign-up required →

#OpenTelemetry可观测性#可观测性三支柱#OpenTelemetry部署#分布式追踪#指标监控#2026#日志关联