Go K8s eBPF Network Monitor: 5 Core Patterns for Cilium Observability
The Problem: 4 Pain Points of K8s Network Monitoring
At 2 AM, your production services are timing out across the board. You open your monitoring dashboard — Service-level latency curves are all red, but which Pod is the culprit? Is DNS resolution slow or did a network policy kill legitimate traffic? Traditional monitoring tools tell you "there's a problem" but can't pinpoint "where the problem is."
Four core pain points of K8s network monitoring:
- Coarse monitoring granularity: Traditional monitoring stops at the Service level, unable to drill down to Pod/Container dimensions. Microservice call chains remain a black box.
- Service-level can't locate Pods: A Service may have dozens of Pods behind it — which one has abnormal latency? Traditional tools can't tell.
- DNS resolution latency is hard to track: CoreDNS resolution timeouts are the #1 cause of K8s network issues, but traditional tools can't trace the full lifecycle of individual DNS queries.
- Network policy effects are invisible: After CiliumNetworkPolicy takes effect, which connections did it reject? Is the policy too strict? No visualization means no verification.
eBPF's natural advantage of collecting network events at the kernel layer, combined with Cilium and Hubble, turns K8s networking from a "black box" into a "glass box."
Core Concepts Quick Reference
| Concept | Description | Key Value |
|---|---|---|
| eBPF | Extended Berkeley Packet Filter, Linux kernel programmable sandbox | Collect network events at kernel level without modifying kernel |
| Cilium | eBPF-based K8s CNI plugin | Replace kube-proxy, provide high-performance data plane and observability foundation |
| Hubble | Cilium's network observability platform | Real-time visualization of service dependencies, traffic topology, and DNS resolution |
| Network Observability | Full-link awareness of network traffic | Complete traffic visibility from L3 to L7 |
| Traffic Topology | Visual graph of service call relationships | One-click discovery of anomalous call chains and bottleneck nodes |
| DNS Monitoring | Tracking and measurement of DNS queries/responses | Identify domain resolution latency and NXDOMAIN errors |
| L7/L4 Monitoring | Application/transport layer protocol parsing | Fine-grained observability of HTTP methods/paths/gRPC methods |
| Network Policy Audit | Recording connections rejected/allowed by policies | Verify policy effectiveness, assist with policy tuning |
Problem Analysis: 5 Challenges of eBPF Network Monitoring
1. eBPF Program Development Complexity: eBPF C programs require manual memory management, boundary checking, and verifier rule compliance. The development barrier is high and debugging is difficult. A single pointer out-of-bounds renders the program unloadable.
2. Kernel Version Compatibility: Different eBPF features depend on different kernel versions — ringbuf requires 5.8+, bpf_skb_ecn_set_ce requires 5.1+. Inconsistent kernel versions across multi-cluster environments cause feature fragmentation.
3. Monitoring Data Explosion: Large-scale clusters generate hundreds of thousands of network events per second. Hubble Flow log storage costs grow linearly. How to control data volume without losing critical events?
4. Performance Overhead Control: eBPF programs run on the kernel hot path — every packet passes through eBPF processing. Improper implementation causes CPU overhead spikes that impact business throughput.
5. Multi-Cluster Observability: Production environments typically use multi-cluster architectures. Hubble defaults to a single-cluster view. Cross-cluster traffic topology requires ClusterMesh + Hubble Relay cascading, which is complex to configure.
Pattern 1: Cilium Installation and Hubble Observability Configuration
Hubble is Cilium's built-in network observability component that collects L3-L7 full-link traffic without Sidecars.
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium \
--namespace kube-system \
--set kubeProxyReplacement=strict \
--set hubble.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,http,port-distribution,icmp,httpV2}" \
--set operator.replicas=1 \
--set ipv4NativeRoutingCIDR="10.0.0.0/8"
cilium status
cilium connectivity test
Enable Hubble UI and access traffic topology:
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
hubble observe --since 5m --namespace production
hubble observe --pod-name payment-service-7d9f8b6c4-x2k1j
hubble observe --protocol http --type trace --since 1m
Hubble Flow export to Prometheus:
apiVersion: v1
kind: ConfigMap
metadata:
name: hubble-metrics-config
namespace: kube-system
data:
hubble-metrics: |
dns:
enabled: true
drop:
enabled: true
tcp:
enabled: true
flow:
enabled: true
http:
enabled: true
httpV2:
enabled: true
labels:
source_pod: true
destination_pod: true
port-distribution:
enabled: true
Pattern 2: Go eBPF Program Development and Network Event Collection
When Hubble's built-in metrics can't meet customization needs, develop custom network event collectors using the cilium/ebpf library.
eBPF C program net_monitor.c:
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/udp.h>
#include <bpf/bpf_helpers.h>
struct net_event {
__u32 src_ip;
__u32 dst_ip;
__u16 src_port;
__u16 dst_port;
__u8 protocol;
__u8 direction;
__u64 timestamp_ns;
__u32 pkt_len;
};
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 1 << 24);
} net_events SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 1024);
__type(key, __u32);
__type(value, __u8);
} monitored_ns SEC(".maps");
SEC("tc")
int tc_monitor(struct __sk_buff *skb) {
void *data_end = (void *)(long)skb->data_end;
void *data = (void *)(long)skb->data;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
return TC_ACT_OK;
if (eth->h_proto != __builtin_bswap16(ETH_P_IP))
return TC_ACT_OK;
struct iphdr *ip = (void *)(eth + 1);
if ((void *)(ip + 1) > data_end)
return TC_ACT_OK;
__u32 dst_ip = ip->daddr;
__u8 *monitored = bpf_map_lookup_elem(&monitored_ns, &dst_ip);
if (!monitored)
return TC_ACT_OK;
struct net_event *e = bpf_ringbuf_reserve(&net_events, sizeof(*e), 0);
if (!e)
return TC_ACT_OK;
e->src_ip = ip->saddr;
e->dst_ip = dst_ip;
e->protocol = ip->protocol;
e->timestamp_ns = bpf_ktime_get_ns();
e->pkt_len = skb->len;
e->direction = 0;
if (ip->protocol == IPPROTO_TCP) {
struct tcphdr *tcp = (void *)(ip + 1);
if ((void *)(tcp + 1) <= data_end) {
e->src_port = __builtin_bswap16(tcp->source);
e->dst_port = __builtin_bswap16(tcp->dest);
}
} else if (ip->protocol == IPPROTO_UDP) {
struct udphdr *udp = (void *)(ip + 1);
if ((void *)(udp + 1) <= data_end) {
e->src_port = __builtin_bswap16(udp->source);
e->dst_port = __builtin_bswap16(udp->dest);
}
}
bpf_ringbuf_submit(e, 0);
return TC_ACT_OK;
}
char _license[] SEC("license") = "GPL";
Go userspace collector:
package main
import (
"bytes"
"encoding/binary"
"fmt"
"log"
"net"
"os"
"os/signal"
"syscall"
"time"
"github.com/cilium/ebpf"
"github.com/cilium/ebpf/link"
"github.com/cilium/ebpf/ringbuf"
)
type netEvent struct {
SrcIP uint32
DstIP uint32
SrcPort uint16
DstPort uint16
Protocol uint8
Direction uint8
TimestampNs uint64
PktLen uint32
}
//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -type net_event bpf ./net_monitor.c
func main() {
spec, err := ebpf.LoadCollectionSpec("bpf.o")
if err != nil {
log.Fatalf("load spec: %v", err)
}
coll, err := ebpf.NewCollection(spec)
if err != nil {
log.Fatalf("new collection: %v", err)
}
defer coll.Close()
iface, _ := net.InterfaceByName("eth0")
l, err := link.AttachTC(link.TCAttach{
Program: coll.Programs["tc_monitor"],
Interface: iface.Index,
})
if err != nil {
log.Fatalf("attach tc: %v", err)
}
defer l.Close()
rd, err := ringbuf.NewReader(coll.Maps["net_events"])
if err != nil {
log.Fatalf("ringbuf reader: %v", err)
}
defer rd.Close()
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
go func() {
for {
record, err := rd.Read()
if err != nil {
log.Printf("read error: %v", err)
continue
}
var e netEvent
if err := binary.Read(bytes.NewReader(record.RawSample), binary.LittleEndian, &e); err != nil {
continue
}
srcIP := intToIP(e.SrcIP)
dstIP := intToIP(e.DstIP)
proto := protoName(e.Protocol)
ts := time.Unix(0, int64(e.TimestampNs))
fmt.Printf("[%s] %s:%d -> %s:%d proto=%s len=%d\n",
ts.Format("15:04:05.000"), srcIP, e.SrcPort, dstIP, e.DstPort, proto, e.PktLen)
}
}()
<-sig
fmt.Println("shutting down...")
}
func intToIP(n uint32) net.IP {
ip := make(net.IP, 4)
binary.LittleEndian.PutUint32(ip, n)
return ip
}
func protoName(p uint8) string {
switch p {
case 6:
return "TCP"
case 17:
return "UDP"
case 1:
return "ICMP"
default:
return fmt.Sprintf("%d", p)
}
}
Pattern 3: DNS Resolution Monitoring and Latency Tracking
DNS resolution latency is the invisible killer of K8s network issues. Hubble's built-in DNS monitoring tracks the full lifecycle of every DNS query.
hubble observe --protocol dns --since 5m
hubble observe --protocol dns --dns-response-code NXDomain
hubble observe --protocol dns --namespace kube-system --label k8s-app=kube-dns
Go program collecting DNS metrics via Hubble gRPC API:
package main
import (
"context"
"fmt"
"log"
"time"
"github.com/cilium/hubble/api/v1/observer"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
type dnsMetric struct {
queryName string
queryType string
latencyMs float64
responseCode string
sourcePod string
}
func monitorDNS(hubbleAddr string) error {
conn, err := grpc.NewClient(hubbleAddr,
grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return fmt.Errorf("connect hubble: %w", err)
}
defer conn.Close()
client := observer.NewObserverClient(conn)
stream, err := client.GetFlows(context.Background(),
&observer.GetFlowsRequest{
Whitelist: []*observer.FlowFilter{
{Protocol: []string{"dns"}},
},
})
if err != nil {
return fmt.Errorf("get flows: %w", err)
}
for {
resp, err := stream.Recv()
if err != nil {
return fmt.Errorf("recv: %w", err)
}
flow := resp.GetFlow()
if flow == nil || flow.GetDns() == nil {
continue
}
dns := flow.GetDns()
latency := float64(flow.GetTime().AsTime().Sub(
flow.GetTime().AsTime())) / float64(time.Millisecond)
m := dnsMetric{
queryName: dns.GetQuery(),
queryType: dns.GetQtypes()[0],
latencyMs: latency,
responseCode: dns.GetRcode(),
sourcePod: flow.GetSource().GetPodName(),
}
if m.latencyMs > 100 {
log.Printf("[DNS SLOW] pod=%s query=%s type=%s latency=%.1fms rcode=%s",
m.sourcePod, m.queryName, m.queryType, m.latencyMs, m.responseCode)
}
}
}
CoreDNS latency Prometheus alert rules:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: dns-latency-alerts
namespace: kube-system
spec:
groups:
- name: dns.rules
rules:
- alert: DNSResolutionSlow
expr: histogram_quantile(0.99, sum(rate(hubble_dns_response_latency_seconds_bucket[5m])) by (le, source_pod)) > 0.1
for: 3m
labels:
severity: warning
annotations:
summary: "DNS resolution slow for pod {{ $labels.source_pod }}"
description: "99th percentile DNS latency exceeds 100ms"
- alert: DNSNXDomainSpike
expr: sum(rate(hubble_dns_response_total{rcode="NXDomain"}[5m])) / sum(rate(hubble_dns_response_total[5m])) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: "NXDomain response rate exceeds 5%"
Pattern 4: L7 Traffic Monitoring and HTTP/gRPC Observability
Hubble parses application-layer protocols via eBPF, capturing HTTP methods, paths, status codes, and gRPC methods without Sidecars.
hubble observe --protocol http --since 5m
hubble observe --protocol http --http-status 5xx
hubble observe --protocol grpc --since 5m
hubble observe --protocol http --namespace production --label app=api-gateway
Go program collecting HTTP traffic metrics with structured logging:
package main
import (
"context"
"encoding/json"
"log"
"os"
"github.com/cilium/hubble/api/v1/observer"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
type httpFlowRecord struct {
Timestamp string `json:"timestamp"`
SourcePod string `json:"sourcePod"`
DestPod string `json:"destPod"`
Method string `json:"method"`
Path string `json:"path"`
StatusCode uint32 `json:"statusCode"`
LatencyNs uint64 `json:"latencyNs"`
Namespace string `json:"namespace"`
}
func monitorHTTP(hubbleAddr string) error {
conn, err := grpc.NewClient(hubbleAddr,
grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return err
}
defer conn.Close()
client := observer.NewObserverClient(conn)
stream, err := client.GetFlows(context.Background(),
&observer.GetFlowsRequest{
Whitelist: []*observer.FlowFilter{
{Protocol: []string{"http"}},
},
})
if err != nil {
return err
}
encoder := json.NewEncoder(os.Stdout)
for {
resp, err := stream.Recv()
if err != nil {
return err
}
flow := resp.GetFlow()
if flow == nil || flow.GetL7() == nil {
continue
}
l7 := flow.GetL7()
http := l7.GetHttp()
if http == nil {
continue
}
record := httpFlowRecord{
Timestamp: flow.GetTime().AsTime().Format("2006-01-02T15:04:05.000Z07:00"),
SourcePod: flow.GetSource().GetPodName(),
DestPod: flow.GetDestination().GetPodName(),
Method: http.GetMethod(),
Path: http.GetUrl(),
StatusCode: http.GetStatusCode(),
LatencyNs: l7.GetLatencyNs(),
Namespace: flow.GetSource().GetNamespace(),
}
if record.StatusCode >= 400 {
encoder.Encode(record)
}
}
}
Hubble metrics configuration for gRPC method-level monitoring:
apiVersion: v1
kind: ConfigMap
metadata:
name: hubble-l7-config
namespace: kube-system
data:
hubble-metrics: |
httpV2:
enabled: true
labels:
source_pod: true
destination_pod: true
source_namespace: true
destination_namespace: true
dns:
enabled: true
drop:
enabled: true
flow:
enabled: true
Pattern 5: Network Policy Auditing and Visualization
Traffic auditing after network policies take effect is critical for K8s network security. Hubble records connections rejected by policies, helping verify policy effectiveness.
hubble observe --type trace --verdict DROPPED --since 10m
hubble observe --type trace --verdict DROPPED --namespace production
hubble observe --type trace --drop-reason POLICY_DENIED
Go program collecting policy audit events:
package main
import (
"context"
"fmt"
"log"
"time"
"github.com/cilium/hubble/api/v1/observer"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
type policyAuditEvent struct {
timestamp time.Time
sourcePod string
destPod string
destPort uint32
protocol string
policyName string
action string
namespace string
}
func auditNetworkPolicy(hubbleAddr string) error {
conn, err := grpc.NewClient(hubbleAddr,
grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return err
}
defer conn.Close()
client := observer.NewObserverClient(conn)
stream, err := client.GetFlows(context.Background(),
&observer.GetFlowsRequest{
Whitelist: []*observer.FlowFilter{
{Verdict: []observer.Verdict{observer.Verdict_DROPPED}},
},
})
if err != nil {
return err
}
for {
resp, err := stream.Recv()
if err != nil {
return err
}
flow := resp.GetFlow()
if flow == nil {
continue
}
evt := policyAuditEvent{
timestamp: flow.GetTime().AsTime(),
sourcePod: flow.GetSource().GetPodName(),
destPod: flow.GetDestination().GetPodName(),
destPort: flow.GetDestination().GetPort(),
protocol: flow.GetType().String(),
action: flow.GetVerdict().String(),
namespace: flow.GetSource().GetNamespace(),
}
if flow.GetDropReason() != observer.DropReason_DROP_REASON_UNKNOWN {
evt.policyName = flow.GetDropReason().String()
}
log.Printf("[AUDIT] %s %s -> %s:%d proto=%s action=%s reason=%s ns=%s",
evt.timestamp.Format("15:04:05"),
evt.sourcePod, evt.destPod, evt.destPort,
evt.protocol, evt.action, evt.policyName, evt.namespace)
}
}
Policy audit Grafana dashboard key PromQL:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: policy-audit-alerts
namespace: kube-system
spec:
groups:
- name: policy.audit
rules:
- alert: ExcessivePolicyDrops
expr: sum(rate(hubble_drop_total{reason="policy_denied"}[5m])) by (source_pod, destination_pod) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "Excessive policy drops from {{ $labels.source_pod }} to {{ $labels.destination_pod }}"
- alert: LegitimateTrafficDropped
expr: sum(rate(hubble_drop_total{reason="policy_denied"}[5m])) by (namespace) / sum(rate(hubble_flow_total[5m])) by (namespace) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "Over 1% of traffic dropped by policy in namespace {{ $labels.namespace }}"
Pitfall Guide: 5 Common Traps
1. ❌ Hubble is not enabled by default → ✅ You must explicitly set hubble.enabled=true and hubble.relay.enabled=true during installation, otherwise hubble observe and the UI won't work.
2. ❌ Monitoring all namespace traffic → ✅ Full Flow log collection in large clusters causes storage explosion. Use --namespace and --label filters to monitor only critical namespaces.
3. ❌ Ignoring eBPF program instruction limits → ✅ The kernel verifier limits single eBPF programs to 1M instructions (5.2+). Use bpf_tail_call to split complex logic.
4. ❌ RingBuffer reads blocking the main goroutine → ✅ RingBuffer.Read() is a blocking call — it must run in a separate goroutine, otherwise it blocks program exit.
5. ❌ No retry on Hubble Relay connection timeout → ✅ Hubble Relay startup depends on Cilium Agent readiness. Implement exponential backoff retry to avoid connection failures from Pod startup ordering.
Troubleshooting: 10 Common Errors
| Error Message | Cause | Solution |
|---|---|---|
hubble observe: unable to connect to Hubble Relay |
Relay not enabled or Service not ready | Confirm hubble.relay.enabled=true, check kubectl get pods -n kube-system -l k8s-app=hubble-relay |
ringbuf: failed to read: ring buffer not available |
Kernel version below 5.8, BPF_MAP_TYPE_RINGBUF unsupported | Upgrade kernel to 5.8+ or switch to BPF_MAP_TYPE_PERF_EVENT_ARRAY |
tc attach failed: cannot attach program to interface |
Interface already has a tc eBPF program | Unload first: tc filter del dev eth0 ingress, then reattach |
bpf verifier: unreachable instruction |
eBPF program has unreachable code | Check for dead code and conditional branches, ensure all paths are reachable |
hubble flow: DNS query not visible |
Hubble DNS monitoring not enabled | Set --set hubble.metrics.enabled="{dns}", verify CoreDNS configuration |
cilium/ebpf: collection load: invalid argument |
eBPF bytecode incompatible with kernel version | Recompile with bpf2go, ensure target kernel version matches |
Hubble UI: no service map data |
L7 protocol parsing not enabled | Enable httpV2 metrics: --set hubble.metrics.enabled="{httpV2}" |
grpc dial: connection refused to hubble-relay |
Relay port not exposed or blocked by network policy | Check hubble-relay Service and NetworkPolicy allow rules |
eBPF program too large: 1M instruction limit exceeded |
Single program instruction count exceeded | Use bpf_tail_call to split into multiple sub-programs |
hubble observe: context deadline exceeded |
Flow data volume too large, Relay processing timeout | Add filter conditions to narrow scope, or increase Relay --buffer-size |
Advanced Optimization Tips
1. Hubble Flow Sampling: Enable Flow sampling in large clusters to record only 1/N of traffic events while fully retaining critical events (DROPPED/ERROR): --set hubble.eventBufferCapacity=16384 --set hubble.eventQueueSize=8192.
2. eBPF Program Tail Call Chains: Use bpf_tail_call to split network monitoring into L3 parsing, L4 parsing, and L7 parsing sub-programs, breaking through instruction limits while achieving modular protocol stack processing.
3. Hubble Metrics + Grafana Integration: Feed Hubble-exported Prometheus metrics into Grafana to build network SLO dashboards — DNS P99 latency, HTTP 5xx rate, and policy rejection rate at a glance.
4. Multi-Cluster Hubble Relay Cascading: Cascade multiple clusters' Hubble Relays to a central Relay via ClusterMesh for cross-cluster traffic topology visualization. Configure hubble.relay.enabled=true and clustermesh.enabled=true.
5. Custom eBPF Map Aggregation: Use Per-CPU Hash Maps in eBPF kernel-side to aggregate traffic statistics, only passing aggregated results to userspace — reducing RingBuffer data volume by 10x or more.
Comparison: Cilium Hubble vs Istio Kiali vs Pixie vs DeepFlow
| Feature | Cilium Hubble | Istio Kiali | Pixie | DeepFlow |
|---|---|---|---|---|
| Data Plane | eBPF kernel layer | Sidecar proxy | eBPF kernel layer | eBPF kernel layer |
| Sidecar | ❌ None | ✅ Envoy | ❌ None | ❌ None |
| L3/L4 Observability | ✅ | ✅ | ✅ | ✅ |
| L7 Protocol Parsing | HTTP/gRPC/Kafka | HTTP/gRPC | HTTP/gRPC/MySQL/Redis | 3000+ protocols |
| DNS Monitoring | ✅ Native | ❌ | ✅ | ✅ |
| Traffic Topology | ✅ Service+Pod level | ✅ Service level | ✅ Service+Pod level | ✅ Service+Pod level |
| Network Policy Audit | ✅ Native | ❌ | ❌ | ❌ |
| Performance Overhead | <2% | 5-15% | <2% | <3% |
| Multi-Cluster | ✅ ClusterMesh | ✅ | ❌ | ✅ |
| Open Source | ✅ | ✅ | ✅ | ✅ Community Edition |
| Learning Curve | Medium | High | Low | Medium |
Summary and Outlook
eBPF is redefining the boundaries of K8s network observability. From Cilium Hubble's zero-Sidecar full-link monitoring, to Go eBPF program customization, from DNS latency tracking to L7 traffic fine-grained observability, from network policy auditing to cross-cluster traffic topology — 5 core patterns build a complete production-grade K8s network monitoring system. In 2026, as eBPF enters the Linux kernel mainline and Hubble's multi-cluster capabilities mature, eBPF network monitoring will become the de facto standard for K8s observability. Mastering these patterns now means laying a solid foundation for the future of cloud-native network architecture.
Recommended Online Tools
-
JSON Formatter - Format and validate Hubble Flow JSON output to quickly identify anomalous traffic events.
-
Hash Encoding Tool - Generate key-value hashes for eBPF Maps, or compute hash values for network policy header matching rules.
-
cURL to Code Converter - Convert Hubble gRPC API cURL requests into Go/Python code for integration into automated monitoring scripts.
Try these browser-local tools — no sign-up required →