Grafana Observability in Practice: Dashboards, PromQL, Alerting, and Provisioning

Why Grafana Is the Hub of Observability

Metrics, Logs, and Traces are the three pillars of observability, but what ties them together is usually Grafana. Its value is not "drawing charts" but aggregating scattered signals into one interface that answers "what is happening to the system right now."

Pillar	Source	Question answered
Metrics	Prometheus	Is the system healthy overall?
Logs	Loki	What exactly happened on error?
Traces	Tempo	Which service is a slow request stuck in?

---

Building Effective Dashboards

Principles of a good dashboard: one panel tells one story, top-down from coarse to fine. A typical layout:

1. Top: global SLOs (error rate, P99 latency, QPS).
2. Middle: per-service / per-instance breakdown panels.
3. Bottom: raw logs and a single trace link.

{
  "panels": [
    {
      "title": "P99 Latency",
      "type": "timeseries",
      "targets": [
        { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))" }
      ]
    }
  ]
}

Dashboard JSON is usually large; run it through the JSON Formatter tool before editing to avoid breaking nested levels.

---

Writing Good PromQL: The Language of Observability

Rate beats raw counters

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

Quantiles use histogram_quantile

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Compare by label

# QPS per instance
sum(rate(http_requests_total[5m])) by (instance)

---

Variables and Templating: One Dashboard for Everything

Use template variables so the same panel switches dynamically by service or env, avoiding dozens of copy-pasted panels.

{
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(http_requests_total, service)"
      }
    ]
  }
}

Reference in panels: sum(rate(http_requests_total{service="$service"}[5m])).

---

Multi-Source: Metrics + Logs + Traces

Grafana's strength is correlation. For example, open "Explore" from a Metrics panel, jump to Tempo with the same trace_id, then to Loki logs with service + time window.

# Filter in Loki by label
{service="checkout", level="error"} |= "timeout"

---

Alerting: From "Seeing" to "Being Notified"

Alert rules should be based on "symptoms," not "causes":

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
              / sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.service }} error rate exceeds 5%"

for: 10m filters out transient spikes and prevents alert storms.

When debugging which status code triggered an alert, the HTTP Status Codes tool helps map 5xx meanings quickly.

---

Annotations and Provisioning as Code

Annotations

Mark events like deploys and rollbacks on the timeline so you can instantly tell "was this caused by the recent release?"

Provisioning: Dashboards as Code

Don't hand-click dashboards — declare them as files under Git:

apiVersion: 1
providers:
  - name: default
    folder: ""
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

This makes dashboards reviewable, rollback-able, and reproducible — true GitOps.

---

FAQ

Q1: P99 returns NaN — what now?

Usually the histogram buckets aren't reported, or there's no data in the rate window. Confirm the metric exists before using histogram_quantile.

Q2: Variable dropdown is empty?

Check the variable's datasource and whether the label_values metric name actually exists.

Q3: Alerts keep false-firing?

Add a for duration, raise the threshold, or use absent() to handle disappearing metrics.

Q4: What's the relationship between Grafana and Prometheus alerting?

Prometheus does "compute + fire"; Grafana does "display + route notifications." You can also let Grafana host alerts directly.

Q5: How do I monitor scheduled jobs?

Cross-check the schedule with the Cron Expression tool, then compare against the metric time window to see if it ran on time.

---

Recommended Tools

In Grafana / observability work, these tools help:

• Cron Expression — verify five-field schedules of cron jobs
• HTTP Status Codes — map 5xx/4xx alert semantics
• JSON Formatter — expand Dashboard / alert-rule JSON
• Base64 Encoder — handle datasource credentials and tokens

---

Grafana is not a "chart tool" but the narrative layer that weaves Metrics/Logs/Traces into "the system's current state." Make dashboards code-driven, alerts symptom-based, and reuse variable-driven — and observability becomes real productivity.