K8s DORAメトリクスダッシュボード実践：DevOpsパフォーマンスを定量化する5つのコアパターン
DevOps运维
2026年、DORA（DevOps Research and Assessment）の4つのメトリクスは、エンジニアリングパフォーマンスを測定するゴールドスタンダードとなっています。Googleの研究によれば、エリートパフォーマーは低パフォーマーより208倍高いデプロイ頻度と106倍速いリードタイムを持っています。しかし、Kubernetes環境でこれらのメトリクスを収集、計算、可視化することは容易ではありません。本記事では5つのコア実践パターンを深く掘り下げ、メトリクス定義からGrafanaダッシュボードまで、K8s DORAメトリクス測定体系を完全にマスターします。
コア概念
| 概念 | 説明 | 収集ソース |
|---|---|---|
| デプロイ頻度(DF) | 単位時間あたりの本番環境への成功デプロイ回数 | CI/CD Pipeline、ArgoCD |
| 変更リードタイム(LT) | コミットから本番稼働までの時間 | Git Commit → CI → CD → Production |
| 変更失敗率(CFR) | 本番環境のサービス低下を引き起こす変更の割合 | Incident System + Deploy Records |
| 平均復旧時間(MTTR) | 本番障害からサービス復旧までの平均時間 | Alert System → Incident Resolution |
| DORAダッシュボード | DORA 4メトリクスを可視化するGrafana Dashboard | Prometheus + Loki + Tempo |
問題分析：DevOps測定導入の5つの課題
課題1：デプロイイベント収集の困難さ——デプロイがJenkins/GitHub Actions/ArgoCDなど複数システムに分散し、統一デプロイイベントバスが不足。
課題2：リードタイム追跡の難しさ——Git commitから本番デプロイまで複数システムをまたぎ、タイムスタンプ形式が不統一でチェーンが断絶。
課題3：変更失敗率の関連付けが複雑——インシデントイベントと特定デプロイ変更の正確な関連付けが必要で、手動関連付けは時間がかかりエラーが発生しやすい。
課題4：MTTR計算の定義が不統一——検出時間、対応時間、復旧時間の定義がチームごとに異なり、データが比較不可能。
課題5：ダッシュボードにコンテキストが不足——数値ダッシュボードはメトリクスのみを表示し、コード変更やインシデントチケットとの関連コンテキストがない。
パターン1：DORA 4メトリクス定義と収集アーキテクチャ
収集アーキテクチャ概要
Git Commit → CI Pipeline → CD Pipeline → K8s Deployment
↓ ↓ ↓ ↓
Git Events CI Metrics CD Events Deploy Events
↓ ↓ ↓ ↓
└─────────── Prometheus Pushgateway ──────────┘
↓
Prometheus (保存+計算)
↓
Grafana (可視化+アラート)
デプロイイベントコレクター
#!/usr/bin/env python3
"""
dora_collector.py
DORAメトリクスコレクター：CI/CDシステムからデプロイイベントを収集しPrometheusにプッシュ
"""
import os
import time
import json
import logging
from datetime import datetime, timezone
from dataclasses import dataclass, asdict
from typing import Optional
from prometheus_client import Counter, Histogram, Gauge, push_to_gateway
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
DEPLOY_TOTAL = Counter(
"dora_deploy_total",
"Total number of deployments",
["environment", "service", "status"]
)
DEPLOY_DURATION = Histogram(
"dora_deploy_duration_seconds",
"Deployment duration in seconds",
["environment", "service"],
buckets=[60, 120, 300, 600, 1200, 1800, 3600]
)
LEAD_TIME = Histogram(
"dora_lead_time_seconds",
"Lead time from commit to production in seconds",
["service"],
buckets=[300, 600, 1800, 3600, 7200, 14400, 28800, 86400]
)
CHANGE_FAILURE = Counter(
"dora_change_failure_total",
"Total number of change failures",
["service", "deploy_id"]
)
MTTR = Histogram(
"dora_mttr_seconds",
"Mean Time To Recovery in seconds",
["service", "severity"],
buckets=[60, 300, 600, 1800, 3600, 7200, 14400, 28800]
)
DEPLOY_FREQUENCY = Gauge(
"dora_deploy_frequency",
"Deploy frequency (deploys per day)",
["environment", "service"]
)
PUSHGATEWAY_URL = os.getenv("PUSHGATEWAY_URL", "localhost:9091")
@dataclass
class DeployEvent:
deploy_id: str
service: str
environment: str
status: str
commit_sha: str
commit_timestamp: float
deploy_timestamp: float
deploy_duration: float
@property
def lead_time(self) -> float:
return self.deploy_timestamp - self.commit_timestamp
@dataclass
class IncidentEvent:
incident_id: str
service: str
severity: str
related_deploy_id: Optional[str]
detection_timestamp: float
resolution_timestamp: float
@property
def mttr(self) -> float:
return self.resolution_timestamp - self.detection_timestamp
class DORACollector:
def __init__(self, pushgateway_url: str = PUSHGATEWAY_URL):
self.pushgateway_url = pushgateway_url
def record_deploy(self, event: DeployEvent) -> None:
logger.info(f"Recording deploy: {event.deploy_id} service={event.service} status={event.status}")
DEPLOY_TOTAL.labels(environment=event.environment, service=event.service, status=event.status).inc()
if event.deploy_duration > 0:
DEPLOY_DURATION.labels(environment=event.environment, service=event.service).observe(event.deploy_duration)
if event.status == "success" and event.lead_time > 0:
LEAD_TIME.labels(service=event.service).observe(event.lead_time)
logger.info(f"Lead time: {event.lead_time / 3600:.2f} hours")
self._push_metrics(f"deploy_{event.deploy_id}")
def record_incident(self, event: IncidentEvent) -> None:
logger.info(f"Recording incident: {event.incident_id} severity={event.severity}")
if event.related_deploy_id:
CHANGE_FAILURE.labels(service=event.service, deploy_id=event.related_deploy_id).inc()
MTTR.labels(service=event.service, severity=event.severity).observe(event.mttr)
logger.info(f"MTTR: {event.mttr / 60:.1f} minutes")
self._push_metrics(f"incident_{event.incident_id}")
def update_deploy_frequency(self, service: str, environment: str, frequency: float) -> None:
DEPLOY_FREQUENCY.labels(environment=environment, service=service).set(frequency)
self._push_metrics(f"freq_{service}_{environment}")
def _push_metrics(self, job_name: str) -> None:
try:
push_to_gateway(self.pushgateway_url, job=job_name, registry=None)
except Exception as e:
logger.error(f"Failed to push metrics: {e}")
if __name__ == "__main__":
collector = DORACollector()
now = time.time()
deploy = DeployEvent(
deploy_id="deploy-20260621-001", service="api-server", environment="production",
status="success", commit_sha="abc1234", commit_timestamp=now - 7200,
deploy_timestamp=now, deploy_duration=180
)
collector.record_deploy(deploy)
incident = IncidentEvent(
incident_id="INC-20260621-001", service="api-server", severity="high",
related_deploy_id="deploy-20260621-001", detection_timestamp=now - 1800,
resolution_timestamp=now - 600
)
collector.record_incident(incident)
collector.update_deploy_frequency("api-server", "production", 4.2)
print("✅ DORAメトリクス収集完了！")
ArgoCDデプロイイベント収集
# argocd-dora-metrics.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dora-collector-config
namespace: monitoring
data:
ARGOCD_URL: "https://argocd.example.com"
PUSHGATEWAY_URL: "prometheus-pushgateway:9091"
SERVICES: "api-server,order-service,payment-service"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dora-argocd-collector
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: dora-argocd-collector
template:
metadata:
labels:
app: dora-argocd-collector
spec:
containers:
- name: collector
image: toolsku/dora-collector:latest
envFrom:
- configMapRef:
name: dora-collector-config
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
パターン2：デプロイ頻度とリードタイム追跡
CI/CDパイプラインメトリクス注入
# .github/workflows/dora-metrics.yml
name: Deploy with DORA Metrics
on:
push:
branches: [main]
env:
SERVICE_NAME: "api-server"
PUSHGATEWAY_URL: "prometheus-pushgateway.monitoring:9091"
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Record commit timestamp
id: commit
run: |
COMMIT_TS=$(git log -1 --format=%ct)
echo "commit_ts=${COMMIT_TS}" >> $GITHUB_OUTPUT
echo "commit_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
- name: Build and Push Image
run: |
docker build -t registry.example.com/${{ env.SERVICE_NAME }}:${{ steps.commit.outputs.commit_sha }} .
docker push registry.example.com/${{ env.SERVICE_NAME }}:${{ steps.commit.outputs.commit_sha }}
- name: Deploy to K8s
id: deploy
run: |
DEPLOY_START=$(date +%s)
kubectl set image deployment/${{ env.SERVICE_NAME }} \
${{ env.SERVICE_NAME }}=registry.example.com/${{ env.SERVICE_NAME }}:${{ steps.commit.outputs.commit_sha }} \
-n production
kubectl rollout status deployment/${{ env.SERVICE_NAME }} -n production --timeout=300s
DEPLOY_END=$(date +%s)
echo "deploy_duration=$((DEPLOY_END - DEPLOY_START))" >> $GITHUB_OUTPUT
echo "deploy_ts=${DEPLOY_END}" >> $GITHUB_OUTPUT
- name: Push DORA metrics
if: success()
run: |
LEAD_TIME=$((${{ steps.deploy.outputs.deploy_ts }} - ${{ steps.commit.outputs.commit_ts }}))
cat <<EOF | curl --data-binary @- http://${{ env.PUSHGATEWAY_URL }}/metrics/job/deploy_${{ env.SERVICE_NAME }}
dora_deploy_total{environment="production",service="${{ env.SERVICE_NAME }}",status="success"} 1
dora_lead_time_seconds_bucket{service="${{ env.SERVICE_NAME }}",le="${LEAD_TIME}"} 1
EOF
- name: Push failure metrics
if: failure()
run: |
cat <<EOF | curl --data-binary @- http://${{ env.PUSHGATEWAY_URL }}/metrics/job/deploy_${{ env.SERVICE_NAME }}
dora_deploy_total{environment="production",service="${{ env.SERVICE_NAME }}",status="failed"} 1
EOF
デプロイ頻度PromQLクエリ
# dora-frequency-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: dora-frequency-rules
namespace: monitoring
spec:
groups:
- name: dora_frequency
interval: 5m
rules:
- record: dora:deploy_frequency:daily
expr: sum(increase(dora_deploy_total{status="success"}[1d])) by (environment, service)
- record: dora:deploy_frequency:weekly
expr: sum(increase(dora_deploy_total{status="success"}[7d])) by (environment, service) / 7
- record: dora:lead_time:p50
expr: histogram_quantile(0.5, sum(rate(dora_lead_time_seconds_bucket[7d])) by (service, le))
- record: dora:lead_time:p90
expr: histogram_quantile(0.9, sum(rate(dora_lead_time_seconds_bucket[7d])) by (service, le))
- record: dora:lead_time:p99
expr: histogram_quantile(0.99, sum(rate(dora_lead_time_seconds_bucket[7d])) by (service, le))
パターン3：変更失敗率とMTTR測定
変更失敗率計算
#!/usr/bin/env python3
"""
dora_cfr_calculator.py
変更失敗率(CFR)計算機：デプロイイベントとインシデントイベントの関連付け
"""
import json
import logging
from datetime import datetime, timedelta
from typing import Dict, List
from dataclasses import dataclass
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
@dataclass
class DeployRecord:
deploy_id: str
service: str
timestamp: datetime
commit_sha: str
status: str
@dataclass
class IncidentRecord:
incident_id: str
service: str
detection_time: datetime
resolution_time: datetime
severity: str
root_cause: str
class CFRCalculator:
ASSOCIATION_WINDOW = timedelta(hours=24)
def __init__(self):
self.deploys: List[DeployRecord] = []
self.incidents: List[IncidentRecord] = []
def add_deploy(self, deploy: DeployRecord) -> None:
self.deploys.append(deploy)
def add_incident(self, incident: IncidentRecord) -> None:
self.incidents.append(incident)
def calculate_cfr(self, service: str = None, days: int = 30) -> Dict:
cutoff = datetime.now() - timedelta(days=days)
deploys = [d for d in self.deploys if d.timestamp >= cutoff and (service is None or d.service == service)]
incidents = [i for i in self.incidents if i.detection_time >= cutoff and (service is None or i.service == service)]
total_deploys = len(deploys)
if total_deploys == 0:
return {"cfr": 0, "total_deploys": 0, "failed_deploys": 0}
failed_deploys = set()
for incident in incidents:
for deploy in deploys:
if (deploy.service == incident.service and
deploy.timestamp <= incident.detection_time and
incident.detection_time - deploy.timestamp <= self.ASSOCIATION_WINDOW):
failed_deploys.add(deploy.deploy_id)
cfr = len(failed_deploys) / total_deploys * 100
return {"cfr": round(cfr, 2), "total_deploys": total_deploys, "failed_deploys": len(failed_deploys)}
def calculate_mttr(self, service: str = None, days: int = 30) -> Dict:
cutoff = datetime.now() - timedelta(days=days)
incidents = [i for i in self.incidents if i.detection_time >= cutoff and (service is None or i.service == service)]
if not incidents:
return {"mttr_minutes": 0, "incident_count": 0}
total_recovery = sum((i.resolution_time - i.detection_time).total_seconds() for i in incidents)
mttr_minutes = (total_recovery / len(incidents)) / 60
return {"mttr_minutes": round(mttr_minutes, 1), "incident_count": len(incidents)}
def generate_dora_report(self, service: str = None, days: int = 30) -> Dict:
cfr_result = self.calculate_cfr(service, days)
mttr_result = self.calculate_mttr(service, days)
cfr = cfr_result["cfr"]
mttr_min = mttr_result["mttr_minutes"]
if cfr <= 5 and mttr_min <= 60:
level = "Elite"
elif cfr <= 10 and mttr_min <= 240:
level = "High"
elif cfr <= 15 and mttr_min <= 1440:
level = "Medium"
else:
level = "Low"
return {"dora_level": level, "change_failure_rate": cfr_result, "mttr": mttr_result}
if __name__ == "__main__":
calc = CFRCalculator()
now = datetime.now()
for i in range(20):
calc.add_deploy(DeployRecord(
deploy_id=f"deploy-{i:03d}", service="api-server",
timestamp=now - timedelta(days=30-i), commit_sha=f"abc{i:04d}",
status="success" if i != 5 and i != 12 else "failed"
))
calc.add_incident(IncidentRecord(
incident_id="INC-001", service="api-server",
detection_time=now - timedelta(days=25),
resolution_time=now - timedelta(days=25, hours=-1),
severity="high", root_cause="deploy_related"
))
report = calc.generate_dora_report(service="api-server", days=30)
print(json.dumps(report, indent=2))
print("✅ DORAレポート生成完了！")
CFRとMTTR Prometheusルール
# dora-cfr-mttr-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: dora-cfr-mttr-rules
namespace: monitoring
spec:
groups:
- name: dora_cfr_mttr
interval: 5m
rules:
- record: dora:change_failure_rate:ratio
expr: |
sum(increase(dora_change_failure_total[30d])) / sum(increase(dora_deploy_total{status="success"}[30d]))
- record: dora:mttr:p50
expr: histogram_quantile(0.5, sum(rate(dora_mttr_seconds_bucket[30d])) by (service, le))
- record: dora:mttr:p90
expr: histogram_quantile(0.9, sum(rate(dora_mttr_seconds_bucket[30d])) by (service, le))
- alert: DORAHighChangeFailureRate
expr: dora:change_failure_rate:ratio > 0.15
for: 1h
labels:
severity: warning
annotations:
summary: "変更失敗率が高すぎます"
- alert: DORAHighMTTR
expr: dora:mttr:p90 > 14400
for: 1h
labels:
severity: warning
annotations:
summary: "MTTRが長すぎます"
パターン4：Grafanaダッシュボードとアラート設定
DORAダッシュボードJSON
{
"dashboard": {
"title": "DORA Metrics Dashboard 2026",
"description": "Kubernetes DORA 4メトリクス可視化ダッシュボード",
"tags": ["dora", "devops", "kubernetes"],
"timezone": "browser",
"refresh": "5m",
"panels": [
{
"title": "🏆 DORA Performance Level",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
"targets": [{"expr": "dora:deploy_frequency:daily"}]
},
{
"title": "📊 Deployment Frequency (daily)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
"targets": [{"expr": "sum(increase(dora_deploy_total{status=\"success\"}[1d])) by (service)"}]
},
{
"title": "⏱️ Lead Time for Changes",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
"targets": [
{"expr": "dora:lead_time:p50 / 3600", "legendFormat": "P50"},
{"expr": "dora:lead_time:p90 / 3600", "legendFormat": "P90"}
]
},
{
"title": "❌ Change Failure Rate",
"type": "gauge",
"gridPos": {"h": 8, "w": 6, "x": 0, "y": 12},
"targets": [{"expr": "dora:change_failure_rate:ratio * 100"}]
},
{
"title": "🔧 MTTR",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 6, "y": 12},
"targets": [
{"expr": "dora:mttr:p50 / 60", "legendFormat": "P50"},
{"expr": "dora:mttr:p90 / 60", "legendFormat": "P90"}
]
}
]
}
}
Grafanaダッシュボード自動デプロイ
# grafana-dashboard-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dora-dashboard
namespace: monitoring
labels:
grafana_dashboard: "1"
annotations:
k8s-sidecar-target-directory: "/tmp/dashboards/DORA"
data:
dora-metrics.json: |
{
"dashboard": {
"title": "DORA Metrics Dashboard 2026",
"uid": "dora-metrics-2026",
"tags": ["dora", "devops"],
"panels": [
{
"title": "Deployment Frequency",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
"targets": [{"expr": "sum(increase(dora_deploy_total{status=\"success\"}[1d])) by (service)"}]
}
]
}
}
DORAアラート設定
# dora-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: dora-alerts
namespace: monitoring
spec:
groups:
- name: dora_alerts
rules:
- alert: DORALowDeployFrequency
expr: sum(increase(dora_deploy_total{status="success"}[7d])) by (service) < 1
for: 7d
labels:
severity: info
annotations:
summary: "デプロイ頻度が低すぎます"
- alert: DORAHighLeadTime
expr: dora:lead_time:p90 > 86400
for: 1d
labels:
severity: warning
annotations:
summary: "変更リードタイムが長すぎます"
- alert: DORAHighChangeFailureRate
expr: dora:change_failure_rate:ratio > 0.15
for: 6h
labels:
severity: warning
annotations:
summary: "変更失敗率が高すぎます"
- alert: DORAHighMTTR
expr: dora:mttr:p90 > 14400
for: 6h
labels:
severity: critical
annotations:
summary: "MTTRが長すぎます"
- alert: DORADeployFailureSpike
expr: |
(sum(increase(dora_deploy_total{status="failed"}[1h])) / sum(increase(dora_deploy_total[1h]))) > 0.5
for: 30m
labels:
severity: critical
annotations:
summary: "デプロイ失敗率の急増を検出"
パターン5：本番級DORA測定CI/CD統合
フルチェーンDORAメトリクスパイプライン
#!/usr/bin/env python3
"""
dora_pipeline.py
本番級DORAメトリクスCI/CD統合パイプライン
"""
import os
import sys
import time
import subprocess
import logging
from typing import Dict
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
class DORAPipeline:
def __init__(self):
self.pushgateway_url = os.getenv("PUSHGATEWAY_URL", "localhost:9091")
self.service_name = os.getenv("SERVICE_NAME", "unknown")
self.environment = os.getenv("DEPLOY_ENV", "production")
self.deploy_id = f"deploy-{int(time.time())}"
def get_commit_info(self) -> Dict:
try:
sha = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"], stderr=subprocess.DEVNULL).decode().strip()
timestamp = subprocess.check_output(["git", "log", "-1", "--format=%ct"], stderr=subprocess.DEVNULL).decode().strip()
return {"sha": sha, "timestamp": int(timestamp)}
except Exception as e:
logger.error(f"Git情報の取得に失敗: {e}")
return {"sha": "unknown", "timestamp": int(time.time())}
def record_pipeline_success(self, commit_info: Dict, duration: float) -> None:
now = int(time.time())
lead_time = now - commit_info["timestamp"]
metrics = f'dora_deploy_total{{environment="{self.environment}",service="{self.service_name}",status="success"}} 1\ndora_lead_time_seconds_bucket{{service="{self.service_name}",le="{lead_time}"}} 1\n'
self._push_metrics(metrics)
logger.info(f"Pipeline success: lead_time={lead_time/3600:.2f}h")
def record_pipeline_failure(self, commit_info: Dict, duration: float, error: str) -> None:
metrics = f'dora_deploy_total{{environment="{self.environment}",service="{self.service_name}",status="failed"}} 1\n'
self._push_metrics(metrics)
logger.error(f"Pipeline failed: {error}")
def _push_metrics(self, metrics: str) -> None:
try:
subprocess.run(
["curl", "--data-binary", "@-", "-s", f"http://{self.pushgateway_url}/metrics/job/{self.deploy_id}"],
input=metrics.encode(), capture_output=True, timeout=10
)
except Exception as e:
logger.error(f"メトリクスプッシュエラー: {e}")
def run(self) -> int:
commit_info = self.get_commit_info()
start_time = time.time()
try:
logger.info("Dockerイメージをビルド中...")
time.sleep(2)
logger.info("Kubernetesにデプロイ中...")
time.sleep(3)
duration = time.time() - start_time
self.record_pipeline_success(commit_info, duration)
return 0
except Exception as e:
duration = time.time() - start_time
self.record_pipeline_failure(commit_info, duration, str(e))
return 1
if __name__ == "__main__":
pipeline = DORAPipeline()
sys.exit(pipeline.run())
ArgoCD Application統合
# argocd-dora-integration.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-server
namespace: argocd
annotations:
notifications.argoproj.io/subscribe.on-deployed.slack: devops-alerts
spec:
project: default
source:
repoURL: https://github.com/example/api-server-manifests
targetRevision: main
path: overlays/production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
よくある落とし穴
落とし穴1：デプロイイベントの重複カウント
# ❌ 誤り：集計なしでincrease()を使用
expr: increase(dora_deploy_total[1d])
# ✅ 正解：サービスごとに集計
expr: sum(increase(dora_deploy_total[1d])) by (service)
落とし穴2：リードタイム計算が不正確
# ❌ 誤り：CIパイプライン開始時間を使用
PIPELINE_START=$(date +%s)
# ✅ 正解：Git commitタイムスタンプを起点に使用
COMMIT_TS=$(git log -1 --format=%ct)
PIPELINE_END=$(date +%s)
LEAD_TIME=$((PIPELINE_END - COMMIT_TS))
落とし穴3：CFR関連付けウィンドウが不適切
# ❌ 誤り：関連付けウィンドウが短すぎる
ASSOCIATION_WINDOW = timedelta(hours=1)
# ✅ 正解：24時間ウィンドウで大部分の変更関連障害をカバー
ASSOCIATION_WINDOW = timedelta(hours=24)
落とし穴4：MTTR定義が不統一
# ❌ 誤り：アラートから復旧までの時間のみを計算
mttr = resolution_time - alert_time
# ✅ 正解：異なる復旧フェーズを区別
mttr_detect = response_time - detection_time
mttr_respond = resolution_time - response_time
mttr_total = resolution_time - detection_time
落とし穴5：Pushgatewayメトリクスがクリーンアップされない
# ❌ 誤り：Pushgatewayメトリクスが無限に蓄積
# ✅ 正解：Pushgatewayクリーンアップを設定
spec:
template:
spec:
containers:
- name: pushgateway
args:
- --web.enable-admin-api
- --persistence.interval=5m
エラートラブルシューティング表
| エラー現象 | 考えられる原因 | 診断コマンド | 解決策 |
|---|---|---|---|
| デプロイ頻度が0 | Pushgateway接続失敗 | curl http://pushgateway:9091/metrics |
Pushgateway URLとネットワークを確認 |
| リードタイムが異常に大きい | Git commitタイムスタンプ形式エラー | git log -1 --format=%ct |
Unixタイムスタンプ（秒）を確認 |
| CFRが100%を超える | インシデントとデプロイの重複関連付け | 関連付けロジックを確認 | deploy_idで重複排除 |
| MTTRが0 | 検出時間>復旧時間 | タイムスタンプの順序を確認 | detection < resolutionを確認 |
| Grafanaにデータがない | Prometheusがメトリクスをスクレイプしていない | curl http://prometheus:9090/api/v1/query |
スクレイプ設定を確認 |
| メトリクスラベルが不一致 | CI/CD環境変数の欠落 | echo $SERVICE_NAME |
すべてのパイプラインで環境変数を統一 |
| カウンターがリセット | Pod再起動でカウンターがリセット | promql: resets(dora_deploy_total[1d]) |
increase()でリセットを自動処理 |
| ダッシュボードが空白 | データソース設定エラー | Grafana → Settings → Data Sources | PrometheusデータソースURLを確認 |
| アラートが発火しない | for期間が長すぎる | アラートルールを確認 | for時間を短縮または閾値を調整 |
| 履歴データが消失 | Prometheus retention不足 | retention設定を確認 | retention時間を30d+に増加 |
高度な最適化
1. Traceベースの正確なLead Time
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
attributes/dora:
actions:
- key: dora.deploy_id
from_attribute: deploy.id
action: upsert
exporters:
prometheusremotewrite:
endpoint: http://prometheus:9090/api/v1/write
2. マルチチームDORA比較
- record: dora:deploy_frequency:daily:by_team
expr: |
sum by (team) (
label_replace(
sum(increase(dora_deploy_total{status="success"}[1d])) by (service),
"team", "$1", "service", "(api-server|order-service)"
)
)
3. DORAトレンド予測
- record: dora:lead_time:trend
expr: predict_linear(dora:lead_time:p50[30d], 7*86400)
4. SLO-DORA連動
apiVersion: sloth.slok.dev/v1
kind: PrometheusSLO
spec:
service: "api-server"
slos:
- name: "deploy-reliability"
objective: 99.5
sli:
events:
error_query: sum(increase(dora_deploy_total{status!="success"}[{{ .window }}]))
total_query: sum(increase(dora_deploy_total[{{ .window }}]))
5. 自動化DORAレポート
#!/usr/bin/env python3
"""毎週自動DORAレポートを生成しSlackにプッシュ"""
import requests, os
PROMETHEUS_URL = "http://prometheus:9090"
SLACK_WEBHOOK = os.getenv("SLACK_WEBHOOK_URL")
def query_prometheus(query: str) -> float:
resp = requests.get(f"{PROMETHEUS_URL}/api/v1/query", params={"query": query})
result = resp.json()["data"]["result"]
return float(result[0]["value"][1]) if result else 0
def generate_weekly_report():
df = query_prometheus('sum(increase(dora_deploy_total{status="success"}[7d]))')
lt = query_prometheus("dora:lead_time:p50 / 3600")
cfr = query_prometheus("dora:change_failure_rate:ratio * 100")
mttr = query_prometheus("dora:mttr:p50 / 60")
blocks = [
{"type": "header", "text": {"type": "plain_text", "text": "📊 Weekly DORA Report"}},
{"type": "section", "fields": [
{"type": "mrkdwn", "text": f"*Deploy Frequency:* {df:.1f}/week"},
{"type": "mrkdwn", "text": f"*Lead Time (P50):* {lt:.1f}h"},
{"type": "mrkdwn", "text": f"*Change Failure Rate:* {cfr:.1f}%"},
{"type": "mrkdwn", "text": f"*MTTR (P50):* {mttr:.0f}min"},
]}
]
requests.post(SLACK_WEBHOOK, json={"blocks": blocks})
if __name__ == "__main__":
generate_weekly_report()
比較表
| 次元 | DORA Elite | DORA High | DORA Medium | DORA Low |
|---|---|---|---|---|
| デプロイ頻度 | オンデマンド(複数回/日) | 週次〜月次 | 月次〜半年 | >半年 |
| 変更リードタイム | <1時間 | <1日 | <1週間 | >1ヶ月 |
| 変更失敗率 | <5% | <10% | <15% | >15% |
| MTTR | <1時間 | <1日 | <1週間 | >1ヶ月 |
| 組織特性 | 自律型フルスタックチーム | プラットフォームエンジニアリング支援 | 厳格な承認プロセス | 手動デプロイ |
💡 まとめ：DORAの4つのメトリクスは目的ではなく、継続的改善の手段です。デプロイイベント収集からリードタイム追跡、変更失敗率の関連付けからMTTR測定、GrafanaダッシュボードからCI/CDフルチェーン統合まで——5つのコアパターンが完全なDevOpsパフォーマンス測定体系を構築します。覚えておいてください：測定はチームを評価するためではなく、ボトルネックを発見し改善を推進するためのものです。DORAメトリクスは鏡であり、鞭ではありません。
オンラインツール推奨
- JSONフォーマッター — DORAダッシュボードJSON設定をフォーマット、フォーマットエラーを迅速にトラブルシューティング
- cURL to Code — Prometheus APIクエリをコードに変換、DORAデータを統合
- ハッシュ計算 — デプロイIDハッシュを計算、メトリクスの一意性を確保
ブラウザローカルツールを無料で試す →
#DORA指标#DevOps度量#K8s#CI/CD#2026#DevOps运维