Time to Restore Service Degradation from Missing Incident Correlation

critical

reliabilityUpdated Apr 5, 2024

Mean Time to Recovery (MTTR) increases because deployment-related incidents aren't automatically correlated with the causing deployment, delaying identification of the problematic change and slowing rollback decisions.

Sources

Deployment Pipeline Observability - GitLab Release Documentationgitlab-org.gitlab.io

🦊 GitLab: A Python Script Calculating DORA Metricsdev.to

Technologies:

GitLab CISymptoms of this issue are visible in GitLab CI metrics and logs

GrafanaGrafana metrics correlate with this issue and help confirm diagnosis

How to detect:

Track time between incident creation and deployment rollback or fix deployment. Monitor incidents that occur within 1 hour of deployments but take >30 minutes to resolve. Measure correlation rate between incidents and recent deployments. Alert when MTTR exceeds SLO for deployment-related incidents.

Recommended action:

Implement deployment annotations in monitoring dashboards (Grafana) to visualize deployment events alongside error rates and incidents. Use GitLab's release evidence and deployment tracking to automatically tag incidents with suspect deployments. Enable automatic rollback triggers based on error rate thresholds post-deployment. Integrate incident management with deployment pipelines to surface recent changes during incident response.