Alert Suppression During Maintenance Windows Prevents Incident Detection

During a scheduled maintenance window, the lead SRE acknowledges and silences all subsequent alerts at 12:30 AM, assuming the server reboot behavior is normal. The actual outage begins at 12:26 AM but is not recognized until 2:30 AM when customers cannot reach the site, resulting in a 2-hour delay in incident response.

1. Establish alert classification policies that distinguish between expected maintenance behaviors and actual failures. 2. Require alert review before suppression during maintenance windows - do not auto-silence based on assumptions. 3. Implement continuous health checks during maintenance that validate service availability independent of reboot status. 4. Create escalation procedures that re-alert if service remains unavailable beyond expected maintenance duration. 5. Update on-call procedures to ensure 24-hour operations coverage during scheduled maintenance windows.