# Incident Response Policy

**Effective:** 2026-03-15
**Owner:** Engineering & Operations

## Purpose

Define how TowerOps detects, responds to, and recovers from production incidents to minimize customer impact and maintain trust.

## Severity Levels

| Level | Definition | Response Target | Examples |
|-------|-----------|----------------|----------|
| **SEV-1** | Service down, data loss risk, security breach | 15 minutes | Database corruption, auth bypass, full outage |
| **SEV-2** | Major feature broken, degraded for many customers | 1 hour | Polling failures, alert delivery broken, dashboard errors |
| **SEV-3** | Minor feature issue, workaround exists | 4 hours | UI glitch, non-critical integration failure |
| **SEV-4** | Cosmetic, no customer impact | Next business day | Typo, minor styling issue |

## Response Procedure

1. **Detect** — Monitoring alerts, customer report, or internal discovery.
2. **Acknowledge** — Responder claims the incident within response target.
3. **Assess** — Determine severity, blast radius, and root cause hypothesis.
4. **Mitigate** — Stop the bleeding. Rollback, feature flag, or hotfix.
5. **Resolve** — Fix the root cause. Deploy. Verify.
6. **Communicate** — Notify affected customers. Update status page.
7. **Post-mortem** — Blameless review within 5 business days for SEV-1/SEV-2.

## Communication

- SEV-1/SEV-2: Customer notification within 1 hour of detection.
- Status page updated for all SEV-1/SEV-2 incidents.
- Post-mortem shared with affected customers for SEV-1.

## Post-Mortem Template

Every SEV-1/SEV-2 post-mortem must include:

- Timeline of events
- Root cause analysis
- Impact assessment (customers affected, duration)
- What went well
- What went poorly
- Action items with owners and due dates