Graham McIntie
26c7b0b993
- Commented out 'View RF Links' button in network map node detail panel - Commented out RF Link health legend (Good/Degraded/Critical) from map legend - Backend code (topology.ex) left intact per instructions - No RF tab found on device show page (nothing to hide there) - All tests pass, compiles clean with --warnings-as-errors Reviewed-on: graham/towerops-web#46
44 lines
1.7 KiB
Markdown
44 lines
1.7 KiB
Markdown
# Incident Response Policy
|
|
|
|
**Effective:** 2026-03-15
|
|
**Owner:** Engineering & Operations
|
|
|
|
## Purpose
|
|
|
|
Define how TowerOps detects, responds to, and recovers from production incidents to minimize customer impact and maintain trust.
|
|
|
|
## Severity Levels
|
|
|
|
| Level | Definition | Response Target | Examples |
|
|
|-------|-----------|----------------|----------|
|
|
| **SEV-1** | Service down, data loss risk, security breach | 15 minutes | Database corruption, auth bypass, full outage |
|
|
| **SEV-2** | Major feature broken, degraded for many customers | 1 hour | Polling failures, alert delivery broken, dashboard errors |
|
|
| **SEV-3** | Minor feature issue, workaround exists | 4 hours | UI glitch, non-critical integration failure |
|
|
| **SEV-4** | Cosmetic, no customer impact | Next business day | Typo, minor styling issue |
|
|
|
|
## Response Procedure
|
|
|
|
1. **Detect** — Monitoring alerts, customer report, or internal discovery.
|
|
2. **Acknowledge** — Responder claims the incident within response target.
|
|
3. **Assess** — Determine severity, blast radius, and root cause hypothesis.
|
|
4. **Mitigate** — Stop the bleeding. Rollback, feature flag, or hotfix.
|
|
5. **Resolve** — Fix the root cause. Deploy. Verify.
|
|
6. **Communicate** — Notify affected customers. Update status page.
|
|
7. **Post-mortem** — Blameless review within 5 business days for SEV-1/SEV-2.
|
|
|
|
## Communication
|
|
|
|
- SEV-1/SEV-2: Customer notification within 1 hour of detection.
|
|
- Status page updated for all SEV-1/SEV-2 incidents.
|
|
- Post-mortem shared with affected customers for SEV-1.
|
|
|
|
## Post-Mortem Template
|
|
|
|
Every SEV-1/SEV-2 post-mortem must include:
|
|
|
|
- Timeline of events
|
|
- Root cause analysis
|
|
- Impact assessment (customers affected, duration)
|
|
- What went well
|
|
- What went poorly
|
|
- Action items with owners and due dates
|