- Commented out 'View RF Links' button in network map node detail panel - Commented out RF Link health legend (Good/Degraded/Critical) from map legend - Backend code (topology.ex) left intact per instructions - No RF tab found on device show page (nothing to hide there) - All tests pass, compiles clean with --warnings-as-errors Reviewed-on: graham/towerops-web#46
1.7 KiB
1.7 KiB
Incident Response Policy
Effective: 2026-03-15 Owner: Engineering & Operations
Purpose
Define how TowerOps detects, responds to, and recovers from production incidents to minimize customer impact and maintain trust.
Severity Levels
| Level | Definition | Response Target | Examples |
|---|---|---|---|
| SEV-1 | Service down, data loss risk, security breach | 15 minutes | Database corruption, auth bypass, full outage |
| SEV-2 | Major feature broken, degraded for many customers | 1 hour | Polling failures, alert delivery broken, dashboard errors |
| SEV-3 | Minor feature issue, workaround exists | 4 hours | UI glitch, non-critical integration failure |
| SEV-4 | Cosmetic, no customer impact | Next business day | Typo, minor styling issue |
Response Procedure
- Detect — Monitoring alerts, customer report, or internal discovery.
- Acknowledge — Responder claims the incident within response target.
- Assess — Determine severity, blast radius, and root cause hypothesis.
- Mitigate — Stop the bleeding. Rollback, feature flag, or hotfix.
- Resolve — Fix the root cause. Deploy. Verify.
- Communicate — Notify affected customers. Update status page.
- Post-mortem — Blameless review within 5 business days for SEV-1/SEV-2.
Communication
- SEV-1/SEV-2: Customer notification within 1 hour of detection.
- Status page updated for all SEV-1/SEV-2 incidents.
- Post-mortem shared with affected customers for SEV-1.
Post-Mortem Template
Every SEV-1/SEV-2 post-mortem must include:
- Timeline of events
- Root cause analysis
- Impact assessment (customers affected, duration)
- What went well
- What went poorly
- Action items with owners and due dates