towerops/policies/INCIDENT_RESPONSE.md
Graham McIntie 26c7b0b993 Hide RF Links UI elements from network map (TOW-44) (#46)
- Commented out 'View RF Links' button in network map node detail panel
- Commented out RF Link health legend (Good/Degraded/Critical) from map legend
- Backend code (topology.ex) left intact per instructions
- No RF tab found on device show page (nothing to hide there)
- All tests pass, compiles clean with --warnings-as-errors

Reviewed-on: graham/towerops-web#46
2026-03-16 15:19:03 -05:00

1.7 KiB

Incident Response Policy

Effective: 2026-03-15 Owner: Engineering & Operations

Purpose

Define how TowerOps detects, responds to, and recovers from production incidents to minimize customer impact and maintain trust.

Severity Levels

Level Definition Response Target Examples
SEV-1 Service down, data loss risk, security breach 15 minutes Database corruption, auth bypass, full outage
SEV-2 Major feature broken, degraded for many customers 1 hour Polling failures, alert delivery broken, dashboard errors
SEV-3 Minor feature issue, workaround exists 4 hours UI glitch, non-critical integration failure
SEV-4 Cosmetic, no customer impact Next business day Typo, minor styling issue

Response Procedure

  1. Detect — Monitoring alerts, customer report, or internal discovery.
  2. Acknowledge — Responder claims the incident within response target.
  3. Assess — Determine severity, blast radius, and root cause hypothesis.
  4. Mitigate — Stop the bleeding. Rollback, feature flag, or hotfix.
  5. Resolve — Fix the root cause. Deploy. Verify.
  6. Communicate — Notify affected customers. Update status page.
  7. Post-mortem — Blameless review within 5 business days for SEV-1/SEV-2.

Communication

  • SEV-1/SEV-2: Customer notification within 1 hour of detection.
  • Status page updated for all SEV-1/SEV-2 incidents.
  • Post-mortem shared with affected customers for SEV-1.

Post-Mortem Template

Every SEV-1/SEV-2 post-mortem must include:

  • Timeline of events
  • Root cause analysis
  • Impact assessment (customers affected, duration)
  • What went well
  • What went poorly
  • Action items with owners and due dates