Prod paired the Oban Pro Smart engine + unique workers with core
Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An
orphaned executing CheckExecutorWorker job rescued back to available collides
with its already-scheduled successor on oban_jobs_unique_index (23505),
crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to
Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via
Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine).
Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness
and startup probes now target it so a transient dependency outage can't kill or
block boot of a healthy pod; readiness keeps the deep /health to gate the load
balancer. (deployment.yaml probe/replica change pushed separately, after the
image carrying /health/live is live.)
Also drop unused POSTGRES_* keys from the secrets example.
- Add organization membership check before API token creation
- Fix admin security allowlist form reading current_user instead of current_scope.user
- Use recursive File.ls instead of Path.wildcard to include hidden files in MIB validation
- Add upload size check before ZIP extraction in MIB controller
- Centralize CSP in SecurityHeaders plug with per-request nonces instead of 'unsafe-inline'
cnHeat-2.0-style automation hooks. Coverage prediction is now a
first-class API resource that Terraform / Ansible / cron can drive.
REST API (under /api/v1, Bearer-token auth scoped to one org)
- GET /coverages list
- POST /coverages create + auto-queue compute
- GET /coverages/:id read (includes status / progress_pct
for polling-based providers)
- PATCH /coverages/:id update editable fields
- DELETE /coverages/:id delete + cleanup
- POST /coverages/:id/recompute requeue, returns 202
- GET /coverages/:id/kmz Google Earth bundle (KML + PNG)
Each response includes the full resource — Terraform read-after-write
reconciles cleanly; status polling hits a stable JSON shape.
Email-on-complete (Towerops.Coverages.Notifier)
- Worker fires deliver_compute_complete on both :ready and :failed
paths. Sends a short text email to every member of the owning
organisation with a link to the show page (or the failure
reason). Mirrors cnHeat 2.0's per-project email alerts.
- Failures are non-fatal: a stuck SMTP relay never blocks the
underlying coverage record from transitioning state.
KMZ export
- Builds a flat KMZ in-memory via :zip.create — doc.kml +
overlay.png, with the LatLonBox set from the coverage's bbox.
XML special chars are escaped. 409 if not yet ready.
Tests (11 new, all green)
- Bearer-token auth happy path + 401 missing
- index / show / create / update / delete + 404 cross-org
- create returns status:'queued' confirming the auto-enqueue
- recompute returns 202 with status:'queued'
Plus docs/terraform-coverages.md showing how to drive the new API
from a third-party restapi provider until the first-party
terraform-provider-towerops gets a coverage resource.
- /coverage now renders a full-bleed Leaflet map of every ready
coverage in the org as one composite heatmap; the prior table view
moves to /coverage/list.
- Top-right tower toggle list with Show All / Hide All. Disabled
towers grey out and their overlay/marker drop from the map.
- Bottom-center opacity slider controls all overlays in unison; the
cnHeat dBm legend renders next to it.
- Top-left base-layer toggle (OSM streets / Esri satellite) and a
Nominatim address search bar that drops a marker.
- Click anywhere on the map → Distance Tool side panel with each
tower sorted by distance and the predicted RSSI at that point
(read from each coverage's GeoTIFF via gdallocationinfo). ft/m
unit toggle, NA for out-of-bbox, color-coded by signal quality.
- Backend additions: Coverages.list_ready_for_organization/1,
Coverages.query_point/3, Raster.query_rssi/3 (gdallocationinfo
wrapper with bbox guard, NoData sentinel handling).
- New JS hook MultiCoverageMap manages the layer set, toolbar,
search, opacity, click-to-probe, and live updates via PubSub
(coverage:enabled-changed and coverage:list-changed).
End-to-end CRUD scaffold for per-site, per-antenna RF coverage
prediction. Schema, context, antenna registry, Oban worker stub, and
three LiveViews are wired up; the actual ITM + LIDAR + buildings
compute pipeline lands in a follow-up.
- Migration + Coverage schema with full validation (RF params, pixel-
budget cap, antenna_slug existence). organization_id excluded from
cast (programmatic-only).
- Coverages context: org-scoped CRUD, validate_site_in_organization
rejects cross-tenant site_id refs at insert/update, queue_compute
with PubSub broadcast on per-coverage and per-org topics.
- MSI Planet .ant parser + persistent_term registry loaded at boot
from priv/antennas/. Two synthetic .ant fixtures (omni + 90deg
sector) bundled.
- CoverageWorker on new :coverage Oban queue (concurrency 2). Job
args carry organization_id; worker refuses to run on org mismatch.
Stub flips status to "failed" with "compute not yet implemented"
so the UI flow is exercisable.
- LiveViews: index (org-wide, live status updates), form (antenna
picker grouped by manufacturer), show (status banners, params
panel, map placeholder). Sidebar nav link added.
- Routes /coverage, /coverage/new, /coverage/:id, /coverage/:id/edit
under :require_authenticated_user_with_default_org.
- 41 new tests covering schema validation, .ant parsing, context
CRUD with cross-org guards, Oban job enqueue.
- Create WeathermapLive module at /weathermap with 60s auto-refresh
- Extend Topology.get_topology_for_weathermap with bandwidth utilization data
- Add WeathermapViewer JavaScript hook with color-coded edges:
* Green (0-30%) → Yellow (30-60%) → Orange (60-80%) → Red (80%+)
- Display throughput/capacity labels on edges (e.g., '847 Mbps / 1 Gbps')
- Add utilization stats dashboard with link count by usage level
- Support full-screen mode for NOC displays
- Integrate with existing dark mode and responsive design
- Add navigation links in sidebar and mobile menu
Technical implementation:
- Leverages existing Capacity module for interface utilization calculations
- Extends topology edges with utilization_level, utilization_pct, throughput_bps
- Builds on NetworkMapLive foundation with enhanced edge styling
- Auto-refresh via Phoenix PubSub and 60s timer
- Filter support for high/critical utilization links
Reviewed-on: graham/towerops-web#172
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
Extend the GraphQL API and socket to accept mobile session tokens
alongside existing API tokens. Add OrganizationScope middleware
that extracts organization_id from query args for mobile users.
Add my_organizations query for mobile app org listing.
Reviewed-on: graham/towerops-web#48
REST endpoints for CRUD, scoped filtering (active/upcoming/past).
GraphQL queries and mutations for maintenance windows.
Fix agent edit test assertion message.
REST API v1 endpoints for schedules (14 endpoints) and escalation
policies (10 endpoints) with full CRUD for nested resources (layers,
members, overrides, rules, targets). GraphQL types, resolvers, and
schema mutations for both resource trees.
Built-in PagerDuty-equivalent system for on-call scheduling and alert
escalation. Users can now manage schedules, rotation layers, overrides,
and escalation policies directly in the app alongside PagerDuty.
- On-call schedules with rotation layers (daily/weekly/custom), member
management, and temporary overrides
- Escalation policies with ordered rules, timeout-based escalation,
and user/schedule targets
- Automatic escalation via Oban worker with configurable repeat count
- Email notifications via Swoosh for on-call alerts
- Resolver computes who's on-call from layer stacking and overrides
- AlertNotificationWorker integration: starts escalation alongside
PagerDuty, acknowledges/resolves incidents on alert state changes
- Device and organization schemas support escalation_policy_id
- Escalation policy picker on device form
- Schedules nav item with tabbed index (schedules + escalation policies)
- Full CRUD UI for schedules, layers, members, overrides, rules, targets
- 62 LiveView tests, 56 context/schema/resolver/escalation tests
- 26 E2E Playwright tests for smoke and critical path coverage
Add CapacityInsightWorker (every 15 min) that generates critical/warning
insights when backhaul utilization exceeds 90%/75% and auto-resolves
when it drops below 70%.
Add capacity and utilization columns to the device ports tab with
set/clear capacity controls. Add organization-level /capacity page
with summary cards and per-site capacity table. Add capacity summary
card to site show page. Add Capacity link to nav menu.
- Add :admin type to RateLimit plug with 100 requests per minute per IP
- Apply rate limiting to all /admin routes (LiveView and controller actions)
- Protects superuser endpoints from abuse and brute force attempts
- Aligns with existing auth (10 req/min) and API (1000 req/min) limits
The select_org LiveView event was updating default_organization_id in
the database but not the session's current_organization_id. Since
load_default_organization prioritizes the session value, users would
always get redirected back to the stale org.
Replaced with a controller POST action that updates both the session
and DB default before redirecting.
Replace the HoneybadgerNoticeFilter that emailed raw stacktraces on
every exception with ErrorTracker for self-hosted error tracking.
Honeybadger itself is retained. ErrorTracker dashboard is mounted at
/admin/errors behind the superuser auth wall.
CRITICAL fixes:
- Mask SNMP community string in agent channel logs (CRITICAL-1)
- Remove snmpv3 passwords from REST API responses, return _set booleans (CRITICAL-2)
- Replace snmp_community with snmp_community_set in GraphQL type (CRITICAL-3)
HIGH fixes:
- Fix cookie same_site from invalid 'Towerops' to 'Lax' (HIGH-4)
- Remove unsafe-eval from CSP script-src (HIGH-6)
- Block GraphQL introspection queries in production (HIGH-7)
- Sanitize LIKE wildcards in SNMP device name search (HIGH-8)
- Reject webhooks when no secret configured instead of accepting (HIGH-9)
MEDIUM fixes:
- Hash mobile session tokens (SHA-256) before DB storage (MEDIUM-10)
- Apply security headers in all environments, not just prod (MEDIUM-14)
- Add GraphQL query complexity limit (500) in production (MEDIUM-16)
- Fix X-Frame-Options to DENY to match frame-ancestors 'none' (MEDIUM-13)
- Remove auto-confirmation on registration (register_user and register_user_with_organization)
- Add deliver_user_confirmation_instructions/2 and confirm_user/1 to Accounts context
- Add verify_email_token_query/2 and by_user_and_contexts_query/2 to UserToken
- Make deliver_confirmation_email public in UserNotifier
- Send confirmation email after registration in both flows
- Block unconfirmed users at password login with helpful message
- Add UserConfirmationController with confirm/resend routes
- Add resend confirmation link to login page
- Update test fixtures to confirm users after registration
- ChangelogParser: parses priv/static/changelog.txt into structured entries
- ChangelogLive: timeline view with DaisyUI styling
- Route at /changelog in authenticated scope
- What's New link in org dropdown menu
- Promote Insights to top-level nav at /insights with source/urgency/status filters
- Unified insights page shows all sources (Preseem, Gaiia, SNMP, System)
- Bulk select/dismiss support
- Integration pages now show sync schedule and next sync estimate
- Expose sync_interval_minutes as editable config field
- Add tab navigation between integration sub-pages
- Dashboard insights section links to /insights
- Integration 'Network Insights' link now points to /insights?source=preseem
- Add product roadmap (ROADMAP.md)
This commit implements the unified checks architecture that consolidates
SNMP monitoring with HTTP/TCP/DNS checks under a single "check" abstraction,
enabling consistent graphing, alerting, and management across all check types.
Database Changes:
- Add source_type and source_id to checks table for tracking auto-discovered
vs manually created checks
- Add value field to check_results for storing numeric sensor readings
- Maintain backward compatibility with existing check_results data
New SNMP Executors:
- SnmpSensorExecutor: Poll sensor OIDs and return formatted values with
status determination (OK/WARNING/CRITICAL based on limits)
- SnmpInterfaceExecutor: Poll interface stats (bandwidth, packets, errors)
- SnmpProcessorExecutor: Poll CPU/processor usage
- SnmpStorageExecutor: Poll disk/memory usage with percentage calculations
Check Execution Worker:
- CheckExecutorWorker: Unified Oban worker that dispatches to appropriate
executor based on check_type (snmp_sensor, snmp_interface, http, tcp, etc.)
- Self-schedules next execution with distributed polling offsets
- Records results in check_results TimescaleDB hypertable
- Updates check state (OK/WARNING/CRITICAL/UNKNOWN)
Discovery Integration:
- Auto-creates checks during SNMP discovery for sensors, interfaces,
processors, and storage
- Links checks to source entities via source_id for data lookup
- Enables/disables checks based on discovery results
UI Enhancements:
- Checks tab on device detail page with grouped display
- FormComponent for adding manual HTTP/TCP/DNS checks
- Empty state with "Run Discovery" prompt
- Check status badges and last checked times
Graphing:
- Update GraphLive to accept check_id parameter
- Query check_results table for time-series data
- Support all check types (SNMP, HTTP response times, etc.)
Testing:
- Comprehensive test suite for SnmpSensorExecutor (5 tests)
- Test suite for CheckExecutorWorker (7 tests)
- Test coverage for discovery check creation (6 tests)
- Remove deprecated monitoring_test.exs testing old API
Bug Fixes:
- Fix SNMP executors reading credentials from correct Device schema fields
(device.snmp_version instead of device.snmp_device.version)
- Update agent channel test to query MonitoringCheck table directly
Code Quality:
- Extract add_snmp_credentials helper to reduce cyclomatic complexity
- Use map-based lookups for sensor formatting and check type grouping
- Apply pattern matching in dispatcher to reduce complexity
- All credo checks passing with no issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
POST /api/v1/webhooks/agent-release triggers all connected agents
to self-update via their existing WebSocket channels. Authenticated
with a shared secret (AGENT_WEBHOOK_SECRET env var).
- Add ReleaseChecker.invalidate_cache/0 for fresh GitHub fetch
- Add Agents.list_updatable_agents/0 (enabled + seen < 10min)
- Add Agents.broadcast_mass_update/0 orchestration function
- Add WebhookAuth plug with timing-safe secret comparison
- Add AgentReleaseWebhookController with :webhook pipeline
- Configure AGENT_WEBHOOK_SECRET in dev/test/runtime configs
