- Wrap agent name + icon inside the offline amber badge so it's clear
what is offline on the device detail page
- Add "Agent:" label before the agent indicator for all states
- Fix TimeSeriesTest deadlock: switch to async: false since check_results
is a TimescaleDB hypertable and concurrent chunk creation deadlocks
- Fix unicode property test: String.length/1 counts grapheme clusters,
not code points; combining chars can make a 2-codepoint string have
only 1 grapheme, legitimately failing the min: 2 validation
Reviewed-on: graham/towerops-web#83
## Summary
- **`list_checks_for_agent` cascade**: the agent query only matched checks with `agent_token_id` set directly on the check record — checks on devices inheriting their agent via site/org/global default were never sent to the Go agent. Now uses left joins with the full cascade (device assignment → site → org default → global default cloud poller), matching the pattern from `list_agent_polling_targets`.
- **At-risk summary double-counting**: `get_impact_summary` called `analyze_device_impact` per down AP, and each returned the full site subscriber count. With 3 APs down at a 147-sub site it showed 441 subs / $31k instead of 147 / $10k. Fixed by deduplicating impact per site. Same fix applied to `calculate_down_impact` in site impact summaries.
## Test plan
- [x] 3 new cascade tests for `list_checks_for_agent` (site, org default, global default)
- [x] 3 new tests for `get_impact_summary` (single site dedup, multi-site sum, zero case)
- [x] All 8,607 existing tests pass
- [ ] Deploy to staging, verify At Risk numbers on dashboard
- [ ] Verify Go agent receives DNS/HTTP checks via cascade
Reviewed-on: graham/towerops-web#79
## Summary
- Replace guard-clause-based `should_skip_snmp/service/ping_check?` with a single `should_skip_check?` function using runtime `in` operator checks
- Guard clauses with module attributes (`@service_check_types`) were not working in Dokku buildpack deployments despite code being correct locally
- Also update `CheckWorker` (dead code safety net) to check `device_has_effective_agent?`
- Use `Logger.info` for skip messages so they appear in staging logs
## Context
PR #77 introduced `device_has_effective_agent?/1` and updated all workers, but the guard-clause-based skip functions in `CheckExecutorWorker` weren't matching at runtime on staging despite passing all tests locally. The DNS check kept producing NXDOMAIN errors from Erlang's `:inet_res` resolver.
Root cause: The `when check_type in @service_check_types` guard clause compiled against module attributes appears to have stale evaluation in the Dokku Heroku buildpack release. Moving the check to a runtime function body fixes the issue.
## Test plan
- [x] All 230 worker tests pass
- [x] All 99 agents tests pass
- [x] Verified on staging: "Skipping check eceb3d51... (dns)" appears in info logs
- [x] No new NXDOMAIN results created after deploy
Reviewed-on: graham/towerops-web#78
## Summary
- Add `device_has_effective_agent?/1` to Agents context that checks full cascade (device → site → org → global default) without distinguishing between cloud pollers and local agents
- Update check_executor_worker, device_poller_worker, and device_monitor_worker to use new function
- Phoenix never executes checks (DNS, ping, HTTP, TCP, SSL, SNMP) when any agent handles the device
## Problem
The global default cloud poller was not being checked in `should_phoenix_poll_device?/1`, causing Phoenix to run DNS/ping/service checks server-side even when the Go agent was handling the device. This produced duplicate results with NXDOMAIN errors from the Erlang DNS resolver.
## Test plan
- [x] 6 new tests for `device_has_effective_agent?/1` covering all cascade levels
- [x] Full test suite passes (8601 tests)
- [ ] Verify staging DNS check stops showing NXDOMAIN errors after deploy
Reviewed-on: graham/towerops-web#77
Chart.js was showing integer-rounded values like '2' instead of '2.1 ms'. Uses toFixed(1) in tick callbacks for proper decimal display with units.
Reviewed-on: graham/towerops-web#76
DNS check graph now shows two datasets: Response Time (ms) on left y-axis and Status (Up/Down) on right y-axis with stepped green fill. Also adds space before units in chart labels.
Reviewed-on: graham/towerops-web#74
Check.changeset/2 did not include state fields (current_state,
current_state_type, current_check_attempt, last_check_at, etc.) in
its cast list. Ecto silently dropped all state updates, so
update_check_state succeeded but wrote nothing to the database.
Added Check.state_changeset/2 for internal state machine updates,
keeping the regular changeset safe from user input. Updated
Monitoring.update_check_state/3 to use it directly.
Reviewed-on: graham/towerops-web#73
## Summary
- **DNS checks**: `parse_server` returned a flat 5-tuple `{a,b,c,d,53}` instead of `{{a,b,c,d}, 53}` — the format `:inet_res` requires for nameservers. All DNS checks using a custom server failed with argument error.
- **Ping checks**: `System.cmd` `:timeout` option not supported on Elixir 1.19.4 (staging). Replaced with `Task.async`/`Task.yield` wrapper for cross-version compatibility.
- **Checks table UI**: Removed unnecessary flex div wrapper on status badge column that caused vertical misalignment with other table cells.
## Test plan
- [x] DNS executor tests pass (custom server resolution now succeeds)
- [x] Ping executor tests pass (no more ArgumentError)
- [x] Full test suite passes (8593 tests, 0 new failures)
Reviewed-on: graham/towerops-web#72
## Summary
- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever
## Changes
- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience
## Test plan
- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)
Reviewed-on: graham/towerops-web#71
## Summary
- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)
## Test plan
- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal
Reviewed-on: graham/towerops-web#68
add ssl check executor that connects via TLS, reads peer certificate
expiry, and returns OK/WARNING/CRITICAL based on configurable
warning_days threshold. includes protobuf config, liveview form fields,
agent channel encoding, oban worker dispatch, and unit tests.
Reviewed-on: graham/towerops-web#64
- Skip HTTP/TCP/DNS checks in Phoenix when device is assigned to agent
- Agents now execute service checks from the device's network location
- Fix cancel button in service check modal (text link instead of button)
- Fix status badge alignment in checks table
Reviewed-on: graham/towerops-web#63
add early bail in DiscoveryWorker when snmp_enabled is false.
mark snmp_not_enabled and device_unresponsive as non-retryable
errors to prevent wasted retries.
Reviewed-on: graham/towerops-web#58
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
send check_jobs protobuf to agents during job dispatch cycle, handle
check_result messages back with state transitions and alert
creation/resolution. auto-assign checks to device's effective agent
token on creation, broadcast check changes via PubSub so connected
agents get immediate updates.
add edit/delete support for service checks on device show page with
form pre-population from existing check config. auto-fill TCP host
from device IP address.
includes validate_check_result/1 for protobuf validation and
list_checks_for_agent/2 for querying agent-assigned service checks.
also fixes pre-existing broken device_type_icon reference in device
index template.
Reviewed-on: graham/towerops-web#50
Extend the GraphQL API and socket to accept mobile session tokens
alongside existing API tokens. Add OrganizationScope middleware
that extracts organization_id from query args for mobile users.
Add my_organizations query for mobile app org listing.
Reviewed-on: graham/towerops-web#48
## Summary
Hide RF link-related UI elements from the network map page while keeping all backend code intact.
### Changes
- Commented out "View RF Links" button in network map node detail panel
- Commented out RF Link health legend (Good signal / Degraded / Critical) from map legend
- Backend topology.ex code untouched — `compute_rf_link_stats/1` and `enrich_edges_with_rf/2` remain
- No RF tab exists on the device show page, so no changes needed there
### Verification
- `mix compile --warnings-as-errors` — zero warnings
- `mix test` — 8490 tests pass, 0 failures
Paperclip issue: TOW-44
Reviewed-on: graham/towerops-web#45
the complete_qr_login endpoint was returning session.token (the SHA256
hash stored in the database) instead of session.raw_token (the plaintext).
this caused all subsequent API calls and WebSocket connections to fail
with 401/REFUSED since the server would hash the already-hashed token.
adds a regression test that verifies the returned token is usable for
authenticated API requests.
Reviewed-on: graham/towerops-web#42
- Changed sidebar from fixed to sticky positioning
- Wrapped sidebar and main content in flex container
- Sidebar now sticks to top along with header when scrolling
- Removed unused Ecto.Query import from gps_sync.ex
Credo improvements (17 → 0 remaining):
- Replaced all length/1 checks with empty list comparisons
- Extracted helper functions to reduce nesting depth
- Split complex functions into smaller, testable units
- Applied 'with' statements for clearer control flow
- Refactored high-complexity functions (cyclomatic complexity)
- Files improved: storm_detector, alert_notification_worker,
alert_digest_worker, gps_sync, statistics_sync, device_sync,
site_sync, site_correlation, reports_live, topology,
pagerduty/client, cn_maestro/sync
Reviewed-on: graham/towerops-web#32
The StormDetector was being added twice:
1. In the main children list (line 127)
2. In background_workers() (line 176)
This caused the supervisor to fail with 'duplicate_child_name' error.
Removed the duplicate from main children list since it's correctly
placed in background_workers() which also handles test env properly.
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- Add 1.5s debounce to dashboard LiveView for alert events, preventing
excessive re-renders during mass outages where hundreds of alerts fire
simultaneously
- Replace global 'device:events' PubSub topic with org-scoped
'device_events:org:{id}' topics across all broadcasters (device_poller_worker,
sensor_change_detector, discovery) so LiveView processes only receive
events for their tenant
- Update EventLogger GenServer to subscribe to per-org topics dynamically,
with runtime subscription support for newly created orgs
- Add Organizations.list_organization_ids/0 for EventLogger bootstrap
- Increase default DB pool_size from 20 to 25 for production concurrent users
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Tower/device icons: role-based shapes (triangle for APs/wireless,
diamond for switches, round-rect for routers, hexagon for firewalls)
- RF link overlays: edges colored by signal health (green/yellow/red)
with bandwidth-proportional line thickness
- Enhanced hover tooltips: device-to-device name, SNR dB, bandwidth,
signal health indicator
- Wireless client count badge on AP nodes
- Filter bar: All Links | Degraded Only | Sites with Alerts | Search
- Search with zoom-to-match and highlight
- Geographic layout toggle (when sites have lat/lng)
- Enhanced detail panel: RF stats (client count, signal health, SNR),
View RF Links button for wireless devices
- Backend: compute_wireless_stats/compute_rf_link_stats enrich topology
- Site nodes include lat/lng for geographic layout
- Edge enrichment with RF signal health from SNR sensors
- Fix pre-existing compile error in notification_rate_limiter.ex
- Increase node font from 11px to 13px, compound labels to 14px
- Replace cose physics layout with deterministic grid preset layout to eliminate site group overlap
- Fix nil source_interface_id crash in topology.ex find_existing_link/1 (was using == nil instead of is_nil/1)
- Fix nil device label in device_to_node/1 fallback to IP or "Unknown"
- LIKE wildcard injection: sanitize %, _ in search queries (devices, sites, gaiia)
- Jason.decode! → Jason.decode with error handling for untrusted input
- inspect() leak: replace with generic error messages, log details server-side
- SSRF protection: URL validator blocks private IPs, localhost, non-HTTP schemes
- SSRF validation added to HTTP monitoring executor and integration credentials
- GraphQL complexity limits: always applied, not just in prod
- GraphQL introspection: also check GET query params, not just body
- Stripe webhook: explicit nil/empty checks for signature and body
- Cookie security: secure flag for session (prod), http_only+secure for remember_me
- Honeybadger API key: read from env var with fallback
- String.to_integer → Integer.parse with fallback for URL params
- String.to_atom → whitelist map for HTTP methods
- Gaiia webhook: remove secret_len and expected signature from log
- Admin API: add rate limiting pipeline
- to_atom_keys: per-key fallback instead of all-or-nothing rescue
- add MobileSocket for token-authenticated mobile WebSocket connections
- add MobileChannel with org-level and device-level real-time events
- add /mobile/socket endpoint for iOS app connections
- add "Link Mobile App" to user dropdown menu for discoverability
- improve QR code page with numbered steps and clearer instructions
- add socket/channel tests (17 total)
When alert routing is set to "builtin" (TowerOps) but no escalation
policy exists, the checklist item now reads "Create an escalation
policy" and links directly to the new escalation policy page instead
of looping back to general settings where the routing already looks
configured.
The sync schedule was shown twice on non-exclusive integration cards
(inline status row + separate box). Remove the box to match the
uniform inline display used by the billing card.
Adjust sidebar bottom section padding from py-3 to py-3.5 to align
the horizontal border with the footer border. Also remove unused
private functions from settings_live and fix credo warnings.
- Replace inline integrations tab with link to /settings/integrations
- Remove ~1500 lines of duplicated integration UI from settings
- Remove duplicated integration event handlers from settings_live
- Standalone IntegrationsLive page remains the single source of truth
- Settings page down from ~2400 to ~900 lines
- Setup Progress bar on dashboard when devices exist but setup incomplete
- Tracks: devices added, billing connected, alert routing configured,
on-call schedule created
- Each step links to the relevant setup page
- Dismissible with X button
- Progress bar shows visual completion percentage
- Optional steps labeled (billing, on-call)
- New 'alert_routing' field on organizations (default: builtin)
- AlertNotificationWorker respects the setting:
- builtin: only built-in escalation policies
- pagerduty: only PagerDuty events
- both: fire both (useful during migration)
- Radio button UI in Settings → General with visual cards
- Warning shown when PagerDuty selected but not configured
- Migration, schema validation, 6 tests
LiveView DOM patching was stripping the client-side sidebar-collapsed
class from #sidebar-wrapper on every navigation. Fix: put the class
on document.documentElement which LiveView never touches.
- Sync script in <head> applies state before first paint (no flash)
- Hook click handler toggles on <html> + saves to localStorage
- CSS selectors updated to html.sidebar-collapsed
- Removed phx-click JS.toggle_class (hook handles toggle directly)
