Commit graph

631 commits

Author SHA1 Message Date
a97340873f feat: Add network weathermap with real-time utilization visualization (#172)
- Create WeathermapLive module at /weathermap with 60s auto-refresh
- Extend Topology.get_topology_for_weathermap with bandwidth utilization data
- Add WeathermapViewer JavaScript hook with color-coded edges:
  * Green (0-30%) → Yellow (30-60%) → Orange (60-80%) → Red (80%+)
- Display throughput/capacity labels on edges (e.g., '847 Mbps / 1 Gbps')
- Add utilization stats dashboard with link count by usage level
- Support full-screen mode for NOC displays
- Integrate with existing dark mode and responsive design
- Add navigation links in sidebar and mobile menu

Technical implementation:
- Leverages existing Capacity module for interface utilization calculations
- Extends topology edges with utilization_level, utilization_pct, throughput_bps
- Builds on NetworkMapLive foundation with enhanced edge styling
- Auto-refresh via Phoenix PubSub and 60s timer
- Filter support for high/critical utilization links

Reviewed-on: graham/towerops-web#172
2026-03-25 16:26:11 -05:00
f442c59ad0 fix/ping-monitoring-without-snmp (#171)
Reviewed-on: graham/towerops-web#171
2026-03-25 16:01:05 -05:00
3d5c74e7ae fix: correct AirFiber proprietary OID mapping and v1-only device support (#170)
Root cause: AF11X devices only support SNMPv1, IF-MIB eth0 counters
return zero, and air0 32-bit counters wrap too fast for backhaul speeds.

Fixes:
- Use correct UBNT-AirFIBER-MIB OID indices from MIB SEQUENCE definition:
  .7.1 = rxOctetsOK (inbound), .6.1 = txOctetsOK (outbound)
  .10.1 = rxErroredFrames, .11.1 = txErroredFrames
- Detect AirFiber devices by sysDescr instead of SNMP probing (v1 devices
  cannot return Counter64 via standard probe queries)
- Force SNMPv1 for AirFiber proprietary queries
- Apply proprietary counters to all real interfaces (not just eth0)

Verified against live AF11X at 10.250.1.90:
- IF-MIB eth0 returns zero counters
- Proprietary .6.1/.7.1 show real traffic (3.75TB rx / 43TB tx)

Reviewed-on: graham/towerops-web#170
2026-03-25 15:02:45 -05:00
ce682991fe fix: correct AirFiber proprietary OID mapping and v1-only device support (#169)
Root cause: AF11X devices only support SNMPv1, IF-MIB eth0 counters
return zero, and air0 32-bit counters wrap too fast for backhaul speeds.

Fixes:
- Use correct UBNT-AirFIBER-MIB OID indices from MIB SEQUENCE definition:
  .7.1 = rxOctetsOK (inbound), .6.1 = txOctetsOK (outbound)
  .10.1 = rxErroredFrames, .11.1 = txErroredFrames
- Detect AirFiber devices by sysDescr instead of SNMP probing (v1 devices
  cannot return Counter64 via standard probe queries)
- Force SNMPv1 for AirFiber proprietary queries
- Apply proprietary counters to all real interfaces (not just eth0)

Verified against live AF11X at 10.250.1.90:
- IF-MIB eth0 returns zero counters
- Proprietary .6.1/.7.1 show real traffic (3.75TB rx / 43TB tx)

Reviewed-on: graham/towerops-web#169
2026-03-25 14:30:13 -05:00
de455d1cf4 fix: improve bandwidth counter handling to match LibreNMS approach (#168)
- Clamp negative counter deltas to zero instead of attempting 32-bit wrap
  correction (matches LibreNMS behavior — wraps are indistinguishable from
  HC counter resets, so discard the ambiguous data point)
- Add rate sanity check: discard values exceeding 400 Gbps as counter anomalies
- Add 4-byte binary decoding for 32-bit SNMP counters in decode_snmp_value
- Track HC vs standard counter usage per interface stat (is_hc field)
- Add migration for is_hc column on snmp_interface_stats
- Update tests to match new clamping behavior

Reviewed-on: graham/towerops-web#168
2026-03-25 13:33:45 -05:00
94fd8780de fix/counter-wrap-detection (#164)
Reviewed-on: graham/towerops-web#164
2026-03-25 12:25:13 -05:00
86db804729 fix: correct 32-bit SNMP counter wrap detection for all traffic calculations (#163)
The wrap detection only applied the 2^32 correction when the previous
counter was in the upper half of the 32-bit range (> 2^31). On fast
links, counters wrap regardless of where in the range they sit. Three
graph rendering paths had no wrap detection at all, just max(0).

Consolidated all bps calculations into a single public
Capacity.calculate_bps/3 that always tries the 2^32 correction for
negative deltas. 64-bit HC counter resets remain clamped to zero since
the corrected delta stays negative.

Reviewed-on: graham/towerops-web#163
2026-03-25 11:03:43 -05:00
4b6538df8e fix: remove duplicate handle_info and schedule_cleanup function definitions in AgentCache (#162)
Removed duplicate function definitions that were causing compile warnings:
- handle_info(:cleanup_expired, state) was defined twice (lines 94 and 110)
- schedule_cleanup/0 was defined twice (lines 105 and 121)

The duplicate definitions at lines 110-124 were exact copies and have been removed.

Reviewed-on: graham/towerops-web#162
2026-03-25 10:18:11 -05:00
f0fe3c025c fix: recover correct bandwidth rate when 32-bit SNMP counters wrap (#161)
Standard ifInOctets/ifOutOctets are 32-bit counters that wrap around
4,294,967,296 bytes (~4.3 GB). At 1 Gbps this wraps every ~34 seconds,
causing brief drops to 0 bps in traffic charts and capacity reports.

When prev > 2^31 and delta is negative, add 2^32 to recover the correct
rate. If the result is still negative (HC 64-bit counter after reboot),
clamp to 0. Small-value decreases (prev < 2^31) continue to clamp to 0
since they cannot be real 32-bit wraps.

Fixes both capacity.ex (utilization reports) and device_live/show.ex
(per-device traffic chart).

Reviewed-on: graham/towerops-web#161
2026-03-25 09:52:15 -05:00
53c8f495d1 fix/critical-logic-and-memory-bugs (#159)
Reviewed-on: graham/towerops-web#159
2026-03-25 09:50:21 -05:00
3f0f6e7d5f feat: add per-organization data retention with 1-year default (#144)
Add data_retention_days field to organizations (default 365, range 30-730).
DataRetentionWorker runs nightly at 1 AM to purge expired time-series data
across 8 tables, respecting each org's retention setting.

TimescaleDB global retention bumped from 90 to 730 days as safety net.
Resolved alerts are cleaned up; unresolved alerts kept regardless of age.

Reviewed-on: graham/towerops-web#144
2026-03-24 15:30:43 -05:00
f703e61b12 fix: code quality improvements and test reliability (#139)
Various code quality improvements and fixes:

- Settings: return error instead of auto-creating unknown settings
- PagerDuty: proper backoff timing with to_timeout()
- Gaiia sync: remove unnecessary transaction wrapper
- HTTP/UISP: simplify test plug setup
- Agent channel: tagged tuple config for check types
- SNMP: fix Task.yield/shutdown pattern, remove stale preload
- Discovery: simplify error handling
- Tests: bump executor timeouts for CI reliability
- Dockerfile: remove unnecessary --force compile
- Renovate/CI: config cleanup

Split from #135.

Reviewed-on: graham/towerops-web#139
2026-03-24 09:12:59 -05:00
4a43d9decc chore: remove dead commented-out code from topology.ex (#137)
Removes the commented-out `apply_inferred_role` function that was left behind when device type was simplified to manual-only.

Split from #135.

Reviewed-on: graham/towerops-web#137
2026-03-24 08:34:06 -05:00
12a737e1f5 fix/security-and-quality-issues (#136)
Reviewed-on: graham/towerops-web#136
2026-03-24 08:06:29 -05:00
872f717dba Fix critical bugs, memory leaks, and N+1 queries (#132)
CRITICAL FIXES:
- Fix unsafe pattern matching in SNMP discovery that would crash on timeout
  - Changed {:ok, _} = ... to proper case statements with error handling
  - Extracted safe_discover/3 helper to reduce cyclomatic complexity
  - Added fallback to empty lists and warning logging for timeouts
  - Affects: discover_vlans, discover_ip_addresses, discover_processors, discover_storage

MEMORY LEAK FIXES (JS Hooks):
- GlobalSearchTrigger: Added destroyed() cleanup for click listener
- SidebarCollapse: Added destroyed() cleanup for click listener
- ThemeSelector: Added destroyed() cleanup for phx:set-theme window listener
- NetworkMap: Added destroyed() cleanup for 3 button listeners (zoom in/out/fit)
- All hooks now properly store handler references and remove listeners on unmount

LIVEVIEW FIXES:
- Added phx-update="ignore" to 5 SensorChart hooks to prevent chart re-initialization
- Added catch-all handle_info(_msg, socket) to 5 LiveViews to prevent crashes on unexpected messages
  - device_live/show.ex, device_live/index.ex, alert_live/index.ex, activity_feed_live.ex, agent_live/index.ex

N+1 QUERY FIXES:
- Created Gaiia.get_site_subscriber_summaries/1 batch function
- Refactored alert_live and device_live to use batch query instead of looping
- Reduces queries from N to 1 when loading site subscriber data

Impact:
- Prevents discovery worker crashes during SNMP timeouts
- Eliminates memory accumulation from leaked event handlers
- Prevents chart state loss on LiveView updates
- Prevents LiveView crashes from unexpected PubSub messages
- Improves database performance by eliminating N+1 queries in alert/device views
- Passes credo --strict with 0 issues

Reviewed-on: graham/towerops-web#132
2026-03-23 17:03:41 -05:00
20d63c408a perf: speed up test suite from 118s to 65s (45% faster) (#126)
Optimized slow tests by reducing unnecessary network timeouts and retry delays:

**PagerDuty tests (7.3s → 0.2s, 97% faster):**
- Added test-specific sleep/backoff configuration
- Use 1ms sleep instead of 1s/2s/4s exponential backoff in tests
- Maintains retry logic for production, fast tests in dev

**HTTP/UISP client tests (7s → <0.1s, 99% faster):**
- Disabled Req retry middleware in test environment
- Added 1s receive_timeout for test requests
- Prevents 7s retry delays on error status codes

**Ping executor tests (13s → 1.5s, 88% faster per test):**
- Reduced ping timeout from 6-10s to 0.5-1s for localhost
- Reduced timeout to 1s for unreachable hosts
- Still validates behavior, just faster

**Changes:**
- lib/towerops/pagerduty/client.ex: test-aware sleep/backoff
- lib/towerops/http.ex: disable retry in tests, 1s timeout
- lib/towerops/uisp/client.ex: disable retry in tests, 1s timeout
- test/towerops/monitoring/executors/ping_executor_test.exs: reduced timeouts

**Result:**
Total test time: 118.4s → 65.3s (45% improvement)
Top 10 slowest: 66.1s → 10.9s (83% improvement)

Reviewed-on: graham/towerops-web#126
2026-03-23 10:34:23 -05:00
7d24e56bd9 fix: wrap Gaiia sync upserts in transaction and remove stale device_role_source ref (#118)
- Wrap upsert_nodes in Repo.transaction so all upserts share one DB
  connection, preventing pool exhaustion under load
- Remove stale device_role_source reference from form test

Reviewed-on: graham/towerops-web#118
2026-03-22 16:47:17 -05:00
2bb040154a feat: add dynamic capacity tracking for backhaul devices (#113)
For devices with device_role="backhaul", traffic graphs now show capacity
reference lines that update over time based on which interfaces are active.

Features:
- Calculates total capacity per timestamp from active monitored interfaces
- Uses configured_capacity_bps if set, falls back to if_speed
- Capacity adjusts as interfaces go up/down throughout the time period
- Shows as dashed lines at +capacity (inbound) and -capacity (outbound)
- Matches the existing 0-axis graph style

Implementation:
- Small graph (device detail page): Updated build_traffic_json_datasets/2
  and add_capacity_datasets/3 in device_live/show.ex
- Full-page graph: Updated build_traffic_datasets/2 and
  add_capacity_to_full_graph/4 in graph_live/show.ex

The capacity lines provide visual reference for backhaul link saturation,
showing how close actual throughput is to maximum capacity at each point
in time.

Reviewed-on: graham/towerops-web#113
2026-03-22 13:54:37 -05:00
35b1c9e753 fix/test-failures (#112)
Reviewed-on: graham/towerops-web#112
2026-03-22 13:02:06 -05:00
5a6acf0d7c feat: simplify device type to manual-only with 6 options (#109)
- Reduce device type options from 10 to 6 (server, switch, router, access_point, backhaul, other)
- Change label from "Device Role" to "Device Type"
- Disable auto-detection - device type is now required and manual-only
- Add migration to map existing device roles to new simplified set
- Update REST API to include device_role and device_role_source fields
- Automatically set device_role_source to "manual" when device_role is provided
- Update wireless detection to include "backhaul" type
- Enhance display formatting for "Access Point"

Breaking Changes:
- Auto-inference of device type has been disabled
- device_role is now a required field in forms
- GraphQL/REST API consumers will see changed device_role values after migration

Files Changed:
- lib/towerops_web/live/device_live/form.html.heex
- lib/towerops/topology.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/devices/device.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/devices_controller.ex
- priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs
- test/towerops/topology_test.exs
- test/towerops_web/live/device_live/form_test.exs

Reviewed-on: graham/towerops-web#109
2026-03-22 11:24:53 -05:00
6ef6b3d61d fix: comprehensive security audit fixes (#108)
This commit addresses multiple CRITICAL, HIGH, and MEDIUM severity security vulnerabilities identified in the security audit:

CRITICAL FIXES:
- Fix weak RNG for recovery codes - replaced Enum.random() with :crypto.strong_rand_bytes/1 for cryptographically secure token generation
- Fix subscription limit race conditions - moved free org and device quota checks inside transactions with FOR UPDATE locks to prevent concurrent bypass
- Fix default organization race condition - moved is_default check inside transaction to prevent multiple defaults per user

HIGH SEVERITY FIXES:
- Fix agent token deletion race condition - moved PubSub broadcast inside transaction to ensure agents only receive notification after successful deletion

MEDIUM SEVERITY FIXES:
- Fix LIKE wildcard injection in search - applied sanitize_like() to all user-facing search queries in devices.ex, sites.ex, and gaiia.ex to prevent enumeration attacks
- Fix Jason.decode! DoS - replaced with safe Jason.decode/1 with error handling in device_live/index.ex
- Fix SSRF vulnerability - added URL validation in HTTP executor to block requests to private/internal IP ranges (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16)
- Fix error information leakage - replaced inspect() in API responses with generic error messages, logging details server-side only
- Fix atom table pollution - HTTP method normalization now uses whitelist mapping instead of String.to_atom()

SECURITY IMPROVEMENTS:
- All quota checks now use pessimistic locking (SELECT FOR UPDATE) to prevent TOCTOU race conditions
- Private IP validation prevents cloud metadata service access (169.254.169.254)
- DNS resolution performed before HTTP requests to detect IP spoofing
- Error details logged server-side but not exposed to clients

Files changed:
- lib/towerops/accounts/user_recovery_code.ex
- lib/towerops/organizations.ex
- lib/towerops/devices.ex
- lib/towerops/sites.ex
- lib/towerops/gaiia.ex
- lib/towerops/agents.ex
- lib/towerops/monitoring/executors/http_executor.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/mib_controller.ex
- lib/towerops_web/controllers/api/v1/agent_release_webhook_controller.ex
- lib/towerops_web/controllers/api/v1/geoip_controller.ex

Reviewed-on: graham/towerops-web#108
2026-03-22 10:10:27 -05:00
c8b275bd4c gleam4 (#105)
Reviewed-on: graham/towerops-web#105
2026-03-21 18:36:12 -05:00
e54ee75513 Rewrite protobuf encoding/decoding in pure Gleam with integrated validation (#104)
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.

- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency

Reviewed-on: graham/towerops-web#104
2026-03-21 16:08:40 -05:00
348975dcc9 Rewrite polling_offset, changelog_parser, and user_agent_parser in Gleam (#103)
Migrate three pure-function modules from Elixir to Gleam:

- polling_offset: deterministic hash-based polling offset (pure Gleam, no FFI)
- changelog_parser: text parsing logic in Gleam, thin Elixir wrapper for File I/O
- user_agent_parser: regex-based UA parsing in Gleam with Erlang FFI for
  atom-keyed map conversion

Add gleam_regexp dependency for regex support in Gleam modules.
Update all call sites and tests to use Gleam module atoms.

Reviewed-on: graham/towerops-web#103
2026-03-21 10:11:06 -05:00
6859525a1b Integrate Gleam into the build pipeline and rewrite three pure-function modules (#98)
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.

Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.

Reviewed-on: graham/towerops-web#98
2026-03-21 07:30:36 -05:00
933bfafeb0 Fix DB pool exhaustion and settings page crashes (#88)
## Summary
- **DB pool exhaustion**: `device_has_effective_agent?` was doing 2 DB queries per call (3-table join + global settings lookup), called 2-3x per worker cycle across hundreds of devices (~6000 unnecessary queries/minute). Added an ETS-backed `AgentCache` with 120s TTL and removed redundant calls within the same worker cycle.
- **Settings page crash**: `KeyError: key :user_agent not found` on BrowserSession — field doesn't exist, replaced with `device_type`.
- **Login history IP truncation**: Changed from side-by-side grid to stacked table layout so IP addresses aren't clipped.

## Test plan
- [x] 8 new tests for AgentCache (cache hit, miss, invalidation, global default)
- [x] Full test suite passes (8651 tests, 0 failures, verified across multiple runs)
- [x] `mix format` and pre-commit hooks pass

Reviewed-on: graham/towerops-web#88
2026-03-19 15:16:34 -05:00
63fc09e58a fix check skip log noise and browser session crash (#87)
- Downgrade check executor skip log from info to debug to stop flooding
  production logs every 60s for every agent-assigned device check
- Fix KeyError on session.user_agent in settings page - BrowserSession
  schema has device_type field, not user_agent

Reviewed-on: graham/towerops-web#87
2026-03-19 14:29:39 -05:00
0c2c4cc354 gdpr (#86)
Reviewed-on: graham/towerops-web#86
2026-03-19 14:28:31 -05:00
2b50fc0732 fix agent check cascade and at-risk subscriber double-counting (#79)
## Summary

- **`list_checks_for_agent` cascade**: the agent query only matched checks with `agent_token_id` set directly on the check record — checks on devices inheriting their agent via site/org/global default were never sent to the Go agent. Now uses left joins with the full cascade (device assignment → site → org default → global default cloud poller), matching the pattern from `list_agent_polling_targets`.
- **At-risk summary double-counting**: `get_impact_summary` called `analyze_device_impact` per down AP, and each returned the full site subscriber count. With 3 APs down at a 147-sub site it showed 441 subs / $31k instead of 147 / $10k. Fixed by deduplicating impact per site. Same fix applied to `calculate_down_impact` in site impact summaries.

## Test plan

- [x] 3 new cascade tests for `list_checks_for_agent` (site, org default, global default)
- [x] 3 new tests for `get_impact_summary` (single site dedup, multi-site sum, zero case)
- [x] All 8,607 existing tests pass
- [ ] Deploy to staging, verify At Risk numbers on dashboard
- [ ] Verify Go agent receives DNS/HTTP checks via cascade

Reviewed-on: graham/towerops-web#79
2026-03-18 20:17:39 -05:00
d6f7dc570a simplify check skip logic to use runtime checks (#78)
## Summary
- Replace guard-clause-based `should_skip_snmp/service/ping_check?` with a single `should_skip_check?` function using runtime `in` operator checks
- Guard clauses with module attributes (`@service_check_types`) were not working in Dokku buildpack deployments despite code being correct locally
- Also update `CheckWorker` (dead code safety net) to check `device_has_effective_agent?`
- Use `Logger.info` for skip messages so they appear in staging logs

## Context
PR #77 introduced `device_has_effective_agent?/1` and updated all workers, but the guard-clause-based skip functions in `CheckExecutorWorker` weren't matching at runtime on staging despite passing all tests locally. The DNS check kept producing NXDOMAIN errors from Erlang's `:inet_res` resolver.

Root cause: The `when check_type in @service_check_types` guard clause compiled against module attributes appears to have stale evaluation in the Dokku Heroku buildpack release. Moving the check to a runtime function body fixes the issue.

## Test plan
- [x] All 230 worker tests pass
- [x] All 99 agents tests pass
- [x] Verified on staging: "Skipping check eceb3d51... (dns)" appears in info logs
- [x] No new NXDOMAIN results created after deploy

Reviewed-on: graham/towerops-web#78
2026-03-18 19:58:31 -05:00
f3cf1a9b3c skip all server-side checks when any agent is assigned (#77)
## Summary
- Add `device_has_effective_agent?/1` to Agents context that checks full cascade (device → site → org → global default) without distinguishing between cloud pollers and local agents
- Update check_executor_worker, device_poller_worker, and device_monitor_worker to use new function
- Phoenix never executes checks (DNS, ping, HTTP, TCP, SSL, SNMP) when any agent handles the device

## Problem
The global default cloud poller was not being checked in `should_phoenix_poll_device?/1`, causing Phoenix to run DNS/ping/service checks server-side even when the Go agent was handling the device. This produced duplicate results with NXDOMAIN errors from the Erlang DNS resolver.

## Test plan
- [x] 6 new tests for `device_has_effective_agent?/1` covering all cascade levels
- [x] Full test suite passes (8601 tests)
- [ ] Verify staging DNS check stops showing NXDOMAIN errors after deploy

Reviewed-on: graham/towerops-web#77
2026-03-18 17:29:36 -05:00
c4156ae334 fix graph 500 error on dns check page (#75)
Float.round/2 crashes on integer values from the status dataset. Adds integer guard clause to format_value/2.

Reviewed-on: graham/towerops-web#75
2026-03-18 15:33:29 -05:00
9d0c824aa5 add dual-axis graph for dns checks (#74)
DNS check graph now shows two datasets: Response Time (ms) on left y-axis and Status (Up/Down) on right y-axis with stepped green fill. Also adds space before units in chart labels.

Reviewed-on: graham/towerops-web#74
2026-03-18 15:13:31 -05:00
272d4bcdc9 fix: check state never persisted due to missing cast fields (#73)
Check.changeset/2 did not include state fields (current_state,
current_state_type, current_check_attempt, last_check_at, etc.) in
its cast list. Ecto silently dropped all state updates, so
update_check_state succeeded but wrote nothing to the database.

Added Check.state_changeset/2 for internal state machine updates,
keeping the regular changeset safe from user input. Updated
Monitoring.update_check_state/3 to use it directly.

Reviewed-on: graham/towerops-web#73
2026-03-18 14:54:52 -05:00
f05cc32ef6 fix dns/ping check execution and checks table alignment (#72)
## Summary
- **DNS checks**: `parse_server` returned a flat 5-tuple `{a,b,c,d,53}` instead of `{{a,b,c,d}, 53}` — the format `:inet_res` requires for nameservers. All DNS checks using a custom server failed with argument error.
- **Ping checks**: `System.cmd` `:timeout` option not supported on Elixir 1.19.4 (staging). Replaced with `Task.async`/`Task.yield` wrapper for cross-version compatibility.
- **Checks table UI**: Removed unnecessary flex div wrapper on status badge column that caused vertical misalignment with other table cells.

## Test plan
- [x] DNS executor tests pass (custom server resolution now succeeds)
- [x] Ping executor tests pass (no more ArgumentError)
- [x] Full test suite passes (8593 tests, 0 new failures)

Reviewed-on: graham/towerops-web#72
2026-03-18 14:05:37 -05:00
997bc46cbf fix: dns and ping checks not executing in production (#71)
## Summary

- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever

## Changes

- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience

## Test plan

- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)

Reviewed-on: graham/towerops-web#71
2026-03-18 08:43:37 -05:00
c8a092196f add auto ICMP ping check for all devices (#68)
## Summary

- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)

## Test plan

- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal

Reviewed-on: graham/towerops-web#68
2026-03-17 17:55:23 -05:00
76b3dd6a93 add ssl/tls certificate expiration check type (#64)
add ssl check executor that connects via TLS, reads peer certificate
expiry, and returns OK/WARNING/CRITICAL based on configurable
warning_days threshold. includes protobuf config, liveview form fields,
agent channel encoding, oban worker dispatch, and unit tests.

Reviewed-on: graham/towerops-web#64
2026-03-17 16:14:22 -05:00
150ffb421d delegate service checks to agents and fix ui issues (#63)
- Skip HTTP/TCP/DNS checks in Phoenix when device is assigned to agent
- Agents now execute service checks from the device's network location
- Fix cancel button in service check modal (text link instead of button)
- Fix status badge alignment in checks table

Reviewed-on: graham/towerops-web#63
2026-03-17 15:43:31 -05:00
fa2e09d7cc skip discovery for devices with snmp disabled (#58)
add early bail in DiscoveryWorker when snmp_enabled is false.
mark snmp_not_enabled and device_unresponsive as non-retryable
errors to prevent wasted retries.

Reviewed-on: graham/towerops-web#58
2026-03-17 11:16:33 -05:00
0c8bb35ef7 add query_helpers tests and fix all credo issues (#56)
- add tests for QueryHelpers.sanitize_like/1 (%, _, \ escaping)
- fix nesting depth in store_check_result, handle_check_state_change,
  graphql check resolver, checks controller, and ping executor
- fix cyclomatic complexity in build_check_protobuf by extracting
  check_type_config/2 function clauses
- fix formatting in api_docs_html template

Reviewed-on: graham/towerops-web#56
2026-03-17 10:43:57 -05:00
28e97ff5f0 add service checks REST API, GraphQL API, and ping executor (#52)
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section

Reviewed-on: graham/towerops-web#52
2026-03-16 18:40:18 -05:00
c6913685ee wire up service checks (HTTP/TCP/DNS) to remote agents (#50)
send check_jobs protobuf to agents during job dispatch cycle, handle
check_result messages back with state transitions and alert
creation/resolution. auto-assign checks to device's effective agent
token on creation, broadcast check changes via PubSub so connected
agents get immediate updates.

add edit/delete support for service checks on device show page with
form pre-population from existing check config. auto-fill TCP host
from device IP address.

includes validate_check_result/1 for protobuf validation and
list_checks_for_agent/2 for querying agent-assigned service checks.

also fixes pre-existing broken device_type_icon reference in device
index template.

Reviewed-on: graham/towerops-web#50
2026-03-16 17:25:02 -05:00
7c8731f86a fix: make sidebar sticky with header and resolve all credo issues (#32)
- Changed sidebar from fixed to sticky positioning
- Wrapped sidebar and main content in flex container
- Sidebar now sticks to top along with header when scrolling
- Removed unused Ecto.Query import from gps_sync.ex

Credo improvements (17 → 0 remaining):
- Replaced all length/1 checks with empty list comparisons
- Extracted helper functions to reduce nesting depth
- Split complex functions into smaller, testable units
- Applied 'with' statements for clearer control flow
- Refactored high-complexity functions (cyclomatic complexity)
- Files improved: storm_detector, alert_notification_worker,
  alert_digest_worker, gps_sync, statistics_sync, device_sync,
  site_sync, site_correlation, reports_live, topology,
  pagerduty/client, cn_maestro/sync

Reviewed-on: graham/towerops-web#32
2026-03-15 18:19:09 -05:00
42bfc8b9d2 paperclip/TOW-21-uisp-integration-phase1 (#31)
Reviewed-on: graham/towerops-web#31
2026-03-15 17:57:36 -05:00
acf70b3bee
fix: remove duplicate StormDetector child spec
The StormDetector was being added twice:
1. In the main children list (line 127)
2. In background_workers() (line 176)

This caused the supervisor to fail with 'duplicate_child_name' error.

Removed the duplicate from main children list since it's correctly
placed in background_workers() which also handles test env properly.
2026-03-15 16:05:30 -05:00
44fc017653
test: improve test coverage and code quality
- Update pre-commit config
- Add test coverage for URL validation, security modules, and monitoring executors
- Improve network map implementation and tests
- Add coverage for API controllers and LiveView endpoints
- Refactor code to fix Credo issues (reduce nesting, extract helper functions)
2026-03-15 14:49:55 -05:00
90af7ac0d1 feat: alert storm correlation & suppression (TOW-13)
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
  suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
  single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
  batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
  Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
  takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
  plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
  before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
  and batch maintenance checks
2026-03-15 14:29:12 -05:00
b134a2e56a perf: LiveView event batching & PubSub optimization (TOW-26)
- Add 1.5s debounce to dashboard LiveView for alert events, preventing
  excessive re-renders during mass outages where hundreds of alerts fire
  simultaneously
- Replace global 'device:events' PubSub topic with org-scoped
  'device_events:org:{id}' topics across all broadcasters (device_poller_worker,
  sensor_change_detector, discovery) so LiveView processes only receive
  events for their tenant
- Update EventLogger GenServer to subscribe to per-org topics dynamically,
  with runtime subscription support for newly created orgs
- Add Organizations.list_organization_ids/0 for EventLogger bootstrap
- Increase default DB pool_size from 20 to 25 for production concurrent users

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-15 14:08:27 -05:00
9e35e1eef2 feat: WISP-tuned network map enhancements (TOW-16)
- Tower/device icons: role-based shapes (triangle for APs/wireless,
  diamond for switches, round-rect for routers, hexagon for firewalls)
- RF link overlays: edges colored by signal health (green/yellow/red)
  with bandwidth-proportional line thickness
- Enhanced hover tooltips: device-to-device name, SNR dB, bandwidth,
  signal health indicator
- Wireless client count badge on AP nodes
- Filter bar: All Links | Degraded Only | Sites with Alerts | Search
- Search with zoom-to-match and highlight
- Geographic layout toggle (when sites have lat/lng)
- Enhanced detail panel: RF stats (client count, signal health, SNR),
  View RF Links button for wireless devices
- Backend: compute_wireless_stats/compute_rf_link_stats enrich topology
- Site nodes include lat/lng for geographic layout
- Edge enrichment with RF signal health from SNR sensors
- Fix pre-existing compile error in notification_rate_limiter.ex
2026-03-15 14:06:06 -05:00