check_type_config was returning `config: {:http, ...}` but the
Check struct has no :config field — it has :http, :tcp, :dns, :ssl
directly. This caused a KeyError crash in build_and_push_check_jobs.
Reviewed-on: graham/towerops-web#121
Seconds-level precision adds noise without value for these fields.
Shows "less than a minute" instead of "0 seconds" for very short durations.
Reviewed-on: graham/towerops-web#120
- Wrap upsert_nodes in Repo.transaction so all upserts share one DB
connection, preventing pool exhaustion under load
- Remove stale device_role_source reference from form test
Reviewed-on: graham/towerops-web#118
- Add missing flex-1/text-right classes to Resolved IP row
- Add justify-end to Operating System flex container
- Fix duplicate time labels on 24h traffic chart by excluding boundary ticks
Reviewed-on: graham/towerops-web#117
- Capacity: Sensor values (Rx/Tx Capacity) now represent total device
capacity, not per-interface. Previous bug applied the same sensor value
to all interfaces and summed them (e.g., 773.6 Mbps × 5 = 3.87 Gbps).
- Wireless sensors: Changed Float.round/2 to :erlang.float_to_binary/2
to prevent scientific notation in frequency display (was showing
"2.41e4 MHz" instead of "24100.0 MHz").
Files:
- lib/towerops_web/live/device_live/show.ex
- lib/towerops_web/live/device_live/show.html.heex
- CHANGELOG.txt
- priv/static/changelog.txt
Reviewed-on: graham/towerops-web#116
Radio frequencies (24 GHz) were displaying as "2.42e4 MHz" instead of
"24200.0 MHz" because Float.round/2 returns a float that Phoenix
converts to scientific notation for large numbers.
Changed format_sensor_value/2 to use :erlang.float_to_binary/2 with
decimals: 1 option, which formats as a string and avoids scientific
notation.
Verified: 24200.0 now displays as "24200.0" instead of "2.42e4"
Reviewed-on: graham/towerops-web#114
For devices with device_role="backhaul", traffic graphs now show capacity
reference lines that update over time based on which interfaces are active.
Features:
- Calculates total capacity per timestamp from active monitored interfaces
- Uses configured_capacity_bps if set, falls back to if_speed
- Capacity adjusts as interfaces go up/down throughout the time period
- Shows as dashed lines at +capacity (inbound) and -capacity (outbound)
- Matches the existing 0-axis graph style
Implementation:
- Small graph (device detail page): Updated build_traffic_json_datasets/2
and add_capacity_datasets/3 in device_live/show.ex
- Full-page graph: Updated build_traffic_datasets/2 and
add_capacity_to_full_graph/4 in graph_live/show.ex
The capacity lines provide visual reference for backhaul link saturation,
showing how close actual throughput is to maximum capacity at each point
in time.
Reviewed-on: graham/towerops-web#113
- Removed device_role_source assignment in sanitize_device_role/1
- Updated structure.sql to reflect database schema without device_role_source column
- Fixes ERROR 42703 (undefined_column) in production workers
The form was still trying to set device_role_source in params even though
the database column was dropped, causing runtime errors in:
- AgentLatencyEvaluator
- CapacityInsightWorker
- WirelessInsightWorker
All device roles are now manually set with default value 'other'.
Reviewed-on: graham/towerops-web#110
- Reduce device type options from 10 to 6 (server, switch, router, access_point, backhaul, other)
- Change label from "Device Role" to "Device Type"
- Disable auto-detection - device type is now required and manual-only
- Add migration to map existing device roles to new simplified set
- Update REST API to include device_role and device_role_source fields
- Automatically set device_role_source to "manual" when device_role is provided
- Update wireless detection to include "backhaul" type
- Enhance display formatting for "Access Point"
Breaking Changes:
- Auto-inference of device type has been disabled
- device_role is now a required field in forms
- GraphQL/REST API consumers will see changed device_role values after migration
Files Changed:
- lib/towerops_web/live/device_live/form.html.heex
- lib/towerops/topology.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/devices/device.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/devices_controller.ex
- priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs
- test/towerops/topology_test.exs
- test/towerops_web/live/device_live/form_test.exs
Reviewed-on: graham/towerops-web#109
This commit addresses multiple CRITICAL, HIGH, and MEDIUM severity security vulnerabilities identified in the security audit:
CRITICAL FIXES:
- Fix weak RNG for recovery codes - replaced Enum.random() with :crypto.strong_rand_bytes/1 for cryptographically secure token generation
- Fix subscription limit race conditions - moved free org and device quota checks inside transactions with FOR UPDATE locks to prevent concurrent bypass
- Fix default organization race condition - moved is_default check inside transaction to prevent multiple defaults per user
HIGH SEVERITY FIXES:
- Fix agent token deletion race condition - moved PubSub broadcast inside transaction to ensure agents only receive notification after successful deletion
MEDIUM SEVERITY FIXES:
- Fix LIKE wildcard injection in search - applied sanitize_like() to all user-facing search queries in devices.ex, sites.ex, and gaiia.ex to prevent enumeration attacks
- Fix Jason.decode! DoS - replaced with safe Jason.decode/1 with error handling in device_live/index.ex
- Fix SSRF vulnerability - added URL validation in HTTP executor to block requests to private/internal IP ranges (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16)
- Fix error information leakage - replaced inspect() in API responses with generic error messages, logging details server-side only
- Fix atom table pollution - HTTP method normalization now uses whitelist mapping instead of String.to_atom()
SECURITY IMPROVEMENTS:
- All quota checks now use pessimistic locking (SELECT FOR UPDATE) to prevent TOCTOU race conditions
- Private IP validation prevents cloud metadata service access (169.254.169.254)
- DNS resolution performed before HTTP requests to detect IP spoofing
- Error details logged server-side but not exposed to clients
Files changed:
- lib/towerops/accounts/user_recovery_code.ex
- lib/towerops/organizations.ex
- lib/towerops/devices.ex
- lib/towerops/sites.ex
- lib/towerops/gaiia.ex
- lib/towerops/agents.ex
- lib/towerops/monitoring/executors/http_executor.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/mib_controller.ex
- lib/towerops_web/controllers/api/v1/agent_release_webhook_controller.ex
- lib/towerops_web/controllers/api/v1/geoip_controller.ex
Reviewed-on: graham/towerops-web#108
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.
- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency
Reviewed-on: graham/towerops-web#104
Migrate three pure-function modules from Elixir to Gleam:
- polling_offset: deterministic hash-based polling offset (pure Gleam, no FFI)
- changelog_parser: text parsing logic in Gleam, thin Elixir wrapper for File I/O
- user_agent_parser: regex-based UA parsing in Gleam with Erlang FFI for
atom-keyed map conversion
Add gleam_regexp dependency for regex support in Gleam modules.
Update all call sites and tests to use Gleam module atoms.
Reviewed-on: graham/towerops-web#103
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.
Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.
Reviewed-on: graham/towerops-web#98
Protobuf oneof fields use a single :config key with tagged tuples
like {:dns, %DnsCheckConfig{}} rather than individual keys. The
check_type_config functions were setting e.g. dns: %DnsCheckConfig{}
directly, causing KeyError crashes in Check.__struct__/1.
Reviewed-on: graham/towerops-web#101
## Summary
- **DB pool exhaustion**: `device_has_effective_agent?` was doing 2 DB queries per call (3-table join + global settings lookup), called 2-3x per worker cycle across hundreds of devices (~6000 unnecessary queries/minute). Added an ETS-backed `AgentCache` with 120s TTL and removed redundant calls within the same worker cycle.
- **Settings page crash**: `KeyError: key :user_agent not found` on BrowserSession — field doesn't exist, replaced with `device_type`.
- **Login history IP truncation**: Changed from side-by-side grid to stacked table layout so IP addresses aren't clipped.
## Test plan
- [x] 8 new tests for AgentCache (cache hit, miss, invalidation, global default)
- [x] Full test suite passes (8651 tests, 0 failures, verified across multiple runs)
- [x] `mix format` and pre-commit hooks pass
Reviewed-on: graham/towerops-web#88
- Downgrade check executor skip log from info to debug to stop flooding
production logs every 60s for every agent-assigned device check
- Fix KeyError on session.user_agent in settings page - BrowserSession
schema has device_type field, not user_agent
Reviewed-on: graham/towerops-web#87
- Wrap agent name + icon inside the offline amber badge so it's clear
what is offline on the device detail page
- Add "Agent:" label before the agent indicator for all states
- Fix TimeSeriesTest deadlock: switch to async: false since check_results
is a TimescaleDB hypertable and concurrent chunk creation deadlocks
- Fix unicode property test: String.length/1 counts grapheme clusters,
not code points; combining chars can make a 2-codepoint string have
only 1 grapheme, legitimately failing the min: 2 validation
Reviewed-on: graham/towerops-web#83
## Summary
- **`list_checks_for_agent` cascade**: the agent query only matched checks with `agent_token_id` set directly on the check record — checks on devices inheriting their agent via site/org/global default were never sent to the Go agent. Now uses left joins with the full cascade (device assignment → site → org default → global default cloud poller), matching the pattern from `list_agent_polling_targets`.
- **At-risk summary double-counting**: `get_impact_summary` called `analyze_device_impact` per down AP, and each returned the full site subscriber count. With 3 APs down at a 147-sub site it showed 441 subs / $31k instead of 147 / $10k. Fixed by deduplicating impact per site. Same fix applied to `calculate_down_impact` in site impact summaries.
## Test plan
- [x] 3 new cascade tests for `list_checks_for_agent` (site, org default, global default)
- [x] 3 new tests for `get_impact_summary` (single site dedup, multi-site sum, zero case)
- [x] All 8,607 existing tests pass
- [ ] Deploy to staging, verify At Risk numbers on dashboard
- [ ] Verify Go agent receives DNS/HTTP checks via cascade
Reviewed-on: graham/towerops-web#79
## Summary
- Replace guard-clause-based `should_skip_snmp/service/ping_check?` with a single `should_skip_check?` function using runtime `in` operator checks
- Guard clauses with module attributes (`@service_check_types`) were not working in Dokku buildpack deployments despite code being correct locally
- Also update `CheckWorker` (dead code safety net) to check `device_has_effective_agent?`
- Use `Logger.info` for skip messages so they appear in staging logs
## Context
PR #77 introduced `device_has_effective_agent?/1` and updated all workers, but the guard-clause-based skip functions in `CheckExecutorWorker` weren't matching at runtime on staging despite passing all tests locally. The DNS check kept producing NXDOMAIN errors from Erlang's `:inet_res` resolver.
Root cause: The `when check_type in @service_check_types` guard clause compiled against module attributes appears to have stale evaluation in the Dokku Heroku buildpack release. Moving the check to a runtime function body fixes the issue.
## Test plan
- [x] All 230 worker tests pass
- [x] All 99 agents tests pass
- [x] Verified on staging: "Skipping check eceb3d51... (dns)" appears in info logs
- [x] No new NXDOMAIN results created after deploy
Reviewed-on: graham/towerops-web#78
## Summary
- Add `device_has_effective_agent?/1` to Agents context that checks full cascade (device → site → org → global default) without distinguishing between cloud pollers and local agents
- Update check_executor_worker, device_poller_worker, and device_monitor_worker to use new function
- Phoenix never executes checks (DNS, ping, HTTP, TCP, SSL, SNMP) when any agent handles the device
## Problem
The global default cloud poller was not being checked in `should_phoenix_poll_device?/1`, causing Phoenix to run DNS/ping/service checks server-side even when the Go agent was handling the device. This produced duplicate results with NXDOMAIN errors from the Erlang DNS resolver.
## Test plan
- [x] 6 new tests for `device_has_effective_agent?/1` covering all cascade levels
- [x] Full test suite passes (8601 tests)
- [ ] Verify staging DNS check stops showing NXDOMAIN errors after deploy
Reviewed-on: graham/towerops-web#77
Chart.js was showing integer-rounded values like '2' instead of '2.1 ms'. Uses toFixed(1) in tick callbacks for proper decimal display with units.
Reviewed-on: graham/towerops-web#76
DNS check graph now shows two datasets: Response Time (ms) on left y-axis and Status (Up/Down) on right y-axis with stepped green fill. Also adds space before units in chart labels.
Reviewed-on: graham/towerops-web#74
Check.changeset/2 did not include state fields (current_state,
current_state_type, current_check_attempt, last_check_at, etc.) in
its cast list. Ecto silently dropped all state updates, so
update_check_state succeeded but wrote nothing to the database.
Added Check.state_changeset/2 for internal state machine updates,
keeping the regular changeset safe from user input. Updated
Monitoring.update_check_state/3 to use it directly.
Reviewed-on: graham/towerops-web#73
## Summary
- **DNS checks**: `parse_server` returned a flat 5-tuple `{a,b,c,d,53}` instead of `{{a,b,c,d}, 53}` — the format `:inet_res` requires for nameservers. All DNS checks using a custom server failed with argument error.
- **Ping checks**: `System.cmd` `:timeout` option not supported on Elixir 1.19.4 (staging). Replaced with `Task.async`/`Task.yield` wrapper for cross-version compatibility.
- **Checks table UI**: Removed unnecessary flex div wrapper on status badge column that caused vertical misalignment with other table cells.
## Test plan
- [x] DNS executor tests pass (custom server resolution now succeeds)
- [x] Ping executor tests pass (no more ArgumentError)
- [x] Full test suite passes (8593 tests, 0 new failures)
Reviewed-on: graham/towerops-web#72
## Summary
- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever
## Changes
- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience
## Test plan
- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)
Reviewed-on: graham/towerops-web#71
## Summary
- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)
## Test plan
- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal
Reviewed-on: graham/towerops-web#68
add ssl check executor that connects via TLS, reads peer certificate
expiry, and returns OK/WARNING/CRITICAL based on configurable
warning_days threshold. includes protobuf config, liveview form fields,
agent channel encoding, oban worker dispatch, and unit tests.
Reviewed-on: graham/towerops-web#64
- Skip HTTP/TCP/DNS checks in Phoenix when device is assigned to agent
- Agents now execute service checks from the device's network location
- Fix cancel button in service check modal (text link instead of button)
- Fix status badge alignment in checks table
Reviewed-on: graham/towerops-web#63
add early bail in DiscoveryWorker when snmp_enabled is false.
mark snmp_not_enabled and device_unresponsive as non-retryable
errors to prevent wasted retries.
Reviewed-on: graham/towerops-web#58
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
send check_jobs protobuf to agents during job dispatch cycle, handle
check_result messages back with state transitions and alert
creation/resolution. auto-assign checks to device's effective agent
token on creation, broadcast check changes via PubSub so connected
agents get immediate updates.
add edit/delete support for service checks on device show page with
form pre-population from existing check config. auto-fill TCP host
from device IP address.
includes validate_check_result/1 for protobuf validation and
list_checks_for_agent/2 for querying agent-assigned service checks.
also fixes pre-existing broken device_type_icon reference in device
index template.
Reviewed-on: graham/towerops-web#50
Extend the GraphQL API and socket to accept mobile session tokens
alongside existing API tokens. Add OrganizationScope middleware
that extracts organization_id from query args for mobile users.
Add my_organizations query for mobile app org listing.
Reviewed-on: graham/towerops-web#48
## Summary
Hide RF link-related UI elements from the network map page while keeping all backend code intact.
### Changes
- Commented out "View RF Links" button in network map node detail panel
- Commented out RF Link health legend (Good signal / Degraded / Critical) from map legend
- Backend topology.ex code untouched — `compute_rf_link_stats/1` and `enrich_edges_with_rf/2` remain
- No RF tab exists on the device show page, so no changes needed there
### Verification
- `mix compile --warnings-as-errors` — zero warnings
- `mix test` — 8490 tests pass, 0 failures
Paperclip issue: TOW-44
Reviewed-on: graham/towerops-web#45
