When an agent joins, we update last_seen_at in the database but don't set
last_heartbeat_db_update in the socket. This causes the heartbeat throttling
logic to be out of sync:
- On join: DB updated, but last_heartbeat_db_update not set (nil)
- First heartbeat: Updates DB again because last_heartbeat_db_update is nil
- Subsequent heartbeats: Check if >30s since last DB update
- Problem: last_heartbeat_db_update tracks the first heartbeat, not the join DB update
This manifests when Phoenix recompiles during development - the channel process
continues running with old socket assigns, and if last_heartbeat_db_update is
recent, heartbeats won't update the DB. After 10+ minutes without DB updates,
the agent appears offline even though it's still connected.
Fix: Set last_heartbeat_db_update when we update the DB on join, so the
throttling logic accurately tracks the most recent DB update.
Reviewed-on: graham/towerops-web#142
The Check protobuf has a `oneof config` field which generates separate
fields in the Elixir struct: :http, :tcp, :dns, :ssl. However,
check_type_config/2 was returning [config: {:http, ...}] which doesn't
match the struct definition, causing a KeyError crash in
build_and_push_check_jobs/1.
This bug was originally fixed in e0a74e2f but was accidentally
reintroduced in f703e61b when attempting to use "tagged tuple config".
Fix: Return field-specific keyword lists (e.g., [http: %HttpCheckConfig{}])
instead of [config: {:http, ...}].
Also added comprehensive test that creates all 4 check types and verifies
no KeyError crash occurs when sending check jobs to agent.
Reviewed-on: graham/towerops-web#141
## Problem
On backhaul devices, the Y axis auto-scale includes capacity reference line values (e.g. 100 Mbps) alongside actual traffic (e.g. 5 Mbps). This makes the traffic data appear as a flat line near zero — effectively invisible.
## Fix
Exclude datasets with `Capacity` prefix from the max-value calculation used for symmetric Y axis scaling. The capacity dashed lines still render and clip at the chart edges as a visual ceiling reference.
## 1-line change
```js
if (dataset.label?.startsWith("Capacity")) return
```
Reviewed-on: graham/towerops-web#140
Removes the commented-out `apply_inferred_role` function that was left behind when device type was simplified to manual-only.
Split from #135.
Reviewed-on: graham/towerops-web#137
CRITICAL FIXES:
- Fix unsafe pattern matching in SNMP discovery that would crash on timeout
- Changed {:ok, _} = ... to proper case statements with error handling
- Extracted safe_discover/3 helper to reduce cyclomatic complexity
- Added fallback to empty lists and warning logging for timeouts
- Affects: discover_vlans, discover_ip_addresses, discover_processors, discover_storage
MEMORY LEAK FIXES (JS Hooks):
- GlobalSearchTrigger: Added destroyed() cleanup for click listener
- SidebarCollapse: Added destroyed() cleanup for click listener
- ThemeSelector: Added destroyed() cleanup for phx:set-theme window listener
- NetworkMap: Added destroyed() cleanup for 3 button listeners (zoom in/out/fit)
- All hooks now properly store handler references and remove listeners on unmount
LIVEVIEW FIXES:
- Added phx-update="ignore" to 5 SensorChart hooks to prevent chart re-initialization
- Added catch-all handle_info(_msg, socket) to 5 LiveViews to prevent crashes on unexpected messages
- device_live/show.ex, device_live/index.ex, alert_live/index.ex, activity_feed_live.ex, agent_live/index.ex
N+1 QUERY FIXES:
- Created Gaiia.get_site_subscriber_summaries/1 batch function
- Refactored alert_live and device_live to use batch query instead of looping
- Reduces queries from N to 1 when loading site subscriber data
Impact:
- Prevents discovery worker crashes during SNMP timeouts
- Eliminates memory accumulation from leaked event handlers
- Prevents chart state loss on LiveView updates
- Prevents LiveView crashes from unexpected PubSub messages
- Improves database performance by eliminating N+1 queries in alert/device views
- Passes credo --strict with 0 issues
Reviewed-on: graham/towerops-web#132
- Add vendored snmpkit/snmp_lib paths in src/ to ignore patterns
- Document and ignore Gleam<->Erlang type system mismatches
- Gleam's Int doesn't map to non_neg_integer() (contract_supertype)
- Gleam's opaque types not understood by Dialyzer (call_without_opaque)
- All Gleam code compiles and passes tests - these are false positives
Result: Zero dialyzer errors in our own code (1284/1311 skipped)
Reviewed-on: graham/towerops-web#131
Dependency update PRs from Renovate don't need to be deployed to
staging. Uses job-level condition to check head branch prefix.
Reviewed-on: graham/towerops-web#124
check_type_config was returning `config: {:http, ...}` but the
Check struct has no :config field — it has :http, :tcp, :dns, :ssl
directly. This caused a KeyError crash in build_and_push_check_jobs.
Reviewed-on: graham/towerops-web#121
Seconds-level precision adds noise without value for these fields.
Shows "less than a minute" instead of "0 seconds" for very short durations.
Reviewed-on: graham/towerops-web#120
Incremental compilation can miss schema field removals in dependent
modules, causing undefined_column errors in production.
Reviewed-on: graham/towerops-web#119
- Wrap upsert_nodes in Repo.transaction so all upserts share one DB
connection, preventing pool exhaustion under load
- Remove stale device_role_source reference from form test
Reviewed-on: graham/towerops-web#118
- Add missing flex-1/text-right classes to Resolved IP row
- Add justify-end to Operating System flex container
- Fix duplicate time labels on 24h traffic chart by excluding boundary ticks
Reviewed-on: graham/towerops-web#117
- Capacity: Sensor values (Rx/Tx Capacity) now represent total device
capacity, not per-interface. Previous bug applied the same sensor value
to all interfaces and summed them (e.g., 773.6 Mbps × 5 = 3.87 Gbps).
- Wireless sensors: Changed Float.round/2 to :erlang.float_to_binary/2
to prevent scientific notation in frequency display (was showing
"2.41e4 MHz" instead of "24100.0 MHz").
Files:
- lib/towerops_web/live/device_live/show.ex
- lib/towerops_web/live/device_live/show.html.heex
- CHANGELOG.txt
- priv/static/changelog.txt
Reviewed-on: graham/towerops-web#116
Radio frequencies (24 GHz) were displaying as "2.42e4 MHz" instead of
"24200.0 MHz" because Float.round/2 returns a float that Phoenix
converts to scientific notation for large numbers.
Changed format_sensor_value/2 to use :erlang.float_to_binary/2 with
decimals: 1 option, which formats as a string and avoids scientific
notation.
Verified: 24200.0 now displays as "24200.0" instead of "2.42e4"
Reviewed-on: graham/towerops-web#114
For devices with device_role="backhaul", traffic graphs now show capacity
reference lines that update over time based on which interfaces are active.
Features:
- Calculates total capacity per timestamp from active monitored interfaces
- Uses configured_capacity_bps if set, falls back to if_speed
- Capacity adjusts as interfaces go up/down throughout the time period
- Shows as dashed lines at +capacity (inbound) and -capacity (outbound)
- Matches the existing 0-axis graph style
Implementation:
- Small graph (device detail page): Updated build_traffic_json_datasets/2
and add_capacity_datasets/3 in device_live/show.ex
- Full-page graph: Updated build_traffic_datasets/2 and
add_capacity_to_full_graph/4 in graph_live/show.ex
The capacity lines provide visual reference for backhaul link saturation,
showing how close actual throughput is to maximum capacity at each point
in time.
Reviewed-on: graham/towerops-web#113
Add rebaseWhen: 'behind-base-branch' to automatically rebase Renovate PRs
when they fall behind main. This fixes CI failures on dependency update PRs
that were created before recent schema changes (device_role_source removal).
Without auto-rebase, Renovate PRs based on old main fail CI because:
- Old code references removed device_role_source field
- New migrations drop the column
- Tests fail with ERROR 42703 (undefined_column)
Now Renovate will automatically rebase PRs to pick up latest changes.
Reviewed-on: graham/towerops-web#111
- Removed device_role_source assignment in sanitize_device_role/1
- Updated structure.sql to reflect database schema without device_role_source column
- Fixes ERROR 42703 (undefined_column) in production workers
The form was still trying to set device_role_source in params even though
the database column was dropped, causing runtime errors in:
- AgentLatencyEvaluator
- CapacityInsightWorker
- WirelessInsightWorker
All device roles are now manually set with default value 'other'.
Reviewed-on: graham/towerops-web#110
- Reduce device type options from 10 to 6 (server, switch, router, access_point, backhaul, other)
- Change label from "Device Role" to "Device Type"
- Disable auto-detection - device type is now required and manual-only
- Add migration to map existing device roles to new simplified set
- Update REST API to include device_role and device_role_source fields
- Automatically set device_role_source to "manual" when device_role is provided
- Update wireless detection to include "backhaul" type
- Enhance display formatting for "Access Point"
Breaking Changes:
- Auto-inference of device type has been disabled
- device_role is now a required field in forms
- GraphQL/REST API consumers will see changed device_role values after migration
Files Changed:
- lib/towerops_web/live/device_live/form.html.heex
- lib/towerops/topology.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/devices/device.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/devices_controller.ex
- priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs
- test/towerops/topology_test.exs
- test/towerops_web/live/device_live/form_test.exs
Reviewed-on: graham/towerops-web#109
This commit addresses multiple CRITICAL, HIGH, and MEDIUM severity security vulnerabilities identified in the security audit:
CRITICAL FIXES:
- Fix weak RNG for recovery codes - replaced Enum.random() with :crypto.strong_rand_bytes/1 for cryptographically secure token generation
- Fix subscription limit race conditions - moved free org and device quota checks inside transactions with FOR UPDATE locks to prevent concurrent bypass
- Fix default organization race condition - moved is_default check inside transaction to prevent multiple defaults per user
HIGH SEVERITY FIXES:
- Fix agent token deletion race condition - moved PubSub broadcast inside transaction to ensure agents only receive notification after successful deletion
MEDIUM SEVERITY FIXES:
- Fix LIKE wildcard injection in search - applied sanitize_like() to all user-facing search queries in devices.ex, sites.ex, and gaiia.ex to prevent enumeration attacks
- Fix Jason.decode! DoS - replaced with safe Jason.decode/1 with error handling in device_live/index.ex
- Fix SSRF vulnerability - added URL validation in HTTP executor to block requests to private/internal IP ranges (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16)
- Fix error information leakage - replaced inspect() in API responses with generic error messages, logging details server-side only
- Fix atom table pollution - HTTP method normalization now uses whitelist mapping instead of String.to_atom()
SECURITY IMPROVEMENTS:
- All quota checks now use pessimistic locking (SELECT FOR UPDATE) to prevent TOCTOU race conditions
- Private IP validation prevents cloud metadata service access (169.254.169.254)
- DNS resolution performed before HTTP requests to detect IP spoofing
- Error details logged server-side but not exposed to clients
Files changed:
- lib/towerops/accounts/user_recovery_code.ex
- lib/towerops/organizations.ex
- lib/towerops/devices.ex
- lib/towerops/sites.ex
- lib/towerops/gaiia.ex
- lib/towerops/agents.ex
- lib/towerops/monitoring/executors/http_executor.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/mib_controller.ex
- lib/towerops_web/controllers/api/v1/agent_release_webhook_controller.ex
- lib/towerops_web/controllers/api/v1/geoip_controller.ex
Reviewed-on: graham/towerops-web#108
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.
- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency
Reviewed-on: graham/towerops-web#104
Migrate three pure-function modules from Elixir to Gleam:
- polling_offset: deterministic hash-based polling offset (pure Gleam, no FFI)
- changelog_parser: text parsing logic in Gleam, thin Elixir wrapper for File I/O
- user_agent_parser: regex-based UA parsing in Gleam with Erlang FFI for
atom-keyed map conversion
Add gleam_regexp dependency for regex support in Gleam modules.
Update all call sites and tests to use Gleam module atoms.
Reviewed-on: graham/towerops-web#103
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.
Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.
Reviewed-on: graham/towerops-web#98
Protobuf oneof fields use a single :config key with tagged tuples
like {:dns, %DnsCheckConfig{}} rather than individual keys. The
check_type_config functions were setting e.g. dns: %DnsCheckConfig{}
directly, causing KeyError crashes in Check.__struct__/1.
Reviewed-on: graham/towerops-web#101
Every timeout stack trace pointed at ssl_gen_statem.call/2 — the OTP 27
SSL gen_statem hanging during recv. PG is on private network (10.0.15.20)
and SSL verify was already disabled (verify_none), providing no security
benefit while adding CPU overhead and the gen_statem layer that caused
all connection timeouts.
Removed DATABASE_SSL and DATABASE_SSL_VERIFY env vars from both the
init container and main container. Also disabled SSL on the PG server
(ssl=off in postgresql.conf, hostssl→host in pg_hba.conf).
Reviewed-on: graham/towerops-web#97
Pods were running at 500m/500m (100% of limit) with 44-47% of scheduling
periods being CPU-throttled. SSL decryption is CPU-intensive — when the
BEAM gets throttled mid ssl_gen_statem:call/2, the SSL recv stalls until
the cgroup allows CPU again, causing the 15s DBConnection timeout.
This was the actual root cause of "client timed out because it queued
and checked out the connection for longer than 15000ms" errors. Previous
fixes (TCP keepalive, pool tuning, oban pruner) were treating symptoms.
- CPU limit: 500m → 2000m (4 cores burst capacity)
- CPU request: 100m → 500m (guarantee adequate baseline)
Reviewed-on: graham/towerops-web#96
Default Pruner settings (10k rows/30s = 333/s) can't keep up with job
completion rate (~600+/s from SNMP polling + sync workers), causing
oban_jobs to accumulate millions of completed rows. The bloated table
(47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking
on IO and holding DB connections until they timeout at 15s — the actual
root cause of the "ssl recv: closed" connection pool exhaustion.
Increase to 50k rows/5s (10,000/s) to keep the table small.
Reviewed-on: graham/towerops-web#95
- TCP keepalive more aggressive (14s detection vs 30s) so dead
connections are found before the 15s checkout timeout fires
- Reduce pool size 30→15 per pod (2 pods = 30 total, avoids
exhaustion during rolling deploys when 4 pods run briefly)
- Add idle_interval: 1s to proactively ping idle connections and
detect dead SSL sockets before they're handed to real queries
- Simplify SSL config: remove redundant verify_fun and SNI disable
when verify_none already skips certificate validation
Reviewed-on: graham/towerops-web#93
## Summary
- **DB pool exhaustion**: `device_has_effective_agent?` was doing 2 DB queries per call (3-table join + global settings lookup), called 2-3x per worker cycle across hundreds of devices (~6000 unnecessary queries/minute). Added an ETS-backed `AgentCache` with 120s TTL and removed redundant calls within the same worker cycle.
- **Settings page crash**: `KeyError: key :user_agent not found` on BrowserSession — field doesn't exist, replaced with `device_type`.
- **Login history IP truncation**: Changed from side-by-side grid to stacked table layout so IP addresses aren't clipped.
## Test plan
- [x] 8 new tests for AgentCache (cache hit, miss, invalidation, global default)
- [x] Full test suite passes (8651 tests, 0 failures, verified across multiple runs)
- [x] `mix format` and pre-commit hooks pass
Reviewed-on: graham/towerops-web#88
- Downgrade check executor skip log from info to debug to stop flooding
production logs every 60s for every agent-assigned device check
- Fix KeyError on session.user_agent in settings page - BrowserSession
schema has device_type field, not user_agent
Reviewed-on: graham/towerops-web#87
- Wrap agent name + icon inside the offline amber badge so it's clear
what is offline on the device detail page
- Add "Agent:" label before the agent indicator for all states
- Fix TimeSeriesTest deadlock: switch to async: false since check_results
is a TimescaleDB hypertable and concurrent chunk creation deadlocks
- Fix unicode property test: String.length/1 counts grapheme clusters,
not code points; combining chars can make a 2-codepoint string have
only 1 grapheme, legitimately failing the min: 2 validation
Reviewed-on: graham/towerops-web#83
