This commit addresses multiple CRITICAL, HIGH, and MEDIUM severity security vulnerabilities identified in the security audit:
CRITICAL FIXES:
- Fix weak RNG for recovery codes - replaced Enum.random() with :crypto.strong_rand_bytes/1 for cryptographically secure token generation
- Fix subscription limit race conditions - moved free org and device quota checks inside transactions with FOR UPDATE locks to prevent concurrent bypass
- Fix default organization race condition - moved is_default check inside transaction to prevent multiple defaults per user
HIGH SEVERITY FIXES:
- Fix agent token deletion race condition - moved PubSub broadcast inside transaction to ensure agents only receive notification after successful deletion
MEDIUM SEVERITY FIXES:
- Fix LIKE wildcard injection in search - applied sanitize_like() to all user-facing search queries in devices.ex, sites.ex, and gaiia.ex to prevent enumeration attacks
- Fix Jason.decode! DoS - replaced with safe Jason.decode/1 with error handling in device_live/index.ex
- Fix SSRF vulnerability - added URL validation in HTTP executor to block requests to private/internal IP ranges (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16)
- Fix error information leakage - replaced inspect() in API responses with generic error messages, logging details server-side only
- Fix atom table pollution - HTTP method normalization now uses whitelist mapping instead of String.to_atom()
SECURITY IMPROVEMENTS:
- All quota checks now use pessimistic locking (SELECT FOR UPDATE) to prevent TOCTOU race conditions
- Private IP validation prevents cloud metadata service access (169.254.169.254)
- DNS resolution performed before HTTP requests to detect IP spoofing
- Error details logged server-side but not exposed to clients
Files changed:
- lib/towerops/accounts/user_recovery_code.ex
- lib/towerops/organizations.ex
- lib/towerops/devices.ex
- lib/towerops/sites.ex
- lib/towerops/gaiia.ex
- lib/towerops/agents.ex
- lib/towerops/monitoring/executors/http_executor.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/mib_controller.ex
- lib/towerops_web/controllers/api/v1/agent_release_webhook_controller.ex
- lib/towerops_web/controllers/api/v1/geoip_controller.ex
Reviewed-on: graham/towerops-web#108
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.
- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency
Reviewed-on: graham/towerops-web#104
Migrate three pure-function modules from Elixir to Gleam:
- polling_offset: deterministic hash-based polling offset (pure Gleam, no FFI)
- changelog_parser: text parsing logic in Gleam, thin Elixir wrapper for File I/O
- user_agent_parser: regex-based UA parsing in Gleam with Erlang FFI for
atom-keyed map conversion
Add gleam_regexp dependency for regex support in Gleam modules.
Update all call sites and tests to use Gleam module atoms.
Reviewed-on: graham/towerops-web#103
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.
Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.
Reviewed-on: graham/towerops-web#98
Protobuf oneof fields use a single :config key with tagged tuples
like {:dns, %DnsCheckConfig{}} rather than individual keys. The
check_type_config functions were setting e.g. dns: %DnsCheckConfig{}
directly, causing KeyError crashes in Check.__struct__/1.
Reviewed-on: graham/towerops-web#101
Every timeout stack trace pointed at ssl_gen_statem.call/2 — the OTP 27
SSL gen_statem hanging during recv. PG is on private network (10.0.15.20)
and SSL verify was already disabled (verify_none), providing no security
benefit while adding CPU overhead and the gen_statem layer that caused
all connection timeouts.
Removed DATABASE_SSL and DATABASE_SSL_VERIFY env vars from both the
init container and main container. Also disabled SSL on the PG server
(ssl=off in postgresql.conf, hostssl→host in pg_hba.conf).
Reviewed-on: graham/towerops-web#97
Pods were running at 500m/500m (100% of limit) with 44-47% of scheduling
periods being CPU-throttled. SSL decryption is CPU-intensive — when the
BEAM gets throttled mid ssl_gen_statem:call/2, the SSL recv stalls until
the cgroup allows CPU again, causing the 15s DBConnection timeout.
This was the actual root cause of "client timed out because it queued
and checked out the connection for longer than 15000ms" errors. Previous
fixes (TCP keepalive, pool tuning, oban pruner) were treating symptoms.
- CPU limit: 500m → 2000m (4 cores burst capacity)
- CPU request: 100m → 500m (guarantee adequate baseline)
Reviewed-on: graham/towerops-web#96
Default Pruner settings (10k rows/30s = 333/s) can't keep up with job
completion rate (~600+/s from SNMP polling + sync workers), causing
oban_jobs to accumulate millions of completed rows. The bloated table
(47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking
on IO and holding DB connections until they timeout at 15s — the actual
root cause of the "ssl recv: closed" connection pool exhaustion.
Increase to 50k rows/5s (10,000/s) to keep the table small.
Reviewed-on: graham/towerops-web#95
- TCP keepalive more aggressive (14s detection vs 30s) so dead
connections are found before the 15s checkout timeout fires
- Reduce pool size 30→15 per pod (2 pods = 30 total, avoids
exhaustion during rolling deploys when 4 pods run briefly)
- Add idle_interval: 1s to proactively ping idle connections and
detect dead SSL sockets before they're handed to real queries
- Simplify SSL config: remove redundant verify_fun and SNI disable
when verify_none already skips certificate validation
Reviewed-on: graham/towerops-web#93
## Summary
- **DB pool exhaustion**: `device_has_effective_agent?` was doing 2 DB queries per call (3-table join + global settings lookup), called 2-3x per worker cycle across hundreds of devices (~6000 unnecessary queries/minute). Added an ETS-backed `AgentCache` with 120s TTL and removed redundant calls within the same worker cycle.
- **Settings page crash**: `KeyError: key :user_agent not found` on BrowserSession — field doesn't exist, replaced with `device_type`.
- **Login history IP truncation**: Changed from side-by-side grid to stacked table layout so IP addresses aren't clipped.
## Test plan
- [x] 8 new tests for AgentCache (cache hit, miss, invalidation, global default)
- [x] Full test suite passes (8651 tests, 0 failures, verified across multiple runs)
- [x] `mix format` and pre-commit hooks pass
Reviewed-on: graham/towerops-web#88
- Downgrade check executor skip log from info to debug to stop flooding
production logs every 60s for every agent-assigned device check
- Fix KeyError on session.user_agent in settings page - BrowserSession
schema has device_type field, not user_agent
Reviewed-on: graham/towerops-web#87
- Wrap agent name + icon inside the offline amber badge so it's clear
what is offline on the device detail page
- Add "Agent:" label before the agent indicator for all states
- Fix TimeSeriesTest deadlock: switch to async: false since check_results
is a TimescaleDB hypertable and concurrent chunk creation deadlocks
- Fix unicode property test: String.length/1 counts grapheme clusters,
not code points; combining chars can make a 2-codepoint string have
only 1 grapheme, legitimately failing the min: 2 validation
Reviewed-on: graham/towerops-web#83
## Summary
- **`list_checks_for_agent` cascade**: the agent query only matched checks with `agent_token_id` set directly on the check record — checks on devices inheriting their agent via site/org/global default were never sent to the Go agent. Now uses left joins with the full cascade (device assignment → site → org default → global default cloud poller), matching the pattern from `list_agent_polling_targets`.
- **At-risk summary double-counting**: `get_impact_summary` called `analyze_device_impact` per down AP, and each returned the full site subscriber count. With 3 APs down at a 147-sub site it showed 441 subs / $31k instead of 147 / $10k. Fixed by deduplicating impact per site. Same fix applied to `calculate_down_impact` in site impact summaries.
## Test plan
- [x] 3 new cascade tests for `list_checks_for_agent` (site, org default, global default)
- [x] 3 new tests for `get_impact_summary` (single site dedup, multi-site sum, zero case)
- [x] All 8,607 existing tests pass
- [ ] Deploy to staging, verify At Risk numbers on dashboard
- [ ] Verify Go agent receives DNS/HTTP checks via cascade
Reviewed-on: graham/towerops-web#79
## Summary
- Replace guard-clause-based `should_skip_snmp/service/ping_check?` with a single `should_skip_check?` function using runtime `in` operator checks
- Guard clauses with module attributes (`@service_check_types`) were not working in Dokku buildpack deployments despite code being correct locally
- Also update `CheckWorker` (dead code safety net) to check `device_has_effective_agent?`
- Use `Logger.info` for skip messages so they appear in staging logs
## Context
PR #77 introduced `device_has_effective_agent?/1` and updated all workers, but the guard-clause-based skip functions in `CheckExecutorWorker` weren't matching at runtime on staging despite passing all tests locally. The DNS check kept producing NXDOMAIN errors from Erlang's `:inet_res` resolver.
Root cause: The `when check_type in @service_check_types` guard clause compiled against module attributes appears to have stale evaluation in the Dokku Heroku buildpack release. Moving the check to a runtime function body fixes the issue.
## Test plan
- [x] All 230 worker tests pass
- [x] All 99 agents tests pass
- [x] Verified on staging: "Skipping check eceb3d51... (dns)" appears in info logs
- [x] No new NXDOMAIN results created after deploy
Reviewed-on: graham/towerops-web#78
## Summary
- Add `device_has_effective_agent?/1` to Agents context that checks full cascade (device → site → org → global default) without distinguishing between cloud pollers and local agents
- Update check_executor_worker, device_poller_worker, and device_monitor_worker to use new function
- Phoenix never executes checks (DNS, ping, HTTP, TCP, SSL, SNMP) when any agent handles the device
## Problem
The global default cloud poller was not being checked in `should_phoenix_poll_device?/1`, causing Phoenix to run DNS/ping/service checks server-side even when the Go agent was handling the device. This produced duplicate results with NXDOMAIN errors from the Erlang DNS resolver.
## Test plan
- [x] 6 new tests for `device_has_effective_agent?/1` covering all cascade levels
- [x] Full test suite passes (8601 tests)
- [ ] Verify staging DNS check stops showing NXDOMAIN errors after deploy
Reviewed-on: graham/towerops-web#77
Chart.js was showing integer-rounded values like '2' instead of '2.1 ms'. Uses toFixed(1) in tick callbacks for proper decimal display with units.
Reviewed-on: graham/towerops-web#76
DNS check graph now shows two datasets: Response Time (ms) on left y-axis and Status (Up/Down) on right y-axis with stepped green fill. Also adds space before units in chart labels.
Reviewed-on: graham/towerops-web#74
Check.changeset/2 did not include state fields (current_state,
current_state_type, current_check_attempt, last_check_at, etc.) in
its cast list. Ecto silently dropped all state updates, so
update_check_state succeeded but wrote nothing to the database.
Added Check.state_changeset/2 for internal state machine updates,
keeping the regular changeset safe from user input. Updated
Monitoring.update_check_state/3 to use it directly.
Reviewed-on: graham/towerops-web#73
## Summary
- **DNS checks**: `parse_server` returned a flat 5-tuple `{a,b,c,d,53}` instead of `{{a,b,c,d}, 53}` — the format `:inet_res` requires for nameservers. All DNS checks using a custom server failed with argument error.
- **Ping checks**: `System.cmd` `:timeout` option not supported on Elixir 1.19.4 (staging). Replaced with `Task.async`/`Task.yield` wrapper for cross-version compatibility.
- **Checks table UI**: Removed unnecessary flex div wrapper on status badge column that caused vertical misalignment with other table cells.
## Test plan
- [x] DNS executor tests pass (custom server resolution now succeeds)
- [x] Ping executor tests pass (no more ArgumentError)
- [x] Full test suite passes (8593 tests, 0 new failures)
Reviewed-on: graham/towerops-web#72
## Summary
- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever
## Changes
- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience
## Test plan
- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)
Reviewed-on: graham/towerops-web#71
## Summary
- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)
## Test plan
- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal
Reviewed-on: graham/towerops-web#68
Moves e2e tests from production workflow into a separate workflow that requires manual triggering. This allows e2e tests to be run on-demand without blocking production deployments.
Changes:
- Created new .forgejo/workflows/e2e-tests.yaml with workflow_dispatch trigger
- Removed disabled e2e job from production.yaml workflow
- E2E tests can now be triggered manually from Actions tab with browser selection (all, chromium, firefox, webkit)
- Test results and videos uploaded as artifacts on all runs
Reviewed-on: graham/towerops-web#66
add ssl check executor that connects via TLS, reads peer certificate
expiry, and returns OK/WARNING/CRITICAL based on configurable
warning_days threshold. includes protobuf config, liveview form fields,
agent channel encoding, oban worker dispatch, and unit tests.
Reviewed-on: graham/towerops-web#64
- Skip HTTP/TCP/DNS checks in Phoenix when device is assigned to agent
- Agents now execute service checks from the device's network location
- Fix cancel button in service check modal (text link instead of button)
- Fix status badge alignment in checks table
Reviewed-on: graham/towerops-web#63
The docker buildx driver (required on unprivileged ci runner) does not
support registry cache export/import. Remove cache-from and cache-to
options to unblock builds.
Reviewed-on: graham/towerops-web#62
overlayfs requires mount/unmount capabilities not available on the
unprivileged ci runner. Switch to vfs storage driver and use buildx
docker driver (not docker-container) to avoid nested mount issues.
Reviewed-on: graham/towerops-web#61
add early bail in DiscoveryWorker when snmp_enabled is false.
mark snmp_not_enabled and device_unresponsive as non-retryable
errors to prevent wasted retries.
Reviewed-on: graham/towerops-web#58
The new CI server doesn't have Docker daemon running by default,
causing buildx setup to fail with:
ERROR: failed to connect to the docker API at unix:///var/run/docker.sock
This fix:
- Adds explicit Docker daemon startup check and initialization
- Configures buildx with docker-container driver
- Adds install: true to ensure proper buildx installation
This was already fixed in build-deploy.yml (commit 54280fd7) but
was missing from production.yaml.
Reviewed-on: graham/towerops-web#57
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
send check_jobs protobuf to agents during job dispatch cycle, handle
check_result messages back with state transitions and alert
creation/resolution. auto-assign checks to device's effective agent
token on creation, broadcast check changes via PubSub so connected
agents get immediate updates.
add edit/delete support for service checks on device show page with
form pre-population from existing check config. auto-fill TCP host
from device IP address.
includes validate_check_result/1 for protobuf validation and
list_checks_for_agent/2 for querying agent-assigned service checks.
also fixes pre-existing broken device_type_icon reference in device
index template.
Reviewed-on: graham/towerops-web#50
Extend the GraphQL API and socket to accept mobile session tokens
alongside existing API tokens. Add OrganizationScope middleware
that extracts organization_id from query args for mobile users.
Add my_organizations query for mobile app org listing.
Reviewed-on: graham/towerops-web#48
- Commented out 'View RF Links' button in network map node detail panel
- Commented out RF Link health legend (Good/Degraded/Critical) from map legend
- Backend code (topology.ex) left intact per instructions
- No RF tab found on device show page (nothing to hide there)
- All tests pass, compiles clean with --warnings-as-errors
Reviewed-on: graham/towerops-web#46
