OnCall context
- move_layer/2: up/down position swap for schedule layers, mirror of
move_escalation_rule/2 (tested for both directions + boundary no-op).
ScheduleLive.Show
- New move_layer event handler.
- Up/down arrow buttons on each layer card; the top arrow is disabled on
the first layer, the bottom on the last (matches the escalation policy
level reorder UX shipped in chunk 1).
- Replaced two per-hour walks (build_layer_spans + build_final_spans -
~672 resolve calls per render at 14 days) with Resolver.resolve_range/3
+ a small segments_to_spans/3 mapper. Per-layer view wraps each layer
in a temporary single-layer %Schedule{} so the resolver code path is
shared and overrides don't bleed into the per-layer strip.
Tests: 3 new context tests for move_layer, 1 new integration test for
the layer reorder UI. Full suite 10,214 / 0 failures. Format/dialyzer
clean.
Resolver
- Resolver.resolve_range/3: returns the contiguous on-call segments for a
schedule across [start_at, end_at). Walks layer handoffs + override
boundaries, drops gap segments, and merges adjacent same-user segments.
UserColor module
- New Towerops.OnCall.UserColor with deterministic id -> color mapping
(Tailwind class + matching #RRGGBB hex). Refactored ScheduleLive.Show
to use it instead of its own per-page index palette so the same person
gets the same color across the schedules list and detail pages.
ScheduleLive.Index
- Date navigation: Today / prev / next + 1/2/4-week range selector.
- URL-driven: ?start=YYYY-MM-DD&range=N for shareable views; falls back
to defaults on malformed values.
- Per-row card: schedule name link, on-call-now avatar swatch, day-of-week
header strip, gantt strip rendered as a CSS grid with one column per day,
Today marker overlay.
Tests: 5 new resolver tests, 9 new UserColor tests, 5 new ScheduleLive
integration tests. Full suite: 10,210 / 0 failures. Format/dialyzer clean.
Also: cleaned up two stale inline comments in k8s/deployment.yaml that
disagreed with the values they sat next to (replicas/maxSurge).
Chunk 1 of the on-call/escalation redesign plan
(docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md):
Schema
- on_call.ex: list_devices_using_policy/2 (direct + org-default inheritance)
- on_call.ex: move_escalation_rule/2 for up/down level reordering
- list_escalation_policies/1 now preloads :rules
Show page
- Levels rendered as numbered cards with up/down/delete controls
- "Notify the following users immediately and escalate after N minutes."
- REPEAT footer surfaces repeat_count inline (no longer a standalone column)
- Right-side Services sidebar lists devices using the policy
- Header line shows the handoff notifications mode (when in use by a service /
always / never)
Edit form
- Removed standalone Repeat Count input
- Added handoff notifications mode select
- Added inline "If no one acknowledges, repeat this policy N time(s)" block
Index views (policies tab + standalone)
- Replaced Repeat Count column with a Levels (rule count) column
Other
- rate_limit.ex: bind unmatched :ets.new/2 + schedule_clean/1 returns to fix
dialyzer :unmatched_returns warning under the project's strict flags.
All 10,191 tests pass; mix format clean; mix compile --warnings-as-errors clean;
mix dialyzer passes.
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
Oban 2.18+ introduced the 'suspended' job state for pausing jobs.
Oban.Met.Reporter queries for jobs in this state, causing errors when
the database enum doesn't include it:
ERROR 22P02 (invalid_text_representation) invalid input value
for enum oban_job_state: "suspended"
This migration adds 'suspended' to the oban_job_state enum type.
Required by recent Oban dependency updates (commit 0399b30d).
Note: Uses @disable_ddl_transaction since ALTER TYPE ADD VALUE cannot
run inside a transaction in PostgreSQL.
Reviewed-on: graham/towerops-web#199
Phoenix.PubSub.Redis 3.1.0 automatically appends sync_connect: true to the
redis_opts before passing to Redix.PubSub.start_link. Our configuration was
also passing sync_connect: false, creating a duplicate key that NimbleOptions
rejected with a confusing error message.
The error showed:
NimbleOptions.ValidationError: unknown options [:sync_connect],
valid options are: [..., :sync_connect, ...]
This happened because sync_connect appeared twice in the keyword list,
and NimbleOptions rejects duplicate keys.
The fix removes our sync_connect option entirely, letting phoenix_pubsub_redis
control the sync_connect behavior.
This completes the phoenix_pubsub_redis 3.1.0 migration started in cd660387.
Fixes production pod crash on startup.
Reviewed-on: graham/towerops-web#198
The recent dependency update to phoenix_pubsub_redis 3.1.0 introduced
breaking changes in the configuration API. The new version no longer
accepts connection options at the top level - they must be nested in
:redis_opts.
Changes:
- Move Redix connection options into :redis_opts keyword list
- Keep only :node_name at top level per v3.1.0 requirements
- Maintains all resilience settings (timeouts, backoff, keepalive)
Before (invalid for v3.1.0):
[node_name: node(), host: "...", port: 6379, timeout: 5000, ...]
After (valid for v3.1.0):
[node_name: node(), redis_opts: [host: "...", port: 6379, timeout: 5000, ...]]
This fixes the production crash with error:
NimbleOptions.ValidationError: unknown options [:timeout, :backoff_initial,
:backoff_max, :exit_on_disconnection, :sync_connect]
Related: commit 0399b30d (Update Elixir dependencies)
Reviewed-on: graham/towerops-web#197
Fixes FunctionClauseError when DashboardLive receives {:alert_changed, org_id}
message broadcast by user_auth hooks.
The user_auth module subscribes all LiveViews to the
"organization:#{org_id}:alerts" topic and broadcasts {:alert_changed, org_id}
messages when alerts are created or resolved. DashboardLive was missing a
handle_info/2 clause to handle this message format, causing crashes in
production.
Solution:
- Added handle_info/2 clause for {:alert_changed, _org_id} message
- Uses existing debounced reload pattern to update dashboard
- Added test to verify message is handled without crashing
- Updated both technical and user-facing changelogs
All tests pass. The fix follows the same pattern used in AlertLive.Index
and DeviceLive.Show.
Reviewed-on: graham/towerops-web#189
Add complete memory pool monitoring using HOST-RESOURCES-MIB.
Tracks RAM/swap usage with historical data and real-time updates.
- Add Mempool and MempoolReading schemas
- Integrate discovery and polling with concurrent Task.async_stream
- Add 7 context API functions for querying mempools
- 24 new tests, all 8754 tests passing (0 failures)
Implements C1 from findings_librenms.md
Reviewed-on: graham/towerops-web#184
Commented out weathermap navigation in sidebar and mobile menu.
Feature not ready for production use yet.
Files: lib/towerops_web/components/layouts.ex
Changelog: CHANGELOG.txt
Reviewed-on: graham/towerops-web#174
- Capacity: Sensor values (Rx/Tx Capacity) now represent total device
capacity, not per-interface. Previous bug applied the same sensor value
to all interfaces and summed them (e.g., 773.6 Mbps × 5 = 3.87 Gbps).
- Wireless sensors: Changed Float.round/2 to :erlang.float_to_binary/2
to prevent scientific notation in frequency display (was showing
"2.41e4 MHz" instead of "24100.0 MHz").
Files:
- lib/towerops_web/live/device_live/show.ex
- lib/towerops_web/live/device_live/show.html.heex
- CHANGELOG.txt
- priv/static/changelog.txt
Reviewed-on: graham/towerops-web#116
- Reduce device type options from 10 to 6 (server, switch, router, access_point, backhaul, other)
- Change label from "Device Role" to "Device Type"
- Disable auto-detection - device type is now required and manual-only
- Add migration to map existing device roles to new simplified set
- Update REST API to include device_role and device_role_source fields
- Automatically set device_role_source to "manual" when device_role is provided
- Update wireless detection to include "backhaul" type
- Enhance display formatting for "Access Point"
Breaking Changes:
- Auto-inference of device type has been disabled
- device_role is now a required field in forms
- GraphQL/REST API consumers will see changed device_role values after migration
Files Changed:
- lib/towerops_web/live/device_live/form.html.heex
- lib/towerops/topology.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/devices/device.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/devices_controller.ex
- priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs
- test/towerops/topology_test.exs
- test/towerops_web/live/device_live/form_test.exs
Reviewed-on: graham/towerops-web#109
## Summary
- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever
## Changes
- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience
## Test plan
- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)
Reviewed-on: graham/towerops-web#71
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
- Increase node font from 11px to 13px, compound labels to 14px
- Replace cose physics layout with deterministic grid preset layout to eliminate site group overlap
- Fix nil source_interface_id crash in topology.ex find_existing_link/1 (was using == nil instead of is_nil/1)
- Fix nil device label in device_to_node/1 fallback to IP or "Unknown"
- Replace filter tabs with pill buttons (All/Unmatched/Matched)
- Add IP-based match suggestions when AP IP matches a TowerOps device
- Show "Match found" badge on rows with suggestions; sort suggestions first
- Show suggestion chips in inline linking row for one-click linking
- Make unlinked AP rows clickable to open linking panel
- Make IP addresses clickable http links
- Improve linking row layout and descriptive copy
- Replace bare dash with "Not linked" pill in Linked Device column
- Hide less-critical table columns on small screens (IP, subs, MRR, etc.)
- Add overflow-x-auto wrappers on tables to prevent horizontal scroll
- Fix <.header> component to flex-wrap action buttons on mobile
- Make page action button labels icon-only on small screens
- Add min-w-0 on grid children to prevent viewport overflow
- Right-align device detail dt/dd values consistently
- Fix schedule timeline cards with overflow-x-auto wrappers
- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
The dashboard status indicator favicon (colored circle) uses canvas
to generate a data URL that's set as the favicon. This was blocked
by CSP in staging/production because default-src didn't include data:.
Added data: to default-src directive to fix favicon not updating in
browser tabs when device status changes.
- Add :admin type to RateLimit plug with 100 requests per minute per IP
- Apply rate limiting to all /admin routes (LiveView and controller actions)
- Protects superuser endpoints from abuse and brute force attempts
- Aligns with existing auth (10 req/min) and API (1000 req/min) limits
The test alias was prompting 'Are you sure you want to proceed?' when
the database structure was already loaded, requiring manual confirmation
on every test run.
Add --skip-if-loaded flag to ecto.load command so it silently skips
structure loading if schema_migrations table already exists.
Now 'mix test' runs without user interaction.
Fixes fragmented translation that displayed raw '%{link}' placeholder text
instead of properly interpolated link in development mail adapter notice.
Changes:
- Consolidate separate translation calls into single interpolated string
- Use raw() helper to safely inject HTML link into translated text
- Update Spanish translation to include %{link} placeholder
- Extract updated translations with mix gettext.extract
Before: "To see sent emails, visit %{link}. the mailbox page."
After: "To see sent emails, visit the mailbox page." (with proper link)
Replace sequential migration execution (4m8s for 172 migrations) with
structure.sql loading (416ms), reducing CI database setup time by 99.7%.
Changes:
- Configure Ecto to dump/load schema from priv/repo/structure.sql
- Update test alias: ecto.migrate → ecto.load
- Remove redundant database setup step from CI workflow
- Commit structure.sql to repository for version control
Benefits:
- CI database setup: 4m8s → 416ms (248s → 0.4s)
- Consistent schema snapshots across environments
- Faster local test runs
Maintenance:
- Run 'mix ecto.dump' after adding new migrations
- Structure file auto-updates with schema changes
Batch device count queries to reduce N+1 problem in dashboard.
Changes:
- Add Devices.batch_count_site_devices/1 function
* Uses single GROUP BY query instead of N separate queries
* Returns map of {site_id => %{total: count, down: count}}
* Aggregates both total devices and down devices in one query
- Update Dashboard.get_site_impact_summaries/1 to use batch function
* Eliminates 2N queries (count_site_devices + count_site_devices_down)
* Reduces to 1 query regardless of site count
Performance impact:
- 10 sites: Reduces from 20 device count queries to 1 (95% reduction)
- 50 sites: Reduces from 100 device count queries to 1 (99% reduction)
Remaining N+1 opportunities:
- Gaiia subscriber summaries (N queries)
- Preseem QoE data (M queries where M = total devices)
- Down device listings (N queries)
Files:
- lib/towerops/devices.ex
- lib/towerops/dashboard.ex
- CHANGELOG.txt
Add logger filter to drop Kubernetes health probe logs that were flooding
production logs every few seconds.
Implementation:
- Create ToweropsWeb.TelemetryFilter module with filter_health_checks/2
- Configure as logger :default_handler filter in prod.exs
- Filters based on request_path metadata and message content
- Drops logs for GET /health and HEAD / requests
Impact:
- Eliminates ~120 log entries per minute per pod from K8s probes
- Keeps application logs focused on actual user activity and errors
- No impact on health check functionality - only suppresses logging
Files:
- lib/towerops_web/telemetry_filter.ex (new)
- lib/towerops/application.ex (attach filter on startup)
- config/prod.exs (add filter to logger config)
- CHANGELOG.txt
- Added entries for organization_id alert optimization
- Added entries for cascade delete safety improvements
- Added entry for sensor state validation
- Updated user-facing changelog with performance improvements
- All 23 bugs from reliability audit now addressed
