- Clamp negative counter deltas to zero instead of attempting 32-bit wrap
correction (matches LibreNMS behavior — wraps are indistinguishable from
HC counter resets, so discard the ambiguous data point)
- Add rate sanity check: discard values exceeding 400 Gbps as counter anomalies
- Add 4-byte binary decoding for 32-bit SNMP counters in decode_snmp_value
- Track HC vs standard counter usage per interface stat (is_hc field)
- Add migration for is_hc column on snmp_interface_stats
- Update tests to match new clamping behavior
Reviewed-on: graham/towerops-web#168
Missing indexes detected on 7 foreign key columns:
- on_call_overrides.user_id
- on_call_layer_members.user_id
- site_outages.site_id
- escalation_targets.user_id
- notification_digests.organization_id
- on_call_notifications.user_id
- alerts.site_outage_id (already existed, skipped)
Without indexes, PostgreSQL must perform sequential scans for:
- Referential integrity checks on DELETE/UPDATE of parent rows
- JOIN operations on these columns
- Can cause lock contention and slow queries
Indexes created with CONCURRENTLY option to avoid blocking production
tables during deployment.
Migration uses @disable_ddl_transaction and @disable_migration_lock
to ensure indexes are built without holding locks.
Reviewed-on: graham/towerops-web#167
Add data_retention_days field to organizations (default 365, range 30-730).
DataRetentionWorker runs nightly at 1 AM to purge expired time-series data
across 8 tables, respecting each org's retention setting.
TimescaleDB global retention bumped from 90 to 730 days as safety net.
Resolved alerts are cleaned up; unresolved alerts kept regardless of age.
Reviewed-on: graham/towerops-web#144
- Capacity: Sensor values (Rx/Tx Capacity) now represent total device
capacity, not per-interface. Previous bug applied the same sensor value
to all interfaces and summed them (e.g., 773.6 Mbps × 5 = 3.87 Gbps).
- Wireless sensors: Changed Float.round/2 to :erlang.float_to_binary/2
to prevent scientific notation in frequency display (was showing
"2.41e4 MHz" instead of "24100.0 MHz").
Files:
- lib/towerops_web/live/device_live/show.ex
- lib/towerops_web/live/device_live/show.html.heex
- CHANGELOG.txt
- priv/static/changelog.txt
Reviewed-on: graham/towerops-web#116
- Removed device_role_source assignment in sanitize_device_role/1
- Updated structure.sql to reflect database schema without device_role_source column
- Fixes ERROR 42703 (undefined_column) in production workers
The form was still trying to set device_role_source in params even though
the database column was dropped, causing runtime errors in:
- AgentLatencyEvaluator
- CapacityInsightWorker
- WirelessInsightWorker
All device roles are now manually set with default value 'other'.
Reviewed-on: graham/towerops-web#110
- Reduce device type options from 10 to 6 (server, switch, router, access_point, backhaul, other)
- Change label from "Device Role" to "Device Type"
- Disable auto-detection - device type is now required and manual-only
- Add migration to map existing device roles to new simplified set
- Update REST API to include device_role and device_role_source fields
- Automatically set device_role_source to "manual" when device_role is provided
- Update wireless detection to include "backhaul" type
- Enhance display formatting for "Access Point"
Breaking Changes:
- Auto-inference of device type has been disabled
- device_role is now a required field in forms
- GraphQL/REST API consumers will see changed device_role values after migration
Files Changed:
- lib/towerops_web/live/device_live/form.html.heex
- lib/towerops/topology.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/devices/device.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/devices_controller.ex
- priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs
- test/towerops/topology_test.exs
- test/towerops_web/live/device_live/form_test.exs
Reviewed-on: graham/towerops-web#109
## Summary
- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever
## Changes
- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience
## Test plan
- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)
Reviewed-on: graham/towerops-web#71
## Summary
- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)
## Test plan
- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal
Reviewed-on: graham/towerops-web#68
add ssl check executor that connects via TLS, reads peer certificate
expiry, and returns OK/WARNING/CRITICAL based on configurable
warning_days threshold. includes protobuf config, liveview form fields,
agent channel encoding, oban worker dispatch, and unit tests.
Reviewed-on: graham/towerops-web#64
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
The StormDetector was being added twice:
1. In the main children list (line 127)
2. In background_workers() (line 176)
This caused the supervisor to fail with 'duplicate_child_name' error.
Removed the duplicate from main children list since it's correctly
placed in background_workers() which also handles test env properly.
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- Increase node font from 11px to 13px, compound labels to 14px
- Replace cose physics layout with deterministic grid preset layout to eliminate site group overlap
- Fix nil source_interface_id crash in topology.ex find_existing_link/1 (was using == nil instead of is_nil/1)
- Fix nil device label in device_to_node/1 fallback to IP or "Unknown"
- New 'alert_routing' field on organizations (default: builtin)
- AlertNotificationWorker respects the setting:
- builtin: only built-in escalation policies
- pagerduty: only PagerDuty events
- both: fire both (useful during migration)
- Radio button UI in Settings → General with visual cards
- Warning shown when PagerDuty selected but not configured
- Migration, schema validation, 6 tests
- Hide less-critical table columns on small screens (IP, subs, MRR, etc.)
- Add overflow-x-auto wrappers on tables to prevent horizontal scroll
- Fix <.header> component to flex-wrap action buttons on mobile
- Make page action button labels icon-only on small screens
- Add min-w-0 on grid children to prevent viewport overflow
- Right-align device detail dt/dd values consistently
- Fix schedule timeline cards with overflow-x-auto wrappers
Bump desktop nav breakpoint from md (768px) to lg (1024px) so the
mobile hamburger menu is used on medium-width screens instead of
nav items overflowing outside the viewport.
Clear all 65 bad fuzzy translations in English .po files (e.g.
"Navigation" was incorrectly translated to "Pagination") and fix
49 fuzzy entries in Spanish .po files with correct translations.
Built-in PagerDuty-equivalent system for on-call scheduling and alert
escalation. Users can now manage schedules, rotation layers, overrides,
and escalation policies directly in the app alongside PagerDuty.
- On-call schedules with rotation layers (daily/weekly/custom), member
management, and temporary overrides
- Escalation policies with ordered rules, timeout-based escalation,
and user/schedule targets
- Automatic escalation via Oban worker with configurable repeat count
- Email notifications via Swoosh for on-call alerts
- Resolver computes who's on-call from layer stacking and overrides
- AlertNotificationWorker integration: starts escalation alongside
PagerDuty, acknowledges/resolves incidents on alert state changes
- Device and organization schemas support escalation_policy_id
- Escalation policy picker on device form
- Schedules nav item with tabbed index (schedules + escalation policies)
- Full CRUD UI for schedules, layers, members, overrides, rules, targets
- 62 LiveView tests, 56 context/schema/resolver/escalation tests
- 26 E2E Playwright tests for smoke and critical path coverage
- Create wireless_client_readings hypertable with compression and retention
- Add WirelessClientReading schema and batch insert function
- Update DevicePollerWorker to save historical readings
- Add signal_badge and snr_badge components to CoreComponents
- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
After login, users without an organization are redirected to /orgs instead
of /dashboard. The e2e tests expect /dashboard, so create a test organization
for the user during seeding.
Also made the seed script idempotent by deleting existing test user before
creating a new one.
Tested locally and confirmed working.
Function signature is create_totp_device(user_id, device_name, secret) with
separate arguments, not create_totp_device(user, map).
Tested locally and confirmed working.
Accounts.confirm_user_email/1 doesn't exist - the module only has confirm_user/1
which requires a token. For seeding, we can directly set confirmed_at timestamp
using Ecto.Changeset.change/2.
Created seeds_e2e.exs to automatically create a test user with TOTP enabled
for Playwright e2e tests. This ensures tests can authenticate in CI without
requiring manual user setup.
Test credentials (for CI only):
- Email: test@example.com
- Password: TestPassword123!
- TOTP Secret: JBSWY3DPEHPK3PXP (fixed for reproducibility)
The workflow now:
1. Loads database structure
2. Seeds test user with TOTP
3. Passes credentials to e2e tests via environment variables
Replace standard favicon with stoplight design using red (#ef4444),
yellow (#eab308), and green (#22c55e) circles matching the app's
status indicator colors.
The dashboard status indicator favicon (colored circle) uses canvas
to generate a data URL that's set as the favicon. This was blocked
by CSP in staging/production because default-src didn't include data:.
Added data: to default-src directive to fix favicon not updating in
browser tabs when device status changes.
Allow superadmins to override free device limits and per-device pricing
on a per-organization basis. Overrides cascade through subscription
limits, billing calculations, and user-facing settings display.
- Add custom_free_device_limit and custom_price_per_device to organizations
- Add billing_override_changeset with validation
- Update SubscriptionLimits.effective_device_limit to respect overrides
- Update Billing to use effective free count and price per device
- Add Admin.update_billing_overrides with audit logging
- Add override editing UI to /admin/organizations
- Update org settings page to show effective limits/pricing
- Add :admin type to RateLimit plug with 100 requests per minute per IP
- Apply rate limiting to all /admin routes (LiveView and controller actions)
- Protects superuser endpoints from abuse and brute force attempts
- Aligns with existing auth (10 req/min) and API (1000 req/min) limits
The test alias was prompting 'Are you sure you want to proceed?' when
the database structure was already loaded, requiring manual confirmation
on every test run.
Add --skip-if-loaded flag to ecto.load command so it silently skips
structure loading if schema_migrations table already exists.
Now 'mix test' runs without user interaction.
Fixes fragmented translation that displayed raw '%{link}' placeholder text
instead of properly interpolated link in development mail adapter notice.
Changes:
- Consolidate separate translation calls into single interpolated string
- Use raw() helper to safely inject HTML link into translated text
- Update Spanish translation to include %{link} placeholder
- Extract updated translations with mix gettext.extract
Before: "To see sent emails, visit %{link}. the mailbox page."
After: "To see sent emails, visit the mailbox page." (with proper link)
