Oban 2.18+ introduced the 'suspended' job state for pausing jobs.
Oban.Met.Reporter queries for jobs in this state, causing errors when
the database enum doesn't include it:
ERROR 22P02 (invalid_text_representation) invalid input value
for enum oban_job_state: "suspended"
This migration adds 'suspended' to the oban_job_state enum type.
Required by recent Oban dependency updates (commit 0399b30d).
Note: Uses @disable_ddl_transaction since ALTER TYPE ADD VALUE cannot
run inside a transaction in PostgreSQL.
Reviewed-on: graham/towerops-web#199
Add complete memory pool monitoring using HOST-RESOURCES-MIB.
Tracks RAM/swap usage with historical data and real-time updates.
- Add Mempool and MempoolReading schemas
- Integrate discovery and polling with concurrent Task.async_stream
- Add 7 context API functions for querying mempools
- 24 new tests, all 8754 tests passing (0 failures)
Implements C1 from findings_librenms.md
Reviewed-on: graham/towerops-web#184
- Drop index if exists before creating to handle manual creation
- Prevents duplicate_table error when index was created manually
Reviewed-on: graham/towerops-web#179
- Add @disable_ddl_transaction and @disable_migration_lock for concurrent index
- Increase statement_timeout to 10min for DELETE operation on large tables
- Use concurrently: true on index creation to avoid table locks
- Refactor detect_airfiber_counter_overrides to fix credo nesting warning
- Prevents timeout errors in production deployment
Reviewed-on: graham/towerops-web#178
- Clamp negative counter deltas to zero instead of attempting 32-bit wrap
correction (matches LibreNMS behavior — wraps are indistinguishable from
HC counter resets, so discard the ambiguous data point)
- Add rate sanity check: discard values exceeding 400 Gbps as counter anomalies
- Add 4-byte binary decoding for 32-bit SNMP counters in decode_snmp_value
- Track HC vs standard counter usage per interface stat (is_hc field)
- Add migration for is_hc column on snmp_interface_stats
- Update tests to match new clamping behavior
Reviewed-on: graham/towerops-web#168
Missing indexes detected on 7 foreign key columns:
- on_call_overrides.user_id
- on_call_layer_members.user_id
- site_outages.site_id
- escalation_targets.user_id
- notification_digests.organization_id
- on_call_notifications.user_id
- alerts.site_outage_id (already existed, skipped)
Without indexes, PostgreSQL must perform sequential scans for:
- Referential integrity checks on DELETE/UPDATE of parent rows
- JOIN operations on these columns
- Can cause lock contention and slow queries
Indexes created with CONCURRENTLY option to avoid blocking production
tables during deployment.
Migration uses @disable_ddl_transaction and @disable_migration_lock
to ensure indexes are built without holding locks.
Reviewed-on: graham/towerops-web#167
Add data_retention_days field to organizations (default 365, range 30-730).
DataRetentionWorker runs nightly at 1 AM to purge expired time-series data
across 8 tables, respecting each org's retention setting.
TimescaleDB global retention bumped from 90 to 730 days as safety net.
Resolved alerts are cleaned up; unresolved alerts kept regardless of age.
Reviewed-on: graham/towerops-web#144
- Removed device_role_source assignment in sanitize_device_role/1
- Updated structure.sql to reflect database schema without device_role_source column
- Fixes ERROR 42703 (undefined_column) in production workers
The form was still trying to set device_role_source in params even though
the database column was dropped, causing runtime errors in:
- AgentLatencyEvaluator
- CapacityInsightWorker
- WirelessInsightWorker
All device roles are now manually set with default value 'other'.
Reviewed-on: graham/towerops-web#110
- Reduce device type options from 10 to 6 (server, switch, router, access_point, backhaul, other)
- Change label from "Device Role" to "Device Type"
- Disable auto-detection - device type is now required and manual-only
- Add migration to map existing device roles to new simplified set
- Update REST API to include device_role and device_role_source fields
- Automatically set device_role_source to "manual" when device_role is provided
- Update wireless detection to include "backhaul" type
- Enhance display formatting for "Access Point"
Breaking Changes:
- Auto-inference of device type has been disabled
- device_role is now a required field in forms
- GraphQL/REST API consumers will see changed device_role values after migration
Files Changed:
- lib/towerops_web/live/device_live/form.html.heex
- lib/towerops/topology.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/devices/device.ex
- lib/towerops_web/live/device_live/index.ex
- lib/towerops_web/controllers/api/v1/devices_controller.ex
- priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs
- test/towerops/topology_test.exs
- test/towerops_web/live/device_live/form_test.exs
Reviewed-on: graham/towerops-web#109
## Summary
- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)
## Test plan
- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal
Reviewed-on: graham/towerops-web#68
The StormDetector was being added twice:
1. In the main children list (line 127)
2. In background_workers() (line 176)
This caused the supervisor to fail with 'duplicate_child_name' error.
Removed the duplicate from main children list since it's correctly
placed in background_workers() which also handles test env properly.
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- New 'alert_routing' field on organizations (default: builtin)
- AlertNotificationWorker respects the setting:
- builtin: only built-in escalation policies
- pagerduty: only PagerDuty events
- both: fire both (useful during migration)
- Radio button UI in Settings → General with visual cards
- Warning shown when PagerDuty selected but not configured
- Migration, schema validation, 6 tests
Built-in PagerDuty-equivalent system for on-call scheduling and alert
escalation. Users can now manage schedules, rotation layers, overrides,
and escalation policies directly in the app alongside PagerDuty.
- On-call schedules with rotation layers (daily/weekly/custom), member
management, and temporary overrides
- Escalation policies with ordered rules, timeout-based escalation,
and user/schedule targets
- Automatic escalation via Oban worker with configurable repeat count
- Email notifications via Swoosh for on-call alerts
- Resolver computes who's on-call from layer stacking and overrides
- AlertNotificationWorker integration: starts escalation alongside
PagerDuty, acknowledges/resolves incidents on alert state changes
- Device and organization schemas support escalation_policy_id
- Escalation policy picker on device form
- Schedules nav item with tabbed index (schedules + escalation policies)
- Full CRUD UI for schedules, layers, members, overrides, rules, targets
- 62 LiveView tests, 56 context/schema/resolver/escalation tests
- 26 E2E Playwright tests for smoke and critical path coverage
- Create wireless_client_readings hypertable with compression and retention
- Add WirelessClientReading schema and batch insert function
- Update DevicePollerWorker to save historical readings
- Add signal_badge and snr_badge components to CoreComponents
After login, users without an organization are redirected to /orgs instead
of /dashboard. The e2e tests expect /dashboard, so create a test organization
for the user during seeding.
Also made the seed script idempotent by deleting existing test user before
creating a new one.
Tested locally and confirmed working.
Function signature is create_totp_device(user_id, device_name, secret) with
separate arguments, not create_totp_device(user, map).
Tested locally and confirmed working.
Accounts.confirm_user_email/1 doesn't exist - the module only has confirm_user/1
which requires a token. For seeding, we can directly set confirmed_at timestamp
using Ecto.Changeset.change/2.
Created seeds_e2e.exs to automatically create a test user with TOTP enabled
for Playwright e2e tests. This ensures tests can authenticate in CI without
requiring manual user setup.
Test credentials (for CI only):
- Email: test@example.com
- Password: TestPassword123!
- TOTP Secret: JBSWY3DPEHPK3PXP (fixed for reproducibility)
The workflow now:
1. Loads database structure
2. Seeds test user with TOTP
3. Passes credentials to e2e tests via environment variables
Allow superadmins to override free device limits and per-device pricing
on a per-organization basis. Overrides cascade through subscription
limits, billing calculations, and user-facing settings display.
- Add custom_free_device_limit and custom_price_per_device to organizations
- Add billing_override_changeset with validation
- Update SubscriptionLimits.effective_device_limit to respect overrides
- Update Billing to use effective free count and price per device
- Add Admin.update_billing_overrides with audit logging
- Add override editing UI to /admin/organizations
- Update org settings page to show effective limits/pricing
The test alias was prompting 'Are you sure you want to proceed?' when
the database structure was already loaded, requiring manual confirmation
on every test run.
Add --skip-if-loaded flag to ecto.load command so it silently skips
structure loading if schema_migrations table already exists.
Now 'mix test' runs without user interaction.
Replace sequential migration execution (4m8s for 172 migrations) with
structure.sql loading (416ms), reducing CI database setup time by 99.7%.
Changes:
- Configure Ecto to dump/load schema from priv/repo/structure.sql
- Update test alias: ecto.migrate → ecto.load
- Remove redundant database setup step from CI workflow
- Commit structure.sql to repository for version control
Benefits:
- CI database setup: 4m8s → 416ms (248s → 0.4s)
- Consistent schema snapshots across environments
- Faster local test runs
Maintenance:
- Run 'mix ecto.dump' after adding new migrations
- Structure file auto-updates with schema changes
- Add idempotent migration to ensure severity, notification_sent, etc. columns exist
- Fixes production issue where migration was marked as run but columns weren't created
- Uses IF NOT EXISTS to safely add missing columns without errors
- Includes all columns and indexes from ExtendAlertsForChecks migration
Implements Phase 1 of network topology discovery using LLDP (Link Layer
Discovery Protocol) via SNMP, inspired by concepts from lldp2map.
New Features:
- LLDP-MIB walker for discovering network neighbors via SNMP
- Database schema for storing neighbor relationships
- Functions to discover, store, and query device neighbors
- Automatic device linking when neighbors are found in database
Components Added:
- Migration: device_neighbors table with unique constraint on
(device_id, local_port, neighbor_name)
- Schema: Towerops.Topology.DeviceNeighbor for neighbor relationships
- Module: Towerops.Topology.Lldp for SNMP LLDP-MIB walking
- Functions: discover_lldp_neighbors/1, list_lldp_neighbors/1,
remove_stale_lldp_neighbors/1 in Topology context
LLDP-MIB OIDs:
- 1.0.8802.1.1.2.1.3.3.0 (lldpLocSysName)
- 1.0.8802.1.1.2.1.3.7.1.4 (lldpLocPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.7 (lldpRemPortId)
- 1.0.8802.1.1.2.1.4.1.1.8 (lldpRemPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.9 (lldpRemSysName)
- 1.0.8802.1.1.2.1.4.2.1.3 (lldpRemManAddr)
CI Fix:
- Updated config/test.exs to use DATABASE_URL from environment in CI
- Fixes "connection refused to localhost:5432" errors in CI builds
- Falls back to localhost config for local development
Code Quality:
- Refactored LLDP module to reduce cyclomatic complexity
- Extracted helper functions to improve readability
- Used 'with' statement to reduce nesting in parse_management_address
- All Credo checks passing
Next Steps (Phase 2):
- Background worker for automated topology discovery
- Topology visualization in LiveView
- Recursive discovery from seed devices
Changed foreign key constraints from CASCADE to RESTRICT for:
- agent_tokens.organization_id
- agent_assignments.agent_token_id
**Problem:**
Deleting an organization would cascade delete all agent tokens, which
would then cascade delete all agent assignments, causing silent data
loss without any audit trail.
**Solution:**
Use RESTRICT instead of CASCADE - this prevents deletion if there are
dependent records, forcing explicit cleanup and preventing accidental
data loss.
**Impact:**
- Organizations cannot be deleted if they have agent tokens
- Agent tokens cannot be deleted if they have active assignments
- User must explicitly clean up dependencies first
This provides better data integrity and prevents accidental data loss.
Fixes Medium Bug #15 from reliability audit.
Previous version used hardcoded 'towerops_dev' which doesn't exist in production.
Now uses current_database() to get the actual database name at runtime.
This fixes the crash: ERROR 3D000 (invalid_catalog_name) database "towerops_dev" does not exist
Fixes ERROR 53400 (configuration_limit_exceeded) during large interface
sync operations in device discovery.
The sync_interfaces operation was decompressing 146,603 tuples but the
default limit was 100,000. Setting to 0 (unlimited) removes this constraint.
Error occurred in: Towerops.Snmp.Discovery.sync_interfaces/2
Replace the HoneybadgerNoticeFilter that emailed raw stacktraces on
every exception with ErrorTracker for self-hosted error tracking.
Honeybadger itself is retained. ErrorTracker dashboard is mounted at
/admin/errors behind the superuser auth wall.
When the Oban queue backs up, the 60-second uniqueness window expires
and duplicate jobs stack up per device. Switch to period: :infinity so
only one poll/monitor job exists per device at any time. Add replace
option to supersede stale scheduled jobs with updated scheduled_at.
Remove :executing from unique states so self-scheduling works while
the current job runs. Set max_attempts: 1 since retrying stale polls
is pointless.
Also fix all credo --strict issues across the codebase.
New role system:
- owner: full access + financials
- admin: full access + financials (except org deletion)
- executive: read-only + full financial visibility (MRR, revenue)
- technician: field ops (devices, sites, alerts) without financials
- member: legacy alias for technician (migrated)
- viewer: read-only, no financials
Financial data (MRR, revenue, billing) gated behind can_view_financials
assign in all LiveView templates: dashboard, alerts, trace, site show.
Includes migration to rename existing member roles to technician.
Each billing sync (Gaiia, Splynx, VISP, Sonar) already computes
subscriber counts and MRR but only stored them in sync messages.
Now persists them on the integrations table so the dashboard can
aggregate across all billing providers instead of only querying
Gaiia network sites (which were never populated).
Also fixes SnmpKit Config environment detection to use Mix.env()
as fallback when MIX_ENV env var is not set.
device_up alerts are informational recovery events, not ongoing issues.
Previously they were created without resolved_at, inflating the
unresolved alert count. Now they're born resolved.
Includes a data migration to fix existing stale device_up alerts.
