## Summary
- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever
## Changes
- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience
## Test plan
- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)
Reviewed-on: graham/towerops-web#71
## Summary
- Every device now automatically gets an ICMP ping check (`source_type: "auto_discovery"`) when created via `Monitoring.ensure_default_ping_check/1`
- When a device IP changes, the ping check host config is synced automatically
- Backfill migration inserts ICMP Ping checks for all existing devices that don't have one
- Fixes checkbox alignment in the Add Service Check modal (verify SSL / follow redirects)
## Test plan
- [x] 4 new tests for `ensure_default_ping_check/1` (create, idempotent, IP update, no-op)
- [x] 3 new tests for device lifecycle (auto-create on device create, sync on IP change, no-op when IP unchanged)
- [x] 13 existing tests updated to account for auto-created ping check
- [x] Full test suite passes (`mix precommit`)
- [ ] Verify backfill migration on staging
- [ ] Create device in UI, confirm ping check appears in service checks list
- [ ] Verify checkbox alignment in Add Service Check modal
Reviewed-on: graham/towerops-web#68
add ssl check executor that connects via TLS, reads peer certificate
expiry, and returns OK/WARNING/CRITICAL based on configurable
warning_days threshold. includes protobuf config, liveview form fields,
agent channel encoding, oban worker dispatch, and unit tests.
Reviewed-on: graham/towerops-web#64
- REST CRUD at /api/v1/checks for HTTP, TCP, DNS, ping checks
- GraphQL queries (checks, check) and mutations (create/update/delete)
- ping executor using system ping with macOS/Linux parsing
- check config validation for ping type (requires host)
- API documentation updated with checks resource section
Reviewed-on: graham/towerops-web#52
The StormDetector was being added twice:
1. In the main children list (line 127)
2. In background_workers() (line 176)
This caused the supervisor to fail with 'duplicate_child_name' error.
Removed the duplicate from main children list since it's correctly
placed in background_workers() which also handles test env properly.
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- Increase node font from 11px to 13px, compound labels to 14px
- Replace cose physics layout with deterministic grid preset layout to eliminate site group overlap
- Fix nil source_interface_id crash in topology.ex find_existing_link/1 (was using == nil instead of is_nil/1)
- Fix nil device label in device_to_node/1 fallback to IP or "Unknown"
- New 'alert_routing' field on organizations (default: builtin)
- AlertNotificationWorker respects the setting:
- builtin: only built-in escalation policies
- pagerduty: only PagerDuty events
- both: fire both (useful during migration)
- Radio button UI in Settings → General with visual cards
- Warning shown when PagerDuty selected but not configured
- Migration, schema validation, 6 tests
- Hide less-critical table columns on small screens (IP, subs, MRR, etc.)
- Add overflow-x-auto wrappers on tables to prevent horizontal scroll
- Fix <.header> component to flex-wrap action buttons on mobile
- Make page action button labels icon-only on small screens
- Add min-w-0 on grid children to prevent viewport overflow
- Right-align device detail dt/dd values consistently
- Fix schedule timeline cards with overflow-x-auto wrappers
Bump desktop nav breakpoint from md (768px) to lg (1024px) so the
mobile hamburger menu is used on medium-width screens instead of
nav items overflowing outside the viewport.
Clear all 65 bad fuzzy translations in English .po files (e.g.
"Navigation" was incorrectly translated to "Pagination") and fix
49 fuzzy entries in Spanish .po files with correct translations.
Built-in PagerDuty-equivalent system for on-call scheduling and alert
escalation. Users can now manage schedules, rotation layers, overrides,
and escalation policies directly in the app alongside PagerDuty.
- On-call schedules with rotation layers (daily/weekly/custom), member
management, and temporary overrides
- Escalation policies with ordered rules, timeout-based escalation,
and user/schedule targets
- Automatic escalation via Oban worker with configurable repeat count
- Email notifications via Swoosh for on-call alerts
- Resolver computes who's on-call from layer stacking and overrides
- AlertNotificationWorker integration: starts escalation alongside
PagerDuty, acknowledges/resolves incidents on alert state changes
- Device and organization schemas support escalation_policy_id
- Escalation policy picker on device form
- Schedules nav item with tabbed index (schedules + escalation policies)
- Full CRUD UI for schedules, layers, members, overrides, rules, targets
- 62 LiveView tests, 56 context/schema/resolver/escalation tests
- 26 E2E Playwright tests for smoke and critical path coverage
- Create wireless_client_readings hypertable with compression and retention
- Add WirelessClientReading schema and batch insert function
- Update DevicePollerWorker to save historical readings
- Add signal_badge and snr_badge components to CoreComponents
- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
After login, users without an organization are redirected to /orgs instead
of /dashboard. The e2e tests expect /dashboard, so create a test organization
for the user during seeding.
Also made the seed script idempotent by deleting existing test user before
creating a new one.
Tested locally and confirmed working.
Function signature is create_totp_device(user_id, device_name, secret) with
separate arguments, not create_totp_device(user, map).
Tested locally and confirmed working.
Accounts.confirm_user_email/1 doesn't exist - the module only has confirm_user/1
which requires a token. For seeding, we can directly set confirmed_at timestamp
using Ecto.Changeset.change/2.
Created seeds_e2e.exs to automatically create a test user with TOTP enabled
for Playwright e2e tests. This ensures tests can authenticate in CI without
requiring manual user setup.
Test credentials (for CI only):
- Email: test@example.com
- Password: TestPassword123!
- TOTP Secret: JBSWY3DPEHPK3PXP (fixed for reproducibility)
The workflow now:
1. Loads database structure
2. Seeds test user with TOTP
3. Passes credentials to e2e tests via environment variables
Replace standard favicon with stoplight design using red (#ef4444),
yellow (#eab308), and green (#22c55e) circles matching the app's
status indicator colors.
The dashboard status indicator favicon (colored circle) uses canvas
to generate a data URL that's set as the favicon. This was blocked
by CSP in staging/production because default-src didn't include data:.
Added data: to default-src directive to fix favicon not updating in
browser tabs when device status changes.
Allow superadmins to override free device limits and per-device pricing
on a per-organization basis. Overrides cascade through subscription
limits, billing calculations, and user-facing settings display.
- Add custom_free_device_limit and custom_price_per_device to organizations
- Add billing_override_changeset with validation
- Update SubscriptionLimits.effective_device_limit to respect overrides
- Update Billing to use effective free count and price per device
- Add Admin.update_billing_overrides with audit logging
- Add override editing UI to /admin/organizations
- Update org settings page to show effective limits/pricing
- Add :admin type to RateLimit plug with 100 requests per minute per IP
- Apply rate limiting to all /admin routes (LiveView and controller actions)
- Protects superuser endpoints from abuse and brute force attempts
- Aligns with existing auth (10 req/min) and API (1000 req/min) limits
The test alias was prompting 'Are you sure you want to proceed?' when
the database structure was already loaded, requiring manual confirmation
on every test run.
Add --skip-if-loaded flag to ecto.load command so it silently skips
structure loading if schema_migrations table already exists.
Now 'mix test' runs without user interaction.
Fixes fragmented translation that displayed raw '%{link}' placeholder text
instead of properly interpolated link in development mail adapter notice.
Changes:
- Consolidate separate translation calls into single interpolated string
- Use raw() helper to safely inject HTML link into translated text
- Update Spanish translation to include %{link} placeholder
- Extract updated translations with mix gettext.extract
Before: "To see sent emails, visit %{link}. the mailbox page."
After: "To see sent emails, visit the mailbox page." (with proper link)
Replace sequential migration execution (4m8s for 172 migrations) with
structure.sql loading (416ms), reducing CI database setup time by 99.7%.
Changes:
- Configure Ecto to dump/load schema from priv/repo/structure.sql
- Update test alias: ecto.migrate → ecto.load
- Remove redundant database setup step from CI workflow
- Commit structure.sql to repository for version control
Benefits:
- CI database setup: 4m8s → 416ms (248s → 0.4s)
- Consistent schema snapshots across environments
- Faster local test runs
Maintenance:
- Run 'mix ecto.dump' after adding new migrations
- Structure file auto-updates with schema changes
- Add idempotent migration to ensure severity, notification_sent, etc. columns exist
- Fixes production issue where migration was marked as run but columns weren't created
- Uses IF NOT EXISTS to safely add missing columns without errors
- Includes all columns and indexes from ExtendAlertsForChecks migration
Implements Phase 1 of network topology discovery using LLDP (Link Layer
Discovery Protocol) via SNMP, inspired by concepts from lldp2map.
New Features:
- LLDP-MIB walker for discovering network neighbors via SNMP
- Database schema for storing neighbor relationships
- Functions to discover, store, and query device neighbors
- Automatic device linking when neighbors are found in database
Components Added:
- Migration: device_neighbors table with unique constraint on
(device_id, local_port, neighbor_name)
- Schema: Towerops.Topology.DeviceNeighbor for neighbor relationships
- Module: Towerops.Topology.Lldp for SNMP LLDP-MIB walking
- Functions: discover_lldp_neighbors/1, list_lldp_neighbors/1,
remove_stale_lldp_neighbors/1 in Topology context
LLDP-MIB OIDs:
- 1.0.8802.1.1.2.1.3.3.0 (lldpLocSysName)
- 1.0.8802.1.1.2.1.3.7.1.4 (lldpLocPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.7 (lldpRemPortId)
- 1.0.8802.1.1.2.1.4.1.1.8 (lldpRemPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.9 (lldpRemSysName)
- 1.0.8802.1.1.2.1.4.2.1.3 (lldpRemManAddr)
CI Fix:
- Updated config/test.exs to use DATABASE_URL from environment in CI
- Fixes "connection refused to localhost:5432" errors in CI builds
- Falls back to localhost config for local development
Code Quality:
- Refactored LLDP module to reduce cyclomatic complexity
- Extracted helper functions to improve readability
- Used 'with' statement to reduce nesting in parse_management_address
- All Credo checks passing
Next Steps (Phase 2):
- Background worker for automated topology discovery
- Topology visualization in LiveView
- Recursive discovery from seed devices
- Added entries for organization_id alert optimization
- Added entries for cascade delete safety improvements
- Added entry for sensor state validation
- Updated user-facing changelog with performance improvements
- All 23 bugs from reliability audit now addressed
Changed foreign key constraints from CASCADE to RESTRICT for:
- agent_tokens.organization_id
- agent_assignments.agent_token_id
**Problem:**
Deleting an organization would cascade delete all agent tokens, which
would then cascade delete all agent assignments, causing silent data
loss without any audit trail.
**Solution:**
Use RESTRICT instead of CASCADE - this prevents deletion if there are
dependent records, forcing explicit cleanup and preventing accidental
data loss.
**Impact:**
- Organizations cannot be deleted if they have agent tokens
- Agent tokens cannot be deleted if they have active assignments
- User must explicitly clean up dependencies first
This provides better data integrity and prevents accidental data loss.
Fixes Medium Bug #15 from reliability audit.
