- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
Replace the HoneybadgerNoticeFilter that emailed raw stacktraces on
every exception with ErrorTracker for self-hosted error tracking.
Honeybadger itself is retained. ErrorTracker dashboard is mounted at
/admin/errors behind the superuser auth wall.
- Escape HEEx template braces in GraphQL/API docs with raw(~S[...])
- Fix test assertions for updated marketing copy and UI text
- Extract helper functions in GraphQL resolvers to reduce nesting depth
- Create shared ErrorHelpers module for API controllers
- Fix ETS race condition in brute force whitelist cache for async tests
- Fix property test generators to use ASCII instead of printable unicode
- Add alert_severity helper to site_live/show
- Update accounts fixtures for explicit user confirmation
- Add 'Checks' tab link to device navigation
- Display empty state when no checks configured
- Add tests for tab visibility and empty state
- Fix unused variable warnings in monitoring context
- Remove unused CheckResult alias and module attributes
- Add stub for legacy get_latency_data for backward compatibility
Tests: 2 new tests added, all passing
Following TDD: RED-GREEN cycle verified for each feature
Changes:
- Fix traffic graph to handle nil interface octets gracefully
- Add comprehensive tests for nil octets scenarios
- Fix agent polling query to support devices assigned directly to org (left join on sites)
- Fix test SNMP connection to inherit credentials from site/org
- Refactor agent_channel process_polling_result to reduce nesting (Credo)
- Add alias for JobCleanupTask in application.ex (Credo)
This fixes crashes when interface stats have nil octets and ensures
devices without a site assignment can be polled by agents.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
User deletion bug:
- Store client IP in socket assigns during mount
- Access stored IP during delete event instead of connect_info
- Fixes RuntimeError when attempting to delete users
Timezone support:
- Add timezone offset maps for common timezones (no external deps)
- Implement shift_timezone/2 helper using DateTime.add/3
- Add timezone-aware format_datetime/2, format_date/2, format_iso8601/2
- Keep backwards-compatible 1-arity versions defaulting to UTC
- Add comprehensive test coverage for timezone conversions
- All tests passing (23 tests, 0 failures)
Note: Timezone offsets are standard time only (no DST handling)
Common timezones supported: America/New_York, America/Los_Angeles,
Europe/London, Europe/Paris, Asia/Tokyo, Asia/Shanghai, Australia/Sydney
Prevents duplicate polling in multi-node production environment:
- Replace DynamicSupervisor with Horde.DynamicSupervisor (distributed)
- Replace Registry with Horde.Registry (cluster-aware)
- Each device monitor runs on exactly ONE node across the cluster
- Each SNMP poller runs on exactly ONE node across the cluster
- Automatic failover when nodes go down
- Automatic rebalancing when nodes join/leave
Architecture:
- Horde.Registry for process naming (distributed lookup)
- Horde.DynamicSupervisor for DeviceMonitor processes
- Horde.DynamicSupervisor for PollerWorker processes
- Task.Supervisor remains local per-node for parallel operations
- members: :auto enables automatic cluster discovery via libcluster
Production impact:
- 2 replicas = devices split ~50/50 across both pods
- No duplicate SNMP polls to same device
- If pod dies, surviving pod takes over all devices
- When new pod starts, devices rebalance automatically
Dependencies added:
- horde ~> 0.9.0 (distributed supervisor/registry)
- delta_crdt (Horde dependency for CRDT-based state)
- libring (Horde dependency for consistent hashing)
- Configure Phoenix.PubSub to use Redis adapter for pod-resilient messaging
- Add PubSub broadcasts for sensor readings updates in PollerWorker
- Add PubSub broadcasts for interface stats updates in PollerWorker
- Add PubSub broadcasts for neighbor discovery updates in PollerWorker
- Add PubSub broadcasts for monitoring check updates in MonitorWorker
- Broadcast interface and sensor events to device-specific topics
- Add event handlers in DeviceLive.Show for all update types
- Device show page now updates in real-time without polling
- Updates survive pod restarts and work across multiple pods
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Add MIB files from LibreNMS in priv/mibs/ for reference
- Create MibParser module to validate OIDs against official MIB definitions
- Add MIB validation tests to ensure hardcoded OIDs match MIB specs
- Refactor SNMP tests to be generic/behavior-focused instead of vendor-specific
- Remove vendor-specific test files (cisco_test, net_snmp_test)
- All 104 tests passing with automated OID validation
Security improvements to prevent potential issues:
1. Port Validation (normalize_port):
- Validate port range is 1-65535
- Use Integer.parse instead of String.to_integer to prevent crashes
- Return default port 161 for any invalid input
2. SNMP Test Validation (validate_test_snmp_input):
- Validate IP address format before allowing SNMP tests
- Require non-empty community string
- Prevent network scanning with invalid/missing IPs
- Return clear error messages for validation failures
These changes ensure user input is properly validated and prevent:
- Integer overflow/underflow with ports
- Process crashes from invalid input
- Unauthorized network scanning via SNMP test feature
- SNMP requests with missing credentials
