- Add Towerops.TaskSupervisor to supervision tree, replace bare spawn/1
in agent channel with supervised tasks for polling data processing
- Change DiscoveryWorker from max_attempts: 1 to 3 with retryable error
classification (transient errors retry, permanent errors discard)
- Name all 9 parallel polling tasks and log timeouts/crashes instead of
silently dropping results via Stream.run()
- Add tests for TaskSupervisor presence and error classification
The previous commit (90e8964) introduced an incorrect LoggerBackends.add/2
call that prevented Oban from starting, causing all discovery jobs to remain
stuck in "available" state without being processed.
Root cause: LoggerBackends.add expects a tuple {module, id} as the first
argument, not the module alone with configuration options.
Fix: Use the correct API pattern:
1. Add backend with tuple identifier: {LoggerFileBackend, :file_log}
2. Configure separately with Logger.configure_backend/2
This resolves device discovery being completely broken - all background
jobs (discovery, polling, monitoring) now process correctly.
Changes:
- Fix traffic graph to handle nil interface octets gracefully
- Add comprehensive tests for nil octets scenarios
- Fix agent polling query to support devices assigned directly to org (left join on sites)
- Fix test SNMP connection to inherit credentials from site/org
- Refactor agent_channel process_polling_result to reduce nesting (Credo)
- Add alias for JobCleanupTask in application.ex (Credo)
This fixes crashes when interface stats have nil octets and ensures
devices without a site assignment can be polled by agents.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added JobCleanupTask that runs on production startup to cancel all
existing polling jobs and reschedule them with the latest worker code.
This ensures:
- No stale jobs run with old code after deployment (e.g., missing SNMPv3 credentials)
- All devices get fresh polling jobs with current logic
- Production deployments are seamless without manual intervention
The task:
- Only runs in production environment (skipped in dev/test)
- Runs asynchronously after supervisor initialization
- Cancels all DevicePollerWorker and DeviceMonitorWorker jobs
- Reschedules polling for all SNMP-enabled devices
- Is idempotent and safe to run multiple times
Removed debug logging from Client and DevicePollerWorker as SNMPv3
implementation is now verified and working correctly.
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Add REDIS_PASSWORD environment variable support to both Phoenix.PubSub.Redis
and Exq background job processor. Password is optional - if not set, connects
without authentication.
Changes:
- application.ex: Add password to PubSub Redis config if REDIS_PASSWORD is set
- exq_supervisor.ex: Add password to Exq Redis config if REDIS_PASSWORD is set
Required for connecting to Proxmox-hosted Redis with authentication.
Fixes crashes in Exq.Node.Server when Redis connections fail by adding
health checks before startup and wrapping Exq in a custom supervisor
with better resilience.
Problem:
- Valkey pod restarting frequently (24 times in 28h) due to K8s Flannel CNI issues
- When Redis disconnects, Exq.Node.Server crashes trying to process_signals/2
- Gets nil from Redis instead of expected list, crashes on Enum.each/2
- Rapid crash/restart cycles reduce application stability
Changes:
1. RedisHealthCheck module (lib/towerops/redis_health_check.ex)
- Waits for Redis availability with exponential backoff before starting Exq
- Prevents Exq from starting when Redis is unavailable
- Handles connection errors gracefully without crashing caller processes
- Supports configurable retry attempts, timeouts, and backoff intervals
2. ExqSupervisor module (lib/towerops/exq_supervisor.ex)
- Custom supervisor that wraps Exq with health checks
- Uses one_for_one strategy with limited restarts (3 per 60s)
- Prevents rapid crash loops when Redis is unstable
- Increased Exq retry/backoff settings for better Redis recovery
- Allows application to continue functioning without background jobs if Redis unavailable
3. Updated Application.ex
- Replaced direct Exq start with ExqSupervisor
- Removed inline exq_config/0 function (moved to supervisor)
4. Tests
- Comprehensive tests for RedisHealthCheck (health checks, retries, timeouts)
- Tests for ExqSupervisor (configuration, restart strategy)
- All 1006 tests passing
Benefits:
- Prevents Exq crashes from propagating when Redis fails
- Application continues running even when Redis is temporarily unavailable
- Better logging of Redis connection issues
- More graceful degradation during infrastructure problems
Infrastructure Issue (to be addressed separately):
- K8s Flannel CNI plugin failing: /run/flannel/subnet.env not found
- Causing Valkey pod network issues and restarts
- Needs investigation of Flannel daemonset and node configuration
🤖 Generated with Claude Code
- Use SNMP poll success as health indicator for SNMP-enabled devices
- Fall back to ICMP ping only for non-SNMP devices or stale SNMP data
- Add connection retry logic to Exq with backoff
- Add better error logging for ping permission issues
- Resolves issue where devices showed as down despite successful SNMP polling
Root cause: Container runs as non-root (UID 65534) without ICMP permissions,
so ping fails even though SNMP (UDP) works. Now using SNMP poll timestamps
to determine device status when SNMP is enabled.
- Configure Phoenix.PubSub to use Redis adapter for pod-resilient messaging
- Add PubSub broadcasts for sensor readings updates in PollerWorker
- Add PubSub broadcasts for interface stats updates in PollerWorker
- Add PubSub broadcasts for neighbor discovery updates in PollerWorker
- Add PubSub broadcasts for monitoring check updates in MonitorWorker
- Broadcast interface and sensor events to device-specific topics
- Add event handlers in DeviceLive.Show for all update types
- Device show page now updates in real-time without polling
- Updates survive pod restarts and work across multiple pods
Created three new Exq workers to handle background jobs:
- PollWorker: SNMP polling operations (queue: polling)
- MonitorWorker: Device monitoring checks (queue: monitoring)
- DiscoveryWorker: SNMP discovery (already existed)
Updated files to use Exq.enqueue instead of Task.start:
- lib/towerops/snmp/poller_worker.ex
- lib/towerops/monitoring/device_monitor.ex
- lib/towerops_web/channels/agent_channel.ex
Each module includes enqueue_* helper functions that:
- Use Task.start in test environment for synchronous execution
- Use Exq.enqueue in dev/prod for proper background job processing
Added "monitoring" queue to Exq configuration in application.ex
All tests passing (824 tests, 0 failures)
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Update GitLab CI deploy job to set DEPLOY_TIMESTAMP when deploying
- Use 'kubectl set env' to update all pods with same timestamp
- Change deployment.yaml to use static placeholder value
- Update documentation to explain GitLab CI timestamp approach
This ensures all pods show the same deployment time, regardless of:
- Which pod handles the request (with replicas=2)
- When individual pods were created or restarted
- Pod restarts from crashes or rolling updates
The timestamp now represents when the deployment was initiated by GitLab CI,
not when individual pods were created.
- Read DEPLOY_TIMESTAMP environment variable for actual deploy time
- Fall back to compile time in development when env var not set
- Add Kubernetes Downward API to inject metadata.creationTimestamp
- Update k8s/deployment.yaml to set DEPLOY_TIMESTAMP env var
- Document deployment timestamp mechanism in k8s/README.md
This ensures the footer shows when the pod was deployed to the cluster,
not when the Docker image was built in CI/CD.
- Add compile-time @build_timestamp to Application module
- Create ToweropsWeb.TimeHelpers module for shared time formatting
- Update footer to display 'Last deployed Xm ago · YYYY-MM-DD HH:MM:SS UTC'
- Refactor AgentLive.Helpers to delegate to shared TimeHelpers
- Consolidate duplicate time formatting logic into single module
- Timestamp updates automatically on each build/deploy
Create HealthController that returns 200 OK for health checks.
Route is outside browser pipeline to avoid SSL redirect and
authentication requirements.
- Migrations now run in Application.start/2 before starting services
- Ecto's advisory locks prevent concurrent migrations in clustered setup
- Simplified deployment - no separate migration job needed
- Removed migrate-job.yaml manifest
Monitoring Infrastructure:
- lib/towerops/monitoring/ping.ex - OS-agnostic ping functionality that works on macOS, Linux, and Windows
- lib/towerops/monitoring/equipment_monitor.ex - GenServer that monitors individual equipment at configurable intervals
- lib/towerops/monitoring/supervisor.ex - DynamicSupervisor with Registry for managing monitor workers
- lib/towerops/monitoring.ex - Context for creating and querying monitoring checks
Database:
- monitoring_checks table stores historical ping results with status, response time, and timestamps
- Indexed on (equipment_id, checked_at) for efficient querying
Real-Time Updates:
- PubSub broadcasting on equipment:#{id} topic when status changes
- Equipment show page subscribes to updates and refreshes automatically
- Manual "Check Now" button to trigger immediate checks
Application Integration:
- Monitoring.Supervisor added to application supervision tree
- All monitored equipment starts monitoring automatically on app startup
- Recent checks displayed in Equipment show page
How It Works
1. On application start, a monitor GenServer is spawned for each equipment with monitoring_enabled: true
2. Each monitor pings its equipment at the configured interval (default 5 minutes)
3. Results are saved to monitoring_checks table
4. Equipment status is updated if it changes (up ↔ down)
5. Status changes are broadcast via PubSub to all connected LiveViews
6. The Equipment show page updates in real-time when status changes
