When Kubernetes performs rolling deployments, the old pod gracefully shuts down
and Oban's internal processes receive EXIT :shutdown signals. These are logged
as "unexpected" even though they're normal during shutdown.
Added custom logger filter to drop these messages in production:
- Oban.Queue.Watchman/Producer receiving EXIT :shutdown
- Oban plugin processes receiving EXIT :shutdown
- Reduces log noise during deployments without hiding real errors
The filter is production-only to preserve full debugging in development.
HTML checkboxes send value='on' when checked, not 'true'. Updated
parse_agent_params to accept both 'on' and 'true' values for the
is_cloud_poller field.
Added debug logging to help diagnose future form submission issues.
Redix expects host to be a string, not a charlist. The to_charlist/1
conversion was causing FunctionClauseError in Redix.StartOptions.__validate_host__/1.
This matches how Phoenix PubSub Redis and Exq are configured, which both
use the string host directly without conversion.
Added logging to identify which component is failing health checks:
- Log database and Redis status on each health check
- Log Redis errors with full error details
- This will help diagnose the production 503 errors
Fixed dialyzer warnings across multiple files by properly handling return
values from functions that are called for side effects.
Changes:
- lib/towerops/agents.ex: Explicitly match broadcast and if-expression returns
- lib/towerops/devices.ex: Match worker start function returns in if-expressions
- lib/towerops/workers/agent_latency_evaluator.ex: Match broadcast return
- lib/towerops/workers/device_monitor_worker.ex: Match perform_check and handle_status_change returns, handle schedule_next_check errors
- lib/towerops/workers/device_poller_worker.ex: Match all PubSub broadcast returns, handle schedule_next_poll errors, remove unreachable pattern
- lib/towerops/workers/stale_agent_worker.ex: Match process_stale_agents and broadcast returns
- lib/towerops_web/channels/agent_channel.ex: Match PubSub.subscribe return, improve get_addr pattern matching, remove unreachable format_ip clause
- lib/towerops_web/live/device_live/form.ex: Match enqueue_discovery return, replace compile-time @dev_env with runtime check
Pattern applied:
- PubSub broadcasts: `_ = Phoenix.PubSub.broadcast(...)`
- If-expressions with side effects: `_ = if condition do ... end`
- Critical operations: Added error logging with case statements
All 3625 tests passing. No dialyzer warnings remaining in lib/towerops or lib/towerops_web.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed discover_all/1 to enqueue individual Oban jobs instead of using
Task.async_stream for better error visibility and resilience.
Benefits:
- Errors are visible in Oban dashboard with full stack traces
- Automatic retries on failure via Oban
- Jobs persist across pod restarts
- No more silent failures from Task crashes
- Better concurrency control via Oban queue configuration
Changes:
- discover_all/1 now enqueues DiscoveryWorker jobs and returns immediately
- Return type changed from %{success, failed, errors} to %{enqueued, failed, errors}
- Updated tests to match new async behavior
- Added DiscoveryWorker alias to Discovery module
All tests passing (3,625 tests, 0 failures).
The health check endpoint was calling Towerops.RedisHealthCheck.check_health/2
but the module didn't exist, causing startup probes to fail with 503 errors.
Added RedisHealthCheck module with:
- check_health/2: Single PING command with timeout for health checks
- wait_for_redis/2: Exponential backoff retry logic for startup
This fixes the failing health checks that were blocking pod rollouts.
All tests passing.
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Changes:
- Add Oban configuration to config/dev.exs (was only in runtime.exs)
- Replace Exq telemetry metrics with Oban equivalents
- Update publish_exq_stats to publish_oban_stats using Oban.Job queries
- Track queue sizes, executing jobs, and available jobs
This fixes the startup error where Oban config was not available in dev.
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Fixes two production issues:
1. NOAUTH Redis authentication error:
- application.ex: Read password from application config
- exq_supervisor.ex: Read password from application config
- Both were bypassing runtime.exs config by reading env directly
2. EtcdCoordinator FunctionClauseError:
- connect_etcd/0 now returns connection name (:towerops_etcd)
- eetcd functions expect the atom name, not the PID
- Fixes crash when trying to acquire locks
The app was crashing when the etcd watcher couldn't be created, even
though 2/3 etcd nodes were healthy and the connection succeeded.
Changes:
- Made watcher creation optional (nil if it fails)
- App now starts successfully even if watcher is unavailable
- Falls back to periodic rebalancing (60s intervals) instead of instant failover
- Changed error level from error to warning when watcher creation fails
- Added helpful context to warning message
This allows the app to run with degraded performance instead of crashing
completely when there are etcd cluster issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.
Error:
** (MatchError) no match of right hand side value:
{:error, :eetcd_conn_unavailable}
This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.
Fix:
- Add case statement to handle both {:ok, watcher} and {:error, reason}
- Log the specific error when watcher creation fails
- Return proper error tuple to supervisor for clean shutdown
The :eetcd Erlang library expects charlists, not binary strings.
This was causing a FunctionClauseError in :lists.reverse/1 because
it was receiving a binary string instead of a charlist.
Error:
(FunctionClauseError) no function clause matching in :lists.reverse/1
(stdlib 7.2) lists.erl:291: :lists.reverse("etcd-0.etcd...")
Solution:
Use ~c sigil to convert endpoint strings to charlists for the
Erlang library to consume correctly.
Add REDIS_PASSWORD environment variable support to both Phoenix.PubSub.Redis
and Exq background job processor. Password is optional - if not set, connects
without authentication.
Changes:
- application.ex: Add password to PubSub Redis config if REDIS_PASSWORD is set
- exq_supervisor.ex: Add password to Exq Redis config if REDIS_PASSWORD is set
Required for connecting to Proxmox-hosted Redis with authentication.
