The health check endpoint was calling Towerops.RedisHealthCheck.check_health/2
but the module didn't exist, causing startup probes to fail with 503 errors.
Added RedisHealthCheck module with:
- check_health/2: Single PING command with timeout for health checks
- wait_for_redis/2: Exponential backoff retry logic for startup
This fixes the failing health checks that were blocking pod rollouts.
All tests passing.
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Changes:
- Add Oban configuration to config/dev.exs (was only in runtime.exs)
- Replace Exq telemetry metrics with Oban equivalents
- Update publish_exq_stats to publish_oban_stats using Oban.Job queries
- Track queue sizes, executing jobs, and available jobs
This fixes the startup error where Oban config was not available in dev.
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Fixes two production issues:
1. NOAUTH Redis authentication error:
- application.ex: Read password from application config
- exq_supervisor.ex: Read password from application config
- Both were bypassing runtime.exs config by reading env directly
2. EtcdCoordinator FunctionClauseError:
- connect_etcd/0 now returns connection name (:towerops_etcd)
- eetcd functions expect the atom name, not the PID
- Fixes crash when trying to acquire locks
The app was crashing when the etcd watcher couldn't be created, even
though 2/3 etcd nodes were healthy and the connection succeeded.
Changes:
- Made watcher creation optional (nil if it fails)
- App now starts successfully even if watcher is unavailable
- Falls back to periodic rebalancing (60s intervals) instead of instant failover
- Changed error level from error to warning when watcher creation fails
- Added helpful context to warning message
This allows the app to run with degraded performance instead of crashing
completely when there are etcd cluster issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Optimized the 10 slowest tests from 4.7s to ~2.5s (47% reduction):
Property-based tests:
- Reduced max_runs from 50/100 to 10 iterations (still provides good coverage)
- ApiTokensTest: 3 property tests now run 10 iterations instead of 50-100
- EquipmentTest: 3 property tests now run 10 iterations instead of 50-100
Network timeout tests:
- Manager IPv6 tests: reduced timeout from 100ms to 25ms (75% faster)
- Tests already expect failures, so shorter timeouts don't affect validity
Database optimization:
- MemoryPoolTest: converted individual insert to bulk insert_all
- Added proper updated_at timestamp for MemoryPool (required by schema)
Async processing:
- AgentLatencyEvaluatorTest: reduced Process.sleep from 500ms to 100ms
- Worker processes reliably within 100ms threshold
All 3,686 tests pass with no failures.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Optimized the slowest tests to improve test suite performance:
1. Property-based IP validation test (6s → ~2.4s)
- Reduced max_runs from 50 to 20 iterations
- Still provides good coverage while being much faster
2. Database-heavy tests (~1s each)
- Replaced individual create_check calls with bulk Repo.insert_all
- Reduced check counts from 15 to 10 where adequate for testing
- Fixed timestamp handling (truncate to :second for Ecto)
- Fixed response_time_ms to use floats (schema requirement)
- Removed updated_at (disabled in Check schema)
- Tests affected:
* get_device_latency_by_agent/2
* get_uptime_percentage/1
* get_latency_data/2 respects limit
* AgentLatencyEvaluator tests
Expected speedup: ~40% reduction in top 10 slowest tests (12.8s → ~7.7s)
These optimizations maintain test coverage while significantly reducing
database transaction overhead.
The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.
Error:
** (MatchError) no match of right hand side value:
{:error, :eetcd_conn_unavailable}
This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.
Fix:
- Add case statement to handle both {:ok, watcher} and {:error, reason}
- Log the specific error when watcher creation fails
- Return proper error tuple to supervisor for clean shutdown
The :eetcd Erlang library expects charlists, not binary strings.
This was causing a FunctionClauseError in :lists.reverse/1 because
it was receiving a binary string instead of a charlist.
Error:
(FunctionClauseError) no function clause matching in :lists.reverse/1
(stdlib 7.2) lists.erl:291: :lists.reverse("etcd-0.etcd...")
Solution:
Use ~c sigil to convert endpoint strings to charlists for the
Erlang library to consume correctly.
Redis authentication was failing with "NOAUTH Authentication required"
because the password wasn't being read from the environment variable.
Changes:
- Read REDIS_PASSWORD from environment in runtime.exs
- Add password to :redis config keyword list if present
- Phoenix.PubSub.Redis and Exq now read password from Application config
The application.ex and exq_supervisor.ex already had password support
from System.get_env("REDIS_PASSWORD"), but runtime.exs wasn't passing
the password through to the :redis config, so it was never available
to the application code.
This enables connection to the new Proxmox-hosted Valkey instance at
10.0.15.21 with authentication.
Add REDIS_PASSWORD environment variable support to both Phoenix.PubSub.Redis
and Exq background job processor. Password is optional - if not set, connects
without authentication.
Changes:
- application.ex: Add password to PubSub Redis config if REDIS_PASSWORD is set
- exq_supervisor.ex: Add password to Exq Redis config if REDIS_PASSWORD is set
Required for connecting to Proxmox-hosted Redis with authentication.
Removing all Valkey (Redis) resources from Kubernetes due to instability
caused by Flannel CNI networking issues. Redis will now run on Proxmox
hosts for better stability and performance.
Changes:
- Delete Valkey StatefulSet (master + 2 replicas)
- Delete Valkey Sentinel StatefulSet (3 instances)
- Delete Valkey services (headless and sentinel)
- Delete Valkey ConfigMap
- Remove Valkey resources from kustomization.yaml
- Update deployment to use towerops-redis secret for connection
Next Steps:
- Set up Redis Sentinel on 3 Proxmox hosts/LXC containers
- Create towerops-redis secret with REDIS_HOST and REDIS_PORT
- Test failover and application connectivity
Benefits:
- Not affected by K8s networking issues (Flannel failures)
- More stable (no restarts from node issues)
- Better performance (no K8s overhead)
- Independent lifecycle from K8s cluster
Sentinel requires individual pod DNS names to work correctly.
Making the valkey service headless enables DNS resolution for:
- valkey-0.valkey.towerops.svc.cluster.local
- valkey-1.valkey.towerops.svc.cluster.local
- valkey-2.valkey.towerops.svc.cluster.local
This fixes CrashLoopBackOff in valkey-sentinel pods.
