fix: handle etcd watcher creation errors gracefully

The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.

Error:
  ** (MatchError) no match of right hand side value:
  {:error, :eetcd_conn_unavailable}

This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.

Fix:
  - Add case statement to handle both {:ok, watcher} and {:error, reason}
  - Log the specific error when watcher creation fails
  - Return proper error tuple to supervisor for clean shutdown
This commit is contained in:
Graham McIntire 2026-01-24 15:02:46 -06:00
parent 51736eeec3
commit 64ad63d3e3
No known key found for this signature in database

View file

@ -38,16 +38,23 @@ defmodule Towerops.Monitoring.EtcdCoordinator do
watch_ctx = :eetcd_watch.new()
watch_with_key = :eetcd_watch.with_key(watch_ctx, "towerops/locks/")
watch_with_prefix = :eetcd_watch.with_prefix(watch_with_key)
{:ok, watcher} = :eetcd_watch.watch(conn, watch_with_prefix)
Logger.info("Watching etcd for lock releases")
# Start acquiring locks for all devices
send(self(), :acquire_locks)
case :eetcd_watch.watch(conn, watch_with_prefix) do
{:ok, watcher} ->
Logger.info("Watching etcd for lock releases")
# Schedule periodic rebalancing (safety net for missed watch events)
schedule_rebalance()
# Start acquiring locks for all devices
send(self(), :acquire_locks)
{:ok, %{conn: conn, watcher: watcher, owned_leases: %{}}}
# Schedule periodic rebalancing (safety net for missed watch events)
schedule_rebalance()
{:ok, %{conn: conn, watcher: watcher, owned_leases: %{}}}
{:error, reason} ->
Logger.error("Failed to create etcd watcher: #{inspect(reason)}")
{:stop, {:etcd_watcher_failed, reason}}
end
{:error, reason} ->
Logger.error("Failed to connect to etcd: #{inspect(reason)}")