fix: handle etcd watcher creation errors gracefully
The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.
Error:
** (MatchError) no match of right hand side value:
{:error, :eetcd_conn_unavailable}
This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.
Fix:
- Add case statement to handle both {:ok, watcher} and {:error, reason}
- Log the specific error when watcher creation fails
- Return proper error tuple to supervisor for clean shutdown
This commit is contained in:
parent
51736eeec3
commit
64ad63d3e3
1 changed files with 14 additions and 7 deletions
|
|
@ -38,16 +38,23 @@ defmodule Towerops.Monitoring.EtcdCoordinator do
|
|||
watch_ctx = :eetcd_watch.new()
|
||||
watch_with_key = :eetcd_watch.with_key(watch_ctx, "towerops/locks/")
|
||||
watch_with_prefix = :eetcd_watch.with_prefix(watch_with_key)
|
||||
{:ok, watcher} = :eetcd_watch.watch(conn, watch_with_prefix)
|
||||
Logger.info("Watching etcd for lock releases")
|
||||
|
||||
# Start acquiring locks for all devices
|
||||
send(self(), :acquire_locks)
|
||||
case :eetcd_watch.watch(conn, watch_with_prefix) do
|
||||
{:ok, watcher} ->
|
||||
Logger.info("Watching etcd for lock releases")
|
||||
|
||||
# Schedule periodic rebalancing (safety net for missed watch events)
|
||||
schedule_rebalance()
|
||||
# Start acquiring locks for all devices
|
||||
send(self(), :acquire_locks)
|
||||
|
||||
{:ok, %{conn: conn, watcher: watcher, owned_leases: %{}}}
|
||||
# Schedule periodic rebalancing (safety net for missed watch events)
|
||||
schedule_rebalance()
|
||||
|
||||
{:ok, %{conn: conn, watcher: watcher, owned_leases: %{}}}
|
||||
|
||||
{:error, reason} ->
|
||||
Logger.error("Failed to create etcd watcher: #{inspect(reason)}")
|
||||
{:stop, {:etcd_watcher_failed, reason}}
|
||||
end
|
||||
|
||||
{:error, reason} ->
|
||||
Logger.error("Failed to connect to etcd: #{inspect(reason)}")
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue