From a6a2c859b421c593c5f741b7177d36ec4300b71d Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Tue, 5 May 2026 17:19:39 -0500 Subject: [PATCH] fix(infra): disable Oban peer on backfill pods so cron leader stays hot All app=prop pods (hot + backfill) join the libcluster cluster and are eligible to win Oban leader election. The backfill pod's Cron plugin is configured with crontab: [], so when it wins, every cron silently stops firing until the leader peer entry expires and rotates. Today's deploy showed the failure mode: backfill won the post-restart election, all crons (PSKR calibration, propagation grid, partition maintenance, etc.) were silent for 75 minutes until I bounced the backfill pod manually. Setting `peer: false` on the backfill role makes it ineligible for leadership entirely. Hot pods compete only with each other. --- config/runtime.exs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/config/runtime.exs b/config/runtime.exs index e8f971c2..05f5210e 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -464,6 +464,17 @@ if config_env() == :prod do queues: oban_queues config :microwaveprop, Oban, plugins: oban_plugins + + # Oban's leader peer runs all leader-only plugins (Cron, Pruner's + # rescue, etc.) on whichever node wins election. All `app=prop` pods + # share the libcluster selector, so without intervention a backfill + # pod can win — and its empty crontab means *no* crons fire until + # the next leader rotation. Disabling peer participation on backfill + # pods makes only hot pods eligible, which is what we actually want. + if prop_role == "backfill" do + config :microwaveprop, Oban, peer: false + end + config :microwaveprop, :email_from, {"NTMS Propagation", "prop@w5isp.com"} # Disable the PromEx Oban queue-depth poller in prod. Its 5 s