fix(infra): disable Oban peer on backfill pods so cron leader stays hot

All app=prop pods (hot + backfill) join the libcluster cluster and are
eligible to win Oban leader election. The backfill pod's Cron plugin is
configured with crontab: [], so when it wins, every cron silently stops
firing until the leader peer entry expires and rotates. Today's deploy
showed the failure mode: backfill won the post-restart election, all
crons (PSKR calibration, propagation grid, partition maintenance, etc.)
were silent for 75 minutes until I bounced the backfill pod manually.

Setting `peer: false` on the backfill role makes it ineligible for
leadership entirely. Hot pods compete only with each other.
This commit is contained in:
Graham McIntire 2026-05-05 17:19:39 -05:00
parent a9366dffeb
commit a6a2c859b4
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -464,6 +464,17 @@ if config_env() == :prod do
queues: oban_queues
config :microwaveprop, Oban, plugins: oban_plugins
# Oban's leader peer runs all leader-only plugins (Cron, Pruner's
# rescue, etc.) on whichever node wins election. All `app=prop` pods
# share the libcluster selector, so without intervention a backfill
# pod can win — and its empty crontab means *no* crons fire until
# the next leader rotation. Disabling peer participation on backfill
# pods makes only hot pods eligible, which is what we actually want.
if prop_role == "backfill" do
config :microwaveprop, Oban, peer: false
end
config :microwaveprop, :email_from, {"NTMS Propagation", "prop@w5isp.com"}
# Disable the PromEx Oban queue-depth poller in prod. Its 5 s