fix(infra): disable Oban peer on backfill pods so cron leader stays hot
All app=prop pods (hot + backfill) join the libcluster cluster and are eligible to win Oban leader election. The backfill pod's Cron plugin is configured with crontab: [], so when it wins, every cron silently stops firing until the leader peer entry expires and rotates. Today's deploy showed the failure mode: backfill won the post-restart election, all crons (PSKR calibration, propagation grid, partition maintenance, etc.) were silent for 75 minutes until I bounced the backfill pod manually. Setting `peer: false` on the backfill role makes it ineligible for leadership entirely. Hot pods compete only with each other.
This commit is contained in:
parent
a9366dffeb
commit
a6a2c859b4
1 changed files with 11 additions and 0 deletions
|
|
@ -464,6 +464,17 @@ if config_env() == :prod do
|
|||
queues: oban_queues
|
||||
|
||||
config :microwaveprop, Oban, plugins: oban_plugins
|
||||
|
||||
# Oban's leader peer runs all leader-only plugins (Cron, Pruner's
|
||||
# rescue, etc.) on whichever node wins election. All `app=prop` pods
|
||||
# share the libcluster selector, so without intervention a backfill
|
||||
# pod can win — and its empty crontab means *no* crons fire until
|
||||
# the next leader rotation. Disabling peer participation on backfill
|
||||
# pods makes only hot pods eligible, which is what we actually want.
|
||||
if prop_role == "backfill" do
|
||||
config :microwaveprop, Oban, peer: false
|
||||
end
|
||||
|
||||
config :microwaveprop, :email_from, {"NTMS Propagation", "prop@w5isp.com"}
|
||||
|
||||
# Disable the PromEx Oban queue-depth poller in prod. Its 5 s
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue