ops: split RoverPathProfileWorker onto :rover_path queue at 1 slot/pod
The :terrain queue at 3 slots/pod was sized for the lightweight per-QSO TerrainProfileWorker, but RoverPathProfileWorker now runs the full PathCompute pipeline (terrain + 9 HRRR profiles + sounding + ionosphere + scoring + loss + forecast) — same heap profile as :propagation, which sits at 1 slot precisely to avoid OOM. A backfill_paths flood today queued ~200 rover-path jobs. With 4 hot pods × 3 slots, the cluster stacked 15 concurrent path-computes and two pods OOM-killed in a loop, taking the libcluster ring with them (visible as the "unable to connect" warnings on the surviving pods). New :rover_path queue at 1 slot/pod caps cluster-wide concurrency at (hot_replicas + 1 backfill), drains the existing 200-job backlog inside ~30 min, and keeps :terrain free for the cheap workers it was sized for.
This commit is contained in:
parent
1abf452558
commit
9709a42190
3 changed files with 16 additions and 1 deletions
|
|
@ -74,6 +74,12 @@ config :microwaveprop, Oban,
|
|||
enqueue: 1,
|
||||
gefs: 2,
|
||||
terrain: 4,
|
||||
# RoverPathProfileWorker runs the same full HRRR/terrain/score
|
||||
# /loss/forecast pipeline as the propagation grid chain. Kept on
|
||||
# its own queue at 1 slot/pod so a backfill flood can't stack
|
||||
# multiple heavy path-computes in one BEAM heap (the OOM that
|
||||
# took down hot pods on 2026-05-03).
|
||||
rover_path: 1,
|
||||
commercial: 2,
|
||||
iemre: 10,
|
||||
# 2 slots so PropagationPruneWorker can run alongside the
|
||||
|
|
|
|||
|
|
@ -172,6 +172,15 @@ if config_env() == :prod do
|
|||
# :hrrr queue removed — per-QSO HRRR fetches flow through
|
||||
# hrrr_fetch_tasks and the Rust hrrr-point-worker (Phase 3 Stream C).
|
||||
terrain: 3,
|
||||
# 1 slot per pod, same reasoning as :propagation. Each
|
||||
# RoverPathProfileWorker job allocates a full path-compute heap
|
||||
# (terrain points + 9 HRRR profiles + sounding + ionosphere +
|
||||
# scoring + loss + forecast); 3 slots × 4 hot pods on the
|
||||
# :terrain queue OOM-killed pods on 2026-05-03 when a backfill
|
||||
# flood landed. Cluster-wide capacity is now (hot replicas + 1
|
||||
# backfill) jobs at a time, which still drains a 200-job
|
||||
# backlog inside ~30 min.
|
||||
rover_path: 1,
|
||||
iemre: 3,
|
||||
backfill_enqueue: 1,
|
||||
admin: 1,
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ defmodule Microwaveprop.Workers.RoverPathProfileWorker do
|
|||
handed `?rover_path_id=UUID` and renders without recomputing.
|
||||
"""
|
||||
use Oban.Worker,
|
||||
queue: :terrain,
|
||||
queue: :rover_path,
|
||||
max_attempts: 5,
|
||||
unique: [
|
||||
period: 300,
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue