Graham McIntire
9709a42190
The :terrain queue at 3 slots/pod was sized for the lightweight per-QSO TerrainProfileWorker, but RoverPathProfileWorker now runs the full PathCompute pipeline (terrain + 9 HRRR profiles + sounding + ionosphere + scoring + loss + forecast) — same heap profile as :propagation, which sits at 1 slot precisely to avoid OOM. A backfill_paths flood today queued ~200 rover-path jobs. With 4 hot pods × 3 slots, the cluster stacked 15 concurrent path-computes and two pods OOM-killed in a loop, taking the libcluster ring with them (visible as the "unable to connect" warnings on the surviving pods). New :rover_path queue at 1 slot/pod caps cluster-wide concurrency at (hot_replicas + 1 backfill), drains the existing 200-job backlog inside ~30 min, and keeps :terrain free for the cheap workers it was sized for. |
||
|---|---|---|
| .. | ||
| microwaveprop | ||
| microwaveprop_web | ||
| mix/tasks | ||
| microwaveprop.ex | ||
| microwaveprop_web.ex | ||