graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(oban): expand :rover_path slot-1 rationale with memory budget

Browse source 
Future-proof the queue config so a "let's bump concurrency for
throughput" change has the per-pod RSS math in front of it. Includes
the actual memory limit (6 GiB), observed steady state (3-5 GiB), the
~500 MiB peak per path-compute, and the explicit recommendation to add
hot replicas via HPA or offload HRRR fetch to hrrr-point-rs before
touching this slot count.
This commit is contained in:
Graham McIntire 2026-05-03 15:47:42 -05:00
parent 9709a42190
commit f7eb888123
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
1 changed files with 15 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
config/config.exs
View file

@ -74,11 +74,21 @@ config :microwaveprop, Oban,
enqueue: 1, enqueue: 1,
gefs: 2, gefs: 2,
terrain: 4, terrain: 4,
# RoverPathProfileWorker runs the same full HRRR/terrain/score # DO NOT raise above 1 without measuring per-pod RSS first.
# /loss/forecast pipeline as the propagation grid chain. Kept on # RoverPathProfileWorker runs the full PathCompute pipeline
# its own queue at 1 slot/pod so a backfill flood can't stack # (terrain sample analysis + 9 HRRR profile fetches/decodes +
# multiple heavy path-computes in one BEAM heap (the OOM that # sounding readout + ionosphere readout + per-band scoring +
# took down hot pods on 2026-05-03). # loss + power + 18 h forecast). Per-job heap is roughly the
# same shape as the :propagation grid chain — ~500 MiB peak.
# With the deployment.yaml memory limit at 6 GiB and steady-state
# hot pods sitting at 3-5 GiB (ScoreCache + GridCache + LiveView
# state), even 2 concurrent path-computes can push a hot pod past
# the limit and trigger OOMKill. Verified on 2026-05-03: a
# backfill flood ran 3 slots/pod × 4 hot pods on the :terrain
# queue and OOMKilled two pods in a CrashLoopBackOff cycle.
# If you need more throughput, prefer adding hot replicas (HPA
# max is 4 in k8s/hpa.yaml) over raising this slot count, OR
# offload the HRRR profile fetch to hrrr-point-rs first.
rover_path: 1, rover_path: 1,
commercial: 2, commercial: 2,
iemre: 10, iemre: 10,