Bump prop pod memory limit 4Gi → 6Gi

Even with the narrow HRRR grid pressure levels (4fa6798), pods are
still OOMKilled partway through the propagation chain. After the
HRRR fetch, the grid worker pipes 92k grid points through native
duct metric merge (~100 MB held for ~2 minutes), builds the full
weather grid cache (92k rows × ~40 fields), runs compute_scores
(460k score maps across 5 bands materialized in a list), and
writes five .ntms files — all while ERA5Submit workers keep
churning through CDS backfill in background processes.

Nodes have ~42% memory headroom on limits; the 2 Gi bump fits
comfortably. Shrinking the in-process allocations is a better
long-term fix but needs a proper pass through compute_scores and
persist_profiles. This gets prod unstuck in the meantime so the
forecast timeline can actually populate.
This commit is contained in:
Graham McIntire 2026-04-15 08:29:05 -05:00
parent 38f6c37194
commit 0c5cbbdc54
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -92,7 +92,7 @@ spec:
memory: 512Mi
limits:
cpu: "2"
memory: 4Gi
memory: 6Gi
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true