From 0c5cbbdc54673ad6a886274487e0f0022645ce8d Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Wed, 15 Apr 2026 08:29:05 -0500 Subject: [PATCH] =?UTF-8?q?Bump=20prop=20pod=20memory=20limit=204Gi=20?= =?UTF-8?q?=E2=86=92=206Gi?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Even with the narrow HRRR grid pressure levels (4fa6798), pods are still OOMKilled partway through the propagation chain. After the HRRR fetch, the grid worker pipes 92k grid points through native duct metric merge (~100 MB held for ~2 minutes), builds the full weather grid cache (92k rows × ~40 fields), runs compute_scores (460k score maps across 5 bands materialized in a list), and writes five .ntms files — all while ERA5Submit workers keep churning through CDS backfill in background processes. Nodes have ~42% memory headroom on limits; the 2 Gi bump fits comfortably. Shrinking the in-process allocations is a better long-term fix but needs a proper pass through compute_scores and persist_profiles. This gets prod unstuck in the meantime so the forecast timeline can actually populate. --- k8s/deployment.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/k8s/deployment.yaml b/k8s/deployment.yaml index eafabf9c..7e007f4b 100644 --- a/k8s/deployment.yaml +++ b/k8s/deployment.yaml @@ -92,7 +92,7 @@ spec: memory: 512Mi limits: cpu: "2" - memory: 4Gi + memory: 6Gi securityContext: allowPrivilegeEscalation: false runAsNonRoot: true