prop/k8s
Graham McIntire 0c5cbbdc54
Bump prop pod memory limit 4Gi → 6Gi
Even with the narrow HRRR grid pressure levels (4fa6798), pods are
still OOMKilled partway through the propagation chain. After the
HRRR fetch, the grid worker pipes 92k grid points through native
duct metric merge (~100 MB held for ~2 minutes), builds the full
weather grid cache (92k rows × ~40 fields), runs compute_scores
(460k score maps across 5 bands materialized in a list), and
writes five .ntms files — all while ERA5Submit workers keep
churning through CDS backfill in background processes.

Nodes have ~42% memory headroom on limits; the 2 Gi bump fits
comfortably. Shrinking the in-process allocations is a better
long-term fix but needs a proper pass through compute_scores and
persist_profiles. This gets prod unstuck in the meantime so the
forecast timeline can actually populate.
2026-04-15 08:29:09 -05:00
..
deployment.yaml Bump prop pod memory limit 4Gi → 6Gi 2026-04-15 08:29:09 -05:00
flux.yaml Add k8s manifests and Forgejo CI for cluster deployment 2026-04-06 09:25:24 -05:00
kustomization.yaml Drop secret.yaml from kustomization to unblock Flux reconciliation 2026-04-12 15:05:48 -05:00
namespace.yaml Add k8s manifests and Forgejo CI for cluster deployment 2026-04-06 09:25:24 -05:00
rbac.yaml Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
service.yaml Add session affinity to service for stable LiveView websockets 2026-04-06 09:48:24 -05:00