Even with the narrow HRRR grid pressure levels (4fa6798), pods are
still OOMKilled partway through the propagation chain. After the
HRRR fetch, the grid worker pipes 92k grid points through native
duct metric merge (~100 MB held for ~2 minutes), builds the full
weather grid cache (92k rows × ~40 fields), runs compute_scores
(460k score maps across 5 bands materialized in a list), and
writes five .ntms files — all while ERA5Submit workers keep
churning through CDS backfill in background processes.
Nodes have ~42% memory headroom on limits; the 2 Gi bump fits
comfortably. Shrinking the in-process allocations is a better
long-term fix but needs a proper pass through compute_scores and
persist_profiles. This gets prod unstuck in the meantime so the
forecast timeline can actually populate.
Graham McIntire
0c5cbbdc54