From 803130ee64b98c2abfb731036cdd0eaa6184744d Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Sun, 3 May 2026 16:25:00 -0500 Subject: [PATCH] =?UTF-8?q?perf(weather):=20drop=20GridCache=20valid-time?= =?UTF-8?q?=20cap=20from=2024=20=E2=86=92=208?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found while auditing pod memory: :weather_grid_cache is by far the largest ETS table (~32 MiB compressed per entry, vs ~2 MiB/entry for ScoreCache) because each cell is a 22-field weather row instead of a scalar score. Cap of 24 meant the cache could hoard up to ~768 MiB on a single pod, and hot pods sit at 3-5 GiB against a 6 GiB limit — most of that headroom belongs to GridCache. The cap was sized for 'analysis + 18 forecasts + stragglers' but the real access pattern is current-hour + a short forward scrub. Beyond that, fall back to ScalarFile.read_bounds (sub-100 ms per file) instead of permanently parking ~500 MiB on data nobody looks at. Worst-case GridCache spend drops 768 MiB → 256 MiB. Steady-state win is smaller (most pods don't fill all 24 slots), but bounds the tail. --- lib/microwaveprop/weather.ex | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/lib/microwaveprop/weather.ex b/lib/microwaveprop/weather.ex index f6d1775f..0b304d5c 100644 --- a/lib/microwaveprop/weather.ex +++ b/lib/microwaveprop/weather.ex @@ -988,10 +988,19 @@ defmodule Microwaveprop.Weather do end end - # Cap on cached valid_times. HRRR runs hourly with 18 forecast hours, so - # 24 covers analysis + 18 forecasts plus a few late stragglers from the - # previous run while a user is scrubbing. - @grid_cache_valid_time_cap 24 + # Cap on cached valid_times. Each entry is a chunked map of the full + # ~92k-cell CONUS grid with 22 fields per cell, ~32 MiB compressed in + # ETS. The previous cap of 24 meant the cache could hoard up to + # ~768 MiB on a single pod against a 6 GiB memory limit (the headroom + # that disappeared in the 2026-05-03 OOM cascade). + # + # 8 covers the current hour plus the next ~7 forecast hours — the + # typical user scrub window. Hours beyond 8 fall back to a disk read + # (`ScalarFile.read_bounds`) which is sub-100 ms per file, so rare + # deep-forecast scrubs pay a one-shot latency hit instead of every + # pod permanently parking ~500 MiB on data the user almost never + # looks at. Worst case ETS spend drops from 768 MiB → 256 MiB. + @grid_cache_valid_time_cap 8 # Hydrate GridCache from the on-disk ScalarFile so concurrent viewport # reads for the same valid_time don't each gunzip+msgpack-decode the