perf(weather): drop GridCache valid-time cap from 24 → 8

Found while auditing pod memory: :weather_grid_cache is by far the
largest ETS table (~32 MiB compressed per entry, vs ~2 MiB/entry for
ScoreCache) because each cell is a 22-field weather row instead of a
scalar score. Cap of 24 meant the cache could hoard up to ~768 MiB on
a single pod, and hot pods sit at 3-5 GiB against a 6 GiB limit — most
of that headroom belongs to GridCache.

The cap was sized for 'analysis + 18 forecasts + stragglers' but the
real access pattern is current-hour + a short forward scrub. Beyond
that, fall back to ScalarFile.read_bounds (sub-100 ms per file) instead
of permanently parking ~500 MiB on data nobody looks at.

Worst-case GridCache spend drops 768 MiB → 256 MiB. Steady-state win
is smaller (most pods don't fill all 24 slots), but bounds the tail.
This commit is contained in:
Graham McIntire 2026-05-03 16:25:00 -05:00
parent 9bec5721d0
commit 803130ee64
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -988,10 +988,19 @@ defmodule Microwaveprop.Weather do
end
end
# Cap on cached valid_times. HRRR runs hourly with 18 forecast hours, so
# 24 covers analysis + 18 forecasts plus a few late stragglers from the
# previous run while a user is scrubbing.
@grid_cache_valid_time_cap 24
# Cap on cached valid_times. Each entry is a chunked map of the full
# ~92k-cell CONUS grid with 22 fields per cell, ~32 MiB compressed in
# ETS. The previous cap of 24 meant the cache could hoard up to
# ~768 MiB on a single pod against a 6 GiB memory limit (the headroom
# that disappeared in the 2026-05-03 OOM cascade).
#
# 8 covers the current hour plus the next ~7 forecast hours — the
# typical user scrub window. Hours beyond 8 fall back to a disk read
# (`ScalarFile.read_bounds`) which is sub-100 ms per file, so rare
# deep-forecast scrubs pay a one-shot latency hit instead of every
# pod permanently parking ~500 MiB on data the user almost never
# looks at. Worst case ETS spend drops from 768 MiB → 256 MiB.
@grid_cache_valid_time_cap 8
# Hydrate GridCache from the on-disk ScalarFile so concurrent viewport
# reads for the same valid_time don't each gunzip+msgpack-decode the