fix(grid-rs): bump memory 2→3 Gi, parallelism 3→2, re-trigger Rust CI

Two bugs surfacing 13 h after Stream A cutover:

1. prop-grid-rs pods OOMKilled 16–18× overnight on the analysis step.
   f00's native-level GRIB2 (~530 MB + wgrib2 working set) runs on top
   of 2 concurrent forecast tasks, briefly reaching ~2 Gi. 2 Gi limit
   was too tight — 3 Gi plus parallelism 3→2 gives the analysis step
   room without eliminating forecast-lane headroom.

2. hrrr-point-rs pod is CrashLoopBackOff because its container is
   running image main-1776640915-65f7963 (pre-Stream-C) which doesn't
   contain the hrrr_point_worker binary. The grid-rs CI run for
   commit 4fefb81 failed and no newer image got published. Touch the
   Dockerfile to re-trigger the workflow so flux picks up a new tag
   with both binaries. No functional Dockerfile change, just a
   docstring update so the path-filter kicks.

Forecast + analysis both claim per FOR UPDATE SKIP LOCKED, so
reducing per-pod parallelism doesn't break the chain — the two
replicas cover 4 slots cluster-wide, which still drains f01..f18 in
~5 min.
This commit is contained in:
Graham McIntire 2026-04-20 08:20:43 -05:00
parent 2421bf562a
commit a3cfff3048
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 19 additions and 15 deletions

View file

@ -82,19 +82,17 @@ spec:
value: "/data/scores"
- name: RUST_LOG
value: "info"
# talos5 is 4c/4t (i5-7260U). Each replica claims up to this
# many forecast hours in parallel. With 2 replicas × 3 → 6
# concurrent tasks, the 18-hour chain completes in ~3 min.
# Per-task peak RAM is ~250 Mi post-streaming refactor, so
# 3×250 ≈ 750 Mi per pod, well under the 2 Gi limit. The
# secondary replica may land on a smaller node — 3 keeps it
# honest.
# talos5 is 4c/4t (i5-7260U). 2 concurrent tasks per pod ×
# 2 replicas = 4 cluster-wide, enough to drain f01..f18 in
# ~5 min while keeping analysis (f00) room for its native-
# duct wgrib2 peak without OOM. Raising to 3 reintroduced
# the Stream-A-cutover OOM on the analysis step.
- name: PROP_GRID_RS_PARALLELISM
value: "3"
value: "2"
# Pool size = parallelism + 2 (NOTIFY + retry slack). Kept in
# sync with PROP_GRID_RS_PARALLELISM when either changes.
- name: PROP_GRID_RS_PG_CONNS
value: "5"
value: "4"
- name: METRICS_ADDR
value: "0.0.0.0:9100"
envFrom:
@ -126,14 +124,16 @@ spec:
resources:
requests:
cpu: 100m
memory: 512Mi
memory: 768Mi
limits:
cpu: "4"
# OOM at 1 Gi during active chain work (92k points × 23
# bands ≈ 2.1M scores, plus decoded GRIB2 buffers, plus
# wgrib2 subprocess memory). 2 Gi leaves headroom; we're
# still an order of magnitude below the 6 Gi Elixir budget.
memory: 2Gi
# Analysis tasks (f00) run the native-level HRRR decode
# (~530 MB GRIB2 + wgrib2 working set) on top of 2 concurrent
# forecast tasks. 2 Gi OOM'd reliably in the 13 h after
# Stream A cutover. 3 Gi leaves room for the analysis
# wgrib2 peak without eliminating the forecast-lane
# parallelism headroom.
memory: 3Gi
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true

View file

@ -6,6 +6,10 @@
# the already-published Elixir image.
#
# Runtime uid 65534 matches the fsGroup on the NFS scores mount.
#
# This image ships two binaries: `prop-grid-rs` (default ENTRYPOINT,
# drains grid_tasks) and `hrrr-point-worker` (k8s `command:` override,
# drains hrrr_fetch_tasks).
ARG WGRIB2_IMAGE=git.mcintire.me/graham/prop:latest
FROM ${WGRIB2_IMAGE} AS wgrib2-src