fix(grid-rs): bump memory 2→3 Gi, parallelism 3→2, re-trigger Rust CI

Two bugs surfacing 13 h after Stream A cutover:

1. prop-grid-rs pods OOMKilled 16–18× overnight on the analysis step.
   f00's native-level GRIB2 (~530 MB + wgrib2 working set) runs on top
   of 2 concurrent forecast tasks, briefly reaching ~2 Gi. 2 Gi limit
   was too tight — 3 Gi plus parallelism 3→2 gives the analysis step
   room without eliminating forecast-lane headroom.

2. hrrr-point-rs pod is CrashLoopBackOff because its container is
   running image main-1776640915-65f7963 (pre-Stream-C) which doesn't
   contain the hrrr_point_worker binary. The grid-rs CI run for
   commit 4fefb81 failed and no newer image got published. Touch the
   Dockerfile to re-trigger the workflow so flux picks up a new tag
   with both binaries. No functional Dockerfile change, just a
   docstring update so the path-filter kicks.

Forecast + analysis both claim per FOR UPDATE SKIP LOCKED, so
reducing per-pod parallelism doesn't break the chain — the two
replicas cover 4 slots cluster-wide, which still drains f01..f18 in
~5 min.
This commit is contained in:
Graham McIntire 2026-04-20 08:20:43 -05:00
parent 2421bf562a
commit a3cfff3048
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 19 additions and 15 deletions

View file

@ -82,19 +82,17 @@ spec:
value: "/data/scores" value: "/data/scores"
- name: RUST_LOG - name: RUST_LOG
value: "info" value: "info"
# talos5 is 4c/4t (i5-7260U). Each replica claims up to this # talos5 is 4c/4t (i5-7260U). 2 concurrent tasks per pod ×
# many forecast hours in parallel. With 2 replicas × 3 → 6 # 2 replicas = 4 cluster-wide, enough to drain f01..f18 in
# concurrent tasks, the 18-hour chain completes in ~3 min. # ~5 min while keeping analysis (f00) room for its native-
# Per-task peak RAM is ~250 Mi post-streaming refactor, so # duct wgrib2 peak without OOM. Raising to 3 reintroduced
# 3×250 ≈ 750 Mi per pod, well under the 2 Gi limit. The # the Stream-A-cutover OOM on the analysis step.
# secondary replica may land on a smaller node — 3 keeps it
# honest.
- name: PROP_GRID_RS_PARALLELISM - name: PROP_GRID_RS_PARALLELISM
value: "3" value: "2"
# Pool size = parallelism + 2 (NOTIFY + retry slack). Kept in # Pool size = parallelism + 2 (NOTIFY + retry slack). Kept in
# sync with PROP_GRID_RS_PARALLELISM when either changes. # sync with PROP_GRID_RS_PARALLELISM when either changes.
- name: PROP_GRID_RS_PG_CONNS - name: PROP_GRID_RS_PG_CONNS
value: "5" value: "4"
- name: METRICS_ADDR - name: METRICS_ADDR
value: "0.0.0.0:9100" value: "0.0.0.0:9100"
envFrom: envFrom:
@ -126,14 +124,16 @@ spec:
resources: resources:
requests: requests:
cpu: 100m cpu: 100m
memory: 512Mi memory: 768Mi
limits: limits:
cpu: "4" cpu: "4"
# OOM at 1 Gi during active chain work (92k points × 23 # Analysis tasks (f00) run the native-level HRRR decode
# bands ≈ 2.1M scores, plus decoded GRIB2 buffers, plus # (~530 MB GRIB2 + wgrib2 working set) on top of 2 concurrent
# wgrib2 subprocess memory). 2 Gi leaves headroom; we're # forecast tasks. 2 Gi OOM'd reliably in the 13 h after
# still an order of magnitude below the 6 Gi Elixir budget. # Stream A cutover. 3 Gi leaves room for the analysis
memory: 2Gi # wgrib2 peak without eliminating the forecast-lane
# parallelism headroom.
memory: 3Gi
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
runAsNonRoot: true runAsNonRoot: true

View file

@ -6,6 +6,10 @@
# the already-published Elixir image. # the already-published Elixir image.
# #
# Runtime uid 65534 matches the fsGroup on the NFS scores mount. # Runtime uid 65534 matches the fsGroup on the NFS scores mount.
#
# This image ships two binaries: `prop-grid-rs` (default ENTRYPOINT,
# drains grid_tasks) and `hrrr-point-worker` (k8s `command:` override,
# drains hrrr_fetch_tasks).
ARG WGRIB2_IMAGE=git.mcintire.me/graham/prop:latest ARG WGRIB2_IMAGE=git.mcintire.me/graham/prop:latest
FROM ${WGRIB2_IMAGE} AS wgrib2-src FROM ${WGRIB2_IMAGE} AS wgrib2-src