k8s(grid-rs): drop parallelism to 1 after 3 Gi still OOMs at 2
Two concurrent forecast tasks plus the NFS score-file write cache (23 files x ~2 MB) plus the analysis step's wgrib2 peak repeatedly crossed the 3 Gi cgroup limit. Single-lane parallelism per pod keeps steady RSS under 2 Gi; with 2 replicas that still gives 2 concurrent tasks cluster-wide. PROP_GRID_RS_PG_CONNS follows the parallelism+2 formula, dropping from 4 to 3.
This commit is contained in:
parent
1135e4b156
commit
68910f4d72
1 changed files with 11 additions and 7 deletions
|
|
@ -82,17 +82,21 @@ spec:
|
||||||
value: "/data/scores"
|
value: "/data/scores"
|
||||||
- name: RUST_LOG
|
- name: RUST_LOG
|
||||||
value: "info"
|
value: "info"
|
||||||
# talos5 is 4c/4t (i5-7260U). 2 concurrent tasks per pod ×
|
# Single concurrent task per pod. 2 replicas → 2 concurrent
|
||||||
# 2 replicas = 4 cluster-wide, enough to drain f01..f18 in
|
# cluster-wide. Raised from 2 after the new 3 Gi limit still
|
||||||
# ~5 min while keeping analysis (f00) room for its native-
|
# OOMed under 2-parallel forecast load — the NFS-backed
|
||||||
# duct wgrib2 peak without OOM. Raising to 3 reintroduced
|
# score-file writes (23 files × ~2 MB each) keep cgroup page
|
||||||
# the Stream-A-cutover OOM on the analysis step.
|
# cache high enough that two concurrent tasks plus the
|
||||||
|
# wgrib2 working set regularly crosses 3 Gi. Single-lane
|
||||||
|
# parallelism stays comfortably under 2 Gi and lets the
|
||||||
|
# analysis step (native duct + NEXRAD + commercial merges)
|
||||||
|
# run safely within the same budget.
|
||||||
- name: PROP_GRID_RS_PARALLELISM
|
- name: PROP_GRID_RS_PARALLELISM
|
||||||
value: "2"
|
value: "1"
|
||||||
# Pool size = parallelism + 2 (NOTIFY + retry slack). Kept in
|
# Pool size = parallelism + 2 (NOTIFY + retry slack). Kept in
|
||||||
# sync with PROP_GRID_RS_PARALLELISM when either changes.
|
# sync with PROP_GRID_RS_PARALLELISM when either changes.
|
||||||
- name: PROP_GRID_RS_PG_CONNS
|
- name: PROP_GRID_RS_PG_CONNS
|
||||||
value: "4"
|
value: "3"
|
||||||
- name: METRICS_ADDR
|
- name: METRICS_ADDR
|
||||||
value: "0.0.0.0:9100"
|
value: "0.0.0.0:9100"
|
||||||
envFrom:
|
envFrom:
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue