Graham McIntire
a1d2e9f94a
Three fixes after hrrr-point-rs restarts on the first live drain: 1. Type mismatch (hard error): the `profile` column on `hrrr_profiles` is `jsonb[]` (one element per pressure level), but the Rust worker was binding a single jsonb array-of-objects cast as `::jsonb`. Every insert failed with `column "profile" is of type jsonb[] but expression is of type jsonb`. Switched to `Vec<sqlx::types::Json<Value>>`, dropped the explicit `::jsonb` cast, and enabled sqlx's `json` feature so the encoder maps the array into `_jsonb` correctly. 2. Silent zero-inserts: four consecutive 2019-09-22 batches completed with `profiles_inserted: 0` and no diagnostic. Most likely the upstream archive doesn't keep cycles that old, but without a log it looks identical to a snap-mismatch bug. Added a WARN when both surface and pressure grids come back empty so fetch-miss vs. projection-miss is distinguishable. 3. OOMKilled: the pod took four OOM restarts in ten minutes at 1 Gi. A single CONUS decode holds the ~40 MB blob, the ~200 MB wgrib2 working set, AND the full 92k-cell merged map until all requested points drain. 1 Gi has no headroom. Bumped to 2 Gi, matching the rest of the per-container budgets in this namespace.
87 lines
3 KiB
YAML
87 lines
3 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: hrrr-point-rs
|
|
namespace: prop
|
|
# Phase 3 Stream C: per-QSO HRRR fetches move off the BEAM pods.
|
|
# Elixir writes to hrrr_fetch_tasks (one row per valid_time with a
|
|
# points JSONB array); this deployment drains via FOR UPDATE SKIP
|
|
# LOCKED, fetches each HRRR cycle once, and upserts hrrr_profiles.
|
|
spec:
|
|
replicas: 1
|
|
strategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxSurge: 0
|
|
maxUnavailable: 1
|
|
selector:
|
|
matchLabels:
|
|
app: hrrr-point-rs
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: hrrr-point-rs
|
|
tier: hrrr-points
|
|
spec:
|
|
affinity:
|
|
# Avoid co-locating with prop-grid-rs pods — they compete for
|
|
# the same wgrib2 subprocess slots and NFS fsyncs. Grid chain
|
|
# is latency-sensitive (users refreshing /map); per-QSO point
|
|
# work is throughput-sensitive but latency-tolerant.
|
|
podAntiAffinity:
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 80
|
|
podAffinityTerm:
|
|
labelSelector:
|
|
matchLabels:
|
|
app: prop-grid-rs
|
|
topologyKey: kubernetes.io/hostname
|
|
imagePullSecrets:
|
|
- name: forgejo-registry
|
|
securityContext:
|
|
runAsUser: 65534
|
|
runAsNonRoot: true
|
|
fsGroup: 65534
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: hrrr-point-rs
|
|
# Shares the prop-grid-rs image so one CI pipeline produces
|
|
# both binaries. k8s `command:` overrides the default
|
|
# ENTRYPOINT to launch the per-QSO point worker.
|
|
image: git.mcintire.me/graham/prop-grid-rs:main-1776693159-141956b # {"$imagepolicy": "flux-system:prop-grid-rs"}
|
|
imagePullPolicy: IfNotPresent
|
|
command: ["/usr/bin/tini", "--", "/usr/local/bin/hrrr-point-worker"]
|
|
env:
|
|
- name: HRRR_BASE_URL
|
|
value: "http://skippy.w5isp.com:8080"
|
|
- name: RUST_LOG
|
|
value: "info"
|
|
- name: PROP_GRID_RS_PG_CONNS
|
|
value: "4"
|
|
envFrom:
|
|
- secretRef:
|
|
name: prop-secrets
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
# Per-batch peak is a single CONUS surface + pressure
|
|
# decode (~40 MB blob, ~200 MB wgrib2 working set plus
|
|
# the full 92k-cell merged CellValues map kept in memory
|
|
# until all requested points are drained). 1 Gi OOMKilled
|
|
# repeatedly — four restarts in ten minutes — so bumped
|
|
# to 2 Gi. The scheduler-on-worker2 placement has plenty
|
|
# of headroom.
|
|
cpu: "2"
|
|
memory: 2Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
runAsNonRoot: true
|
|
runAsUser: 65534
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
seccompProfile:
|
|
type: RuntimeDefault
|