prop-grid-rs already exposed prop_grid_rs_chain_step_duration_seconds
+ tasks_in_flight on :9100 but the comment still pointed at the old
prom server (10.0.15.25). hrrr-point-rs had no metrics listener at all.
- metrics.rs: new POINT_BATCH_DURATION / POINT_BATCHES_TOTAL /
POINTS_PROCESSED_TOTAL series + record_point_batch helper, plus unit
tests serialized via a static Mutex (the prometheus crate's global
registry races otherwise)
- hrrr_point_worker.rs: spawn metrics::serve on METRICS_ADDR (default
0.0.0.0:9100), wrap process_batch in InFlightGuard, record outcome +
per-point counts on every batch
- deployment-hrrr-point-rs.yaml: prometheus.io/{scrape,port,path,job}
annotations, METRICS_ADDR env, ports.metrics, /health readiness probe
- deployment-grid-rs.yaml: refresh stale 10.0.15.25 comment, add
prometheus.io/job for clean dashboard up{} selectors
Also fix Microwaveprop.PromExTest: `assert plugins != []` triggered
Elixir 1.19 type-checker warning ("non_empty_list != empty_list always
true"); replaced with membership assertions for the InstrumentPlugin +
Plugins.Beam.
External Prometheus at 10.0.15.31 already runs the kubernetes-pods job
in prometheus.yml.j2; pods need prometheus.io/scrape + port + path +
job annotations to be picked up. The prop-metrics NodePort selector
also tightened to tier=hot so backfill (PHX_SERVER=false, no :5000
listener) is excluded from the scrape pool.
The image tags here are placeholders — the live tag is set by the
ArgoCD Application's spec.source.kustomize.images field, same as
before. Only the registry hostname and path changed.
Pull secret 'forgejo-registry' in the prop namespace now carries
both git.mcintire.me and codeberg.org auths during the transition.
A single-pod floor meant any pod restart (rolling deploy mid-run,
OOM kill, node drain) took the whole site down for the boot window.
Bump HPA to minReplicas: 2 / maxReplicas: 4 so the floor is HA and
load surges (LiveView traffic coinciding with the hourly propagation
chain) still get headroom. Combined with the existing rolling-update
strategy (maxSurge: 1, maxUnavailable: 0) this means deploys go
2 -> 3 -> 2 with zero downtime.
HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.
Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.
CLAUDE.md updates so this doesn't bite again:
- Document `cargo clippy --all-targets -- -D warnings` as part of
the local Rust precommit ritual. The HRDPS commits in this branch
passed `cargo build` locally but failed CI on lints (manual_range_
contains, iter_kv_map). Adding the recipe under Commands plus a
Rust subsection in Key Conventions so future Rust touches don't
ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
test (fix in 2768fc68) was a textbook example: 50% flake rate
driven entirely by which minute of the hour the test ran in.
The ingress is cloudflared (Cloudflare Tunnel), which keeps an HTTP/2
connection pool to upstream Service IPs and reuses connections
aggressively. After a pod is removed from Service endpoints,
cloudflared can still hold in-flight HTTP/2 streams to the dying pod
for a noticeable window — 10 s wasn't enough, deploys still bounced
502 on LiveView reconnect.
Bump preStop sleep 10s → 25s and terminationGracePeriodSeconds 40s →
60s so cloudflared has time to turn over its upstream connections
before Phoenix begins draining.
Deploys were causing 502 Bad Gateway on LiveView reconnect. The kubelet
sent SIGTERM to the old pod and Phoenix immediately stopped accepting
connections, but the Service endpoint controller and ingress hadn't
yet propagated the pod's removal from endpoints — the browser's
reconnect raced the endpoint update and bounced 502 against the dead
pod, then gave up.
Add a 10s preStop sleep on the container and bump
terminationGracePeriodSeconds to 40 so the load balancer has time to
drain this pod before Phoenix begins shutting down.
maxUnavailable: 1 was killing the only pod before the new one was
ready, taking the site down for the entire image pull + boot window
on every deploy. Switch to maxSurge: 1 / maxUnavailable: 0 so K8s
spins up the replacement and waits for its readinessProbe to pass
before tearing down the old pod.
Set minReplicas == maxReplicas == 1 on the `prop` HPA so the cluster
runs exactly one app pod. HPA still owns the replica count (Flux
ignores spec.replicas on the Deployment), so this is the right knob —
no need to touch deployment.yaml. Restore autoscaling by raising both
bounds together.
