Graham McIntire
5e9513bf02
HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.
Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.
CLAUDE.md updates so this doesn't bite again:
- Document `cargo clippy --all-targets -- -D warnings` as part of
the local Rust precommit ritual. The HRDPS commits in this branch
passed `cargo build` locally but failed CI on lints (manual_range_
contains, iter_kv_map). Adding the recipe under Commands plus a
Rust subsection in Key Conventions so future Rust touches don't
ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
test (fix in 2768fc68) was a textbook example: 50% flake rate
driven entirely by which minute of the hour the test ran in.
146 lines
4.6 KiB
YAML
146 lines
4.6 KiB
YAML
# Horizontal pod autoscalers for the three load-responsive Deployments.
|
||
# prop-backfill is deliberately excluded — it's a single-replica catch-up
|
||
# worker that's bottlenecked on IEM rate-limit waits rather than CPU, so
|
||
# scaling by CPU would just add idle replicas parked on 429 backoffs.
|
||
#
|
||
# CPU targets use AverageValue (absolute cores) rather than Utilization
|
||
# (% of request) so the scale-up thresholds don't depend on the request
|
||
# number. Request fields in the Deployments can be tuned independently
|
||
# for scheduler bin-packing without accidentally changing when HPA
|
||
# decides to add a replica.
|
||
#
|
||
# The corresponding Deployments have their `spec.replicas` removed so
|
||
# Flux's periodic reconcile doesn't fight HPA's live scaling.
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: prop
|
||
namespace: prop
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: prop
|
||
# Pinned at 1 pod (min == max). HPA still owns the replica count so
|
||
# Flux reconcile stays out of the way; raise both bounds together to
|
||
# restore autoscaling.
|
||
minReplicas: 1
|
||
maxReplicas: 1
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
# Average of 800m across pods triggers scale-up. The hourly
|
||
# :05 propagation chain + ScoreCache warming on one pod can
|
||
# peak near 2 cores — averaging across 2–3 replicas keeps the
|
||
# threshold meaningful without flapping on every cron tick.
|
||
type: AverageValue
|
||
averageValue: 800m
|
||
behavior:
|
||
scaleUp:
|
||
# React quickly to sudden surges (LiveView + propagation chain
|
||
# can coincide). Add up to 2 pods or 100% of current replica
|
||
# count per 30 s window, whichever is larger.
|
||
stabilizationWindowSeconds: 30
|
||
policies:
|
||
- type: Pods
|
||
value: 2
|
||
periodSeconds: 30
|
||
- type: Percent
|
||
value: 100
|
||
periodSeconds: 30
|
||
selectPolicy: Max
|
||
scaleDown:
|
||
# Slower scale-down so the hourly chain doesn't immediately
|
||
# trigger a churn of terminations five minutes later.
|
||
stabilizationWindowSeconds: 600
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 120
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: prop-grid-rs
|
||
namespace: prop
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: prop-grid-rs
|
||
minReplicas: 1
|
||
maxReplicas: 4
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: AverageValue
|
||
averageValue: 400m
|
||
behavior:
|
||
scaleUp:
|
||
# Genuine surges (a stuck queue, multiple cycles draining together)
|
||
# still bring up replicas within ~30s. Most hourly chain steps
|
||
# finish in 2-3 min on a single pod, so transient single-pod
|
||
# spikes mostly resolve before HPA notices.
|
||
stabilizationWindowSeconds: 30
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 30
|
||
scaleDown:
|
||
# The previous 10-minute stabilization + 1-pod-per-2-min drain meant
|
||
# an hourly chain spike that briefly scaled to 4 pods sat at 4 pods
|
||
# for ~16 min after the spike cleared. Bring it down to 2 min
|
||
# stabilization + 100%-per-60s drain so the steady state is 1 pod
|
||
# within ~3 min of any spike ending.
|
||
stabilizationWindowSeconds: 120
|
||
policies:
|
||
- type: Percent
|
||
value: 100
|
||
periodSeconds: 60
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: hrrr-point-rs
|
||
namespace: prop
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: hrrr-point-rs
|
||
minReplicas: 1
|
||
maxReplicas: 4
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: AverageValue
|
||
averageValue: 400m
|
||
behavior:
|
||
scaleUp:
|
||
# Genuine surges (a stuck queue, multiple cycles draining together)
|
||
# still bring up replicas within ~30s. Most hourly chain steps
|
||
# finish in 2-3 min on a single pod, so transient single-pod
|
||
# spikes mostly resolve before HPA notices.
|
||
stabilizationWindowSeconds: 30
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 30
|
||
scaleDown:
|
||
# The previous 10-minute stabilization + 1-pod-per-2-min drain meant
|
||
# an hourly chain spike that briefly scaled to 4 pods sat at 4 pods
|
||
# for ~16 min after the spike cleared. Bring it down to 2 min
|
||
# stabilization + 100%-per-60s drain so the steady state is 1 pod
|
||
# within ~3 min of any spike ending.
|
||
stabilizationWindowSeconds: 120
|
||
policies:
|
||
- type: Percent
|
||
value: 100
|
||
periodSeconds: 60
|