Graham McIntire
86b2718916
Adds HorizontalPodAutoscaler resources for the three load-responsive Deployments: - prop (web): min 2, max 5, averageValue 800m CPU - prop-grid-rs: min 1, max 4, averageValue 400m - hrrr-point-rs: min 1, max 4, averageValue 400m - prop-backfill is intentionally unmanaged (queue-driven, not CPU-bound) The scale-up policy reacts in 30 s with up-to-2-pods or +100 % per window; scale-down stabilises over 10 min so the hourly propagation chain doesn't churn replicas right after the :05 cron. Removes `spec.replicas` from the HPA-managed Deployments so Flux's periodic reconcile stops fighting HPA's live scaling decisions. Drops `podAntiAffinity` from prop, prop-grid-rs, and hrrr-point-rs so HPA can actually reach its maxReplicas when one physical host has capacity for two pods; the scheduler picks whatever fits. Updates runbook + project memory to drop the `talos5` hostname — Postgres is now addressed strictly by IP (10.0.15.30).
125 lines
3.3 KiB
YAML
125 lines
3.3 KiB
YAML
# Horizontal pod autoscalers for the three load-responsive Deployments.
|
||
# prop-backfill is deliberately excluded — it's a single-replica catch-up
|
||
# worker that's bottlenecked on IEM rate-limit waits rather than CPU, so
|
||
# scaling by CPU would just add idle replicas parked on 429 backoffs.
|
||
#
|
||
# CPU targets use AverageValue (absolute cores) rather than Utilization
|
||
# (% of request) so the scale-up thresholds don't depend on the request
|
||
# number. Request fields in the Deployments can be tuned independently
|
||
# for scheduler bin-packing without accidentally changing when HPA
|
||
# decides to add a replica.
|
||
#
|
||
# The corresponding Deployments have their `spec.replicas` removed so
|
||
# Flux's periodic reconcile doesn't fight HPA's live scaling.
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: prop
|
||
namespace: prop
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: prop
|
||
minReplicas: 2
|
||
maxReplicas: 5
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
# Average of 800m across pods triggers scale-up. The hourly
|
||
# :05 propagation chain + ScoreCache warming on one pod can
|
||
# peak near 2 cores — averaging across 2–3 replicas keeps the
|
||
# threshold meaningful without flapping on every cron tick.
|
||
type: AverageValue
|
||
averageValue: 800m
|
||
behavior:
|
||
scaleUp:
|
||
# React quickly to sudden surges (LiveView + propagation chain
|
||
# can coincide). Add up to 2 pods or 100% of current replica
|
||
# count per 30 s window, whichever is larger.
|
||
stabilizationWindowSeconds: 30
|
||
policies:
|
||
- type: Pods
|
||
value: 2
|
||
periodSeconds: 30
|
||
- type: Percent
|
||
value: 100
|
||
periodSeconds: 30
|
||
selectPolicy: Max
|
||
scaleDown:
|
||
# Slower scale-down so the hourly chain doesn't immediately
|
||
# trigger a churn of terminations five minutes later.
|
||
stabilizationWindowSeconds: 600
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 120
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: prop-grid-rs
|
||
namespace: prop
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: prop-grid-rs
|
||
minReplicas: 1
|
||
maxReplicas: 4
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: AverageValue
|
||
averageValue: 400m
|
||
behavior:
|
||
scaleUp:
|
||
stabilizationWindowSeconds: 30
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 30
|
||
scaleDown:
|
||
stabilizationWindowSeconds: 600
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 120
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: hrrr-point-rs
|
||
namespace: prop
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: hrrr-point-rs
|
||
minReplicas: 1
|
||
maxReplicas: 4
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: AverageValue
|
||
averageValue: 400m
|
||
behavior:
|
||
scaleUp:
|
||
stabilizationWindowSeconds: 30
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 30
|
||
scaleDown:
|
||
stabilizationWindowSeconds: 600
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 120
|