ops(prop): aggressive HPA scale-down + CLAUDE.md guardrails
HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.
Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.
CLAUDE.md updates so this doesn't bite again:
- Document `cargo clippy --all-targets -- -D warnings` as part of
the local Rust precommit ritual. The HRDPS commits in this branch
passed `cargo build` locally but failed CI on lints (manual_range_
contains, iter_kv_map). Adding the recipe under Commands plus a
Rust subsection in Key Conventions so future Rust touches don't
ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
test (fix in 2768fc68) was a textbook example: 50% flake rate
driven entirely by which minute of the hour the test ran in.
This commit is contained in:
parent
42099f84ad
commit
5e9513bf02
2 changed files with 35 additions and 8 deletions
|
|
@ -56,6 +56,10 @@ mix credo
|
||||||
# Pre-commit check (compile with warnings-as-errors, unlock unused deps, format, test)
|
# Pre-commit check (compile with warnings-as-errors, unlock unused deps, format, test)
|
||||||
mix precommit
|
mix precommit
|
||||||
|
|
||||||
|
# Rust pre-commit (matches the Forgejo CI step). Run from rust/prop_grid_rs/.
|
||||||
|
# `cargo build` alone does NOT catch clippy errors that gate the CI build.
|
||||||
|
cd rust/prop_grid_rs && cargo clippy --all-targets -- -D warnings && cargo test --release
|
||||||
|
|
||||||
# Database
|
# Database
|
||||||
mix ecto.create
|
mix ecto.create
|
||||||
mix ecto.migrate
|
mix ecto.migrate
|
||||||
|
|
@ -211,6 +215,11 @@ User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers
|
||||||
- Stub HTTP calls with `Req.Test.stub` in test setup
|
- Stub HTTP calls with `Req.Test.stub` in test setup
|
||||||
- Oban runs inline in test (`testing: :inline`)
|
- Oban runs inline in test (`testing: :inline`)
|
||||||
- FreshnessMonitor disabled in test via `config :microwaveprop, start_freshness_monitor: false`
|
- FreshnessMonitor disabled in test via `config :microwaveprop, start_freshness_monitor: false`
|
||||||
|
- **Don't anchor relative-time test fixtures on `DateTime.utc_now()` truncated to the hour.** A test that computes `recent_t = (top_of_hour) - 30min` is 30-90 min old of real `utc_now()` depending on what minute the test runs at; any production code that compares against `utc_now()` (e.g. a "drop entries older than 1 h" filter) flakes 50% of the time. Anchor at `top_of_hour` (or use a tiny offset like `now - 1s`) so the test is stable across clock minutes.
|
||||||
|
|
||||||
|
### Rust (`prop_grid_rs`)
|
||||||
|
- **Always run `cargo clippy --all-targets -- -D warnings` before committing Rust changes.** The Forgejo CI step runs clippy with warnings-as-errors and a passing local `cargo build` does NOT catch lints (e.g. `manual_range_contains`, `iter_kv_map`, `useless_format`). A clippy failure blocks the image build and the deploy. The Rust precommit recipe is in the `Commands` section.
|
||||||
|
- The CI build pipeline runs `cargo clippy --all-targets -- -D warnings && cargo test --release && cargo build --release`. Match this locally — `cargo build` alone is not enough.
|
||||||
|
|
||||||
### Deployment
|
### Deployment
|
||||||
- Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...`
|
- Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...`
|
||||||
|
|
|
||||||
34
k8s/hpa.yaml
34
k8s/hpa.yaml
|
|
@ -82,17 +82,26 @@ spec:
|
||||||
averageValue: 400m
|
averageValue: 400m
|
||||||
behavior:
|
behavior:
|
||||||
scaleUp:
|
scaleUp:
|
||||||
|
# Genuine surges (a stuck queue, multiple cycles draining together)
|
||||||
|
# still bring up replicas within ~30s. Most hourly chain steps
|
||||||
|
# finish in 2-3 min on a single pod, so transient single-pod
|
||||||
|
# spikes mostly resolve before HPA notices.
|
||||||
stabilizationWindowSeconds: 30
|
stabilizationWindowSeconds: 30
|
||||||
policies:
|
policies:
|
||||||
- type: Pods
|
- type: Pods
|
||||||
value: 1
|
value: 1
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
scaleDown:
|
scaleDown:
|
||||||
stabilizationWindowSeconds: 600
|
# The previous 10-minute stabilization + 1-pod-per-2-min drain meant
|
||||||
|
# an hourly chain spike that briefly scaled to 4 pods sat at 4 pods
|
||||||
|
# for ~16 min after the spike cleared. Bring it down to 2 min
|
||||||
|
# stabilization + 100%-per-60s drain so the steady state is 1 pod
|
||||||
|
# within ~3 min of any spike ending.
|
||||||
|
stabilizationWindowSeconds: 120
|
||||||
policies:
|
policies:
|
||||||
- type: Pods
|
- type: Percent
|
||||||
value: 1
|
value: 100
|
||||||
periodSeconds: 120
|
periodSeconds: 60
|
||||||
---
|
---
|
||||||
apiVersion: autoscaling/v2
|
apiVersion: autoscaling/v2
|
||||||
kind: HorizontalPodAutoscaler
|
kind: HorizontalPodAutoscaler
|
||||||
|
|
@ -115,14 +124,23 @@ spec:
|
||||||
averageValue: 400m
|
averageValue: 400m
|
||||||
behavior:
|
behavior:
|
||||||
scaleUp:
|
scaleUp:
|
||||||
|
# Genuine surges (a stuck queue, multiple cycles draining together)
|
||||||
|
# still bring up replicas within ~30s. Most hourly chain steps
|
||||||
|
# finish in 2-3 min on a single pod, so transient single-pod
|
||||||
|
# spikes mostly resolve before HPA notices.
|
||||||
stabilizationWindowSeconds: 30
|
stabilizationWindowSeconds: 30
|
||||||
policies:
|
policies:
|
||||||
- type: Pods
|
- type: Pods
|
||||||
value: 1
|
value: 1
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
scaleDown:
|
scaleDown:
|
||||||
stabilizationWindowSeconds: 600
|
# The previous 10-minute stabilization + 1-pod-per-2-min drain meant
|
||||||
|
# an hourly chain spike that briefly scaled to 4 pods sat at 4 pods
|
||||||
|
# for ~16 min after the spike cleared. Bring it down to 2 min
|
||||||
|
# stabilization + 100%-per-60s drain so the steady state is 1 pod
|
||||||
|
# within ~3 min of any spike ending.
|
||||||
|
stabilizationWindowSeconds: 120
|
||||||
policies:
|
policies:
|
||||||
- type: Pods
|
- type: Percent
|
||||||
value: 1
|
value: 100
|
||||||
periodSeconds: 120
|
periodSeconds: 60
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue