ops(prop): aggressive HPA scale-down + CLAUDE.md guardrails

HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.

Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.

CLAUDE.md updates so this doesn't bite again:

- Document `cargo clippy --all-targets -- -D warnings` as part of
  the local Rust precommit ritual. The HRDPS commits in this branch
  passed `cargo build` locally but failed CI on lints (manual_range_
  contains, iter_kv_map). Adding the recipe under Commands plus a
  Rust subsection in Key Conventions so future Rust touches don't
  ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
  utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
  test (fix in 2768fc68) was a textbook example: 50% flake rate
  driven entirely by which minute of the hour the test ran in.
This commit is contained in:
Graham McIntire 2026-04-29 17:44:17 -05:00
parent 42099f84ad
commit 5e9513bf02
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 35 additions and 8 deletions

View file

@ -56,6 +56,10 @@ mix credo
# Pre-commit check (compile with warnings-as-errors, unlock unused deps, format, test) # Pre-commit check (compile with warnings-as-errors, unlock unused deps, format, test)
mix precommit mix precommit
# Rust pre-commit (matches the Forgejo CI step). Run from rust/prop_grid_rs/.
# `cargo build` alone does NOT catch clippy errors that gate the CI build.
cd rust/prop_grid_rs && cargo clippy --all-targets -- -D warnings && cargo test --release
# Database # Database
mix ecto.create mix ecto.create
mix ecto.migrate mix ecto.migrate
@ -211,6 +215,11 @@ User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers
- Stub HTTP calls with `Req.Test.stub` in test setup - Stub HTTP calls with `Req.Test.stub` in test setup
- Oban runs inline in test (`testing: :inline`) - Oban runs inline in test (`testing: :inline`)
- FreshnessMonitor disabled in test via `config :microwaveprop, start_freshness_monitor: false` - FreshnessMonitor disabled in test via `config :microwaveprop, start_freshness_monitor: false`
- **Don't anchor relative-time test fixtures on `DateTime.utc_now()` truncated to the hour.** A test that computes `recent_t = (top_of_hour) - 30min` is 30-90 min old of real `utc_now()` depending on what minute the test runs at; any production code that compares against `utc_now()` (e.g. a "drop entries older than 1 h" filter) flakes 50% of the time. Anchor at `top_of_hour` (or use a tiny offset like `now - 1s`) so the test is stable across clock minutes.
### Rust (`prop_grid_rs`)
- **Always run `cargo clippy --all-targets -- -D warnings` before committing Rust changes.** The Forgejo CI step runs clippy with warnings-as-errors and a passing local `cargo build` does NOT catch lints (e.g. `manual_range_contains`, `iter_kv_map`, `useless_format`). A clippy failure blocks the image build and the deploy. The Rust precommit recipe is in the `Commands` section.
- The CI build pipeline runs `cargo clippy --all-targets -- -D warnings && cargo test --release && cargo build --release`. Match this locally — `cargo build` alone is not enough.
### Deployment ### Deployment
- Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...` - Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...`

View file

@ -82,17 +82,26 @@ spec:
averageValue: 400m averageValue: 400m
behavior: behavior:
scaleUp: scaleUp:
# Genuine surges (a stuck queue, multiple cycles draining together)
# still bring up replicas within ~30s. Most hourly chain steps
# finish in 2-3 min on a single pod, so transient single-pod
# spikes mostly resolve before HPA notices.
stabilizationWindowSeconds: 30 stabilizationWindowSeconds: 30
policies: policies:
- type: Pods - type: Pods
value: 1 value: 1
periodSeconds: 30 periodSeconds: 30
scaleDown: scaleDown:
stabilizationWindowSeconds: 600 # The previous 10-minute stabilization + 1-pod-per-2-min drain meant
# an hourly chain spike that briefly scaled to 4 pods sat at 4 pods
# for ~16 min after the spike cleared. Bring it down to 2 min
# stabilization + 100%-per-60s drain so the steady state is 1 pod
# within ~3 min of any spike ending.
stabilizationWindowSeconds: 120
policies: policies:
- type: Pods - type: Percent
value: 1 value: 100
periodSeconds: 120 periodSeconds: 60
--- ---
apiVersion: autoscaling/v2 apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler kind: HorizontalPodAutoscaler
@ -115,14 +124,23 @@ spec:
averageValue: 400m averageValue: 400m
behavior: behavior:
scaleUp: scaleUp:
# Genuine surges (a stuck queue, multiple cycles draining together)
# still bring up replicas within ~30s. Most hourly chain steps
# finish in 2-3 min on a single pod, so transient single-pod
# spikes mostly resolve before HPA notices.
stabilizationWindowSeconds: 30 stabilizationWindowSeconds: 30
policies: policies:
- type: Pods - type: Pods
value: 1 value: 1
periodSeconds: 30 periodSeconds: 30
scaleDown: scaleDown:
stabilizationWindowSeconds: 600 # The previous 10-minute stabilization + 1-pod-per-2-min drain meant
# an hourly chain spike that briefly scaled to 4 pods sat at 4 pods
# for ~16 min after the spike cleared. Bring it down to 2 min
# stabilization + 100%-per-60s drain so the steady state is 1 pod
# within ~3 min of any spike ending.
stabilizationWindowSeconds: 120
policies: policies:
- type: Pods - type: Percent
value: 1 value: 100
periodSeconds: 120 periodSeconds: 60