prop/k8s
Graham McIntire 5e9513bf02
ops(prop): aggressive HPA scale-down + CLAUDE.md guardrails
HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.

Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.

CLAUDE.md updates so this doesn't bite again:

- Document `cargo clippy --all-targets -- -D warnings` as part of
  the local Rust precommit ritual. The HRDPS commits in this branch
  passed `cargo build` locally but failed CI on lints (manual_range_
  contains, iter_kv_map). Adding the recipe under Commands plus a
  Rust subsection in Key Conventions so future Rust touches don't
  ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
  utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
  test (fix in 2768fc68) was a textbook example: 50% flake rate
  driven entirely by which minute of the hour the test ran in.
2026-04-29 17:44:17 -05:00
..
deployment-backfill.yaml remove obsolete flux manifest and imagepolicy annotations 2026-04-29 11:00:05 -05:00
deployment-grid-rs.yaml remove obsolete flux manifest and imagepolicy annotations 2026-04-29 11:00:05 -05:00
deployment-hrrr-point-rs.yaml remove obsolete flux manifest and imagepolicy annotations 2026-04-29 11:00:05 -05:00
deployment.yaml ops(prop): bump preStop sleep to 25s for cloudflared connection drain 2026-04-29 14:17:40 -05:00
hpa.yaml ops(prop): aggressive HPA scale-down + CLAUDE.md guardrails 2026-04-29 17:44:17 -05:00
kustomization.yaml infra(k8s): load-based HPAs, drop anti-affinity, retire talos5 2026-04-23 18:43:18 -05:00
metrics-service.yaml feat(k8s): NodePort service for external Prometheus scraping 2026-04-18 16:51:46 -05:00
namespace.yaml Add k8s manifests and Forgejo CI for cluster deployment 2026-04-06 09:25:24 -05:00
rbac.yaml Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
service.yaml fix(k8s): tier label isolates hot pods from backfill endpoints 2026-04-19 16:58:13 -05:00