fix(k8s): raise prop-backfill probe timeout to 15s

The exec probe runs `/app/bin/microwaveprop rpc ':ok'` which spawns a
release CLI and round-trips through distributed Erlang. Under an IEM
429 retry storm (WeatherFetchWorker sleeps 1-16 s between attempts),
the BEAM scheduler stays responsive but probe latency spikes past 5 s
and the kubelet SIGKILLs the pod every ~10 min. 15 s is still well
below the 30 s period so the probe still catches a genuinely wedged
VM inside one cycle.
This commit is contained in:
Graham McIntire 2026-04-22 13:56:10 -05:00
parent bfddb898bf
commit 04146af1fa
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -73,20 +73,25 @@ spec:
- secretRef: - secretRef:
name: prop-secrets name: prop-secrets
# No HTTP probes (no web server). Use exec probe against the # No HTTP probes (no web server). Use exec probe against the
# running BEAM to confirm the VM is responsive. # running BEAM to confirm the VM is responsive. timeoutSeconds
# is generous because `rpc ':ok'` has to spawn a release CLI
# and round-trip through distributed Erlang; during an IEM
# 429 retry storm (WeatherFetchWorker sleeping 1-16 s between
# attempts) the scheduler is responsive but slow, and a 5 s
# probe timeout was tripping failureThreshold every ~10 min.
startupProbe: startupProbe:
exec: exec:
command: ["/bin/sh", "-c", "/app/bin/microwaveprop rpc ':ok'"] command: ["/bin/sh", "-c", "/app/bin/microwaveprop rpc ':ok'"]
initialDelaySeconds: 10 initialDelaySeconds: 10
periodSeconds: 5 periodSeconds: 5
failureThreshold: 24 failureThreshold: 24
timeoutSeconds: 5 timeoutSeconds: 15
livenessProbe: livenessProbe:
exec: exec:
command: ["/bin/sh", "-c", "/app/bin/microwaveprop rpc ':ok'"] command: ["/bin/sh", "-c", "/app/bin/microwaveprop rpc ':ok'"]
periodSeconds: 30 periodSeconds: 30
failureThreshold: 3 failureThreshold: 3
timeoutSeconds: 5 timeoutSeconds: 15
resources: resources:
# 8GB T620 box — leave ~3GB for kubelet/containerd/OS. # 8GB T620 box — leave ~3GB for kubelet/containerd/OS.
requests: requests: