fix(k8s): raise prop-backfill probe timeout to 15s

The exec probe runs `/app/bin/microwaveprop rpc ':ok'` which spawns a
release CLI and round-trips through distributed Erlang. Under an IEM
429 retry storm (WeatherFetchWorker sleeps 1-16 s between attempts),
the BEAM scheduler stays responsive but probe latency spikes past 5 s
and the kubelet SIGKILLs the pod every ~10 min. 15 s is still well
below the 30 s period so the probe still catches a genuinely wedged
VM inside one cycle.
This commit is contained in:
Graham McIntire 2026-04-22 13:56:10 -05:00
parent bfddb898bf
commit 04146af1fa
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -73,20 +73,25 @@ spec:
- secretRef:
name: prop-secrets
# No HTTP probes (no web server). Use exec probe against the
# running BEAM to confirm the VM is responsive.
# running BEAM to confirm the VM is responsive. timeoutSeconds
# is generous because `rpc ':ok'` has to spawn a release CLI
# and round-trip through distributed Erlang; during an IEM
# 429 retry storm (WeatherFetchWorker sleeping 1-16 s between
# attempts) the scheduler is responsive but slow, and a 5 s
# probe timeout was tripping failureThreshold every ~10 min.
startupProbe:
exec:
command: ["/bin/sh", "-c", "/app/bin/microwaveprop rpc ':ok'"]
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 24
timeoutSeconds: 5
timeoutSeconds: 15
livenessProbe:
exec:
command: ["/bin/sh", "-c", "/app/bin/microwaveprop rpc ':ok'"]
periodSeconds: 30
failureThreshold: 3
timeoutSeconds: 5
timeoutSeconds: 15
resources:
# 8GB T620 box — leave ~3GB for kubelet/containerd/OS.
requests: