FluxCD
8324d019ca
chore: update prop image to git.mcintire.me/graham/prop:main-1776970997-e7fb34e [skip ci]
2026-04-23 19:07:05 +00:00
FluxCD
ed21758557
chore: update prop image to git.mcintire.me/graham/prop:main-1776968936-764643b [skip ci]
2026-04-23 18:31:46 +00:00
FluxCD
8c0865482c
chore: update prop image to git.mcintire.me/graham/prop:main-1776967519-cfb0c51 [skip ci]
2026-04-23 18:07:33 +00:00
FluxCD
a9d13d8eb9
chore: update prop image to git.mcintire.me/graham/prop:main-1776967071-063e9e3 [skip ci]
2026-04-23 18:00:31 +00:00
FluxCD
e88375e50b
chore: update prop image to git.mcintire.me/graham/prop:main-1776965182-47ba8d3 [skip ci]
2026-04-23 17:28:47 +00:00
FluxCD
da66a613ef
chore: update prop image to git.mcintire.me/graham/prop:main-1776960355-b2ff279 [skip ci]
2026-04-23 16:10:27 +00:00
FluxCD
f0f4ce6b33
chore: update prop image to git.mcintire.me/graham/prop:main-1776959622-66fef4f [skip ci]
2026-04-23 15:56:21 +00:00
FluxCD
e4ed923986
chore: update prop image to git.mcintire.me/graham/prop:main-1776898028-004f302 [skip ci]
2026-04-22 22:54:04 +00:00
FluxCD
a5b080f0af
chore: update prop image to git.mcintire.me/graham/prop:main-1776894891-47a3e1d [skip ci]
2026-04-22 21:58:48 +00:00
47a3e1dbde
chore(k8s): drop backfill nodeSelector/toleration, free talos4
...
podAntiAffinity on app=prop already keeps the backfill pod on its own
physical host, so the dedicated nodeSelector + taint were redundant.
Removing them lets talos4 be used as a normal worker for anything else
the cluster wants to place there. The live taint and label on talos4
were cleared alongside this commit.
2026-04-22 16:54:31 -05:00
024db2a5b4
perf(fleet): reduce IEM 429 storm and raise probe timeouts
...
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the
fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm
(Req exponential backoff up to 32s) piled concurrent sleeps on top of
live workers and tripped Bandit's acceptor + Oban's leader heartbeat.
- Raise default interval to 1500ms (~2.7 req/sec fleet).
- Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps).
- prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the
release CLI round-trips through distributed Erlang and 15s wasn't
enough under load, restart-looping every ~10 min.
2026-04-22 16:50:58 -05:00
FluxCD
d6d50f7729
chore: update prop image to git.mcintire.me/graham/prop:main-1776884183-04146af [skip ci]
2026-04-22 18:58:47 +00:00
04146af1fa
fix(k8s): raise prop-backfill probe timeout to 15s
...
The exec probe runs `/app/bin/microwaveprop rpc ':ok'` which spawns a
release CLI and round-trips through distributed Erlang. Under an IEM
429 retry storm (WeatherFetchWorker sleeps 1-16 s between attempts),
the BEAM scheduler stays responsive but probe latency spikes past 5 s
and the kubelet SIGKILLs the pod every ~10 min. 15 s is still well
below the 30 s period so the probe still catches a genuinely wedged
VM inside one cycle.
2026-04-22 13:56:10 -05:00
FluxCD
5b41ad00e7
chore: update prop image to git.mcintire.me/graham/prop:main-1776869401-e8c2529 [skip ci]
2026-04-22 14:53:39 +00:00
FluxCD
b92c1ea68b
chore: update prop image to git.mcintire.me/graham/prop:main-1776866277-3ec976c [skip ci]
2026-04-22 14:03:26 +00:00
FluxCD
c4a6b8ed8b
chore: update prop image to git.mcintire.me/graham/prop:main-1776864939-c192d6f [skip ci]
2026-04-22 13:43:20 +00:00
FluxCD
84f4f93b87
chore: update prop image to git.mcintire.me/graham/prop:main-1776812585-ee7809e [skip ci]
2026-04-21 23:07:11 +00:00
FluxCD
39cc5292a9
chore: update prop image to git.mcintire.me/graham/prop:main-1776812100-c46e89a [skip ci]
2026-04-21 22:57:43 +00:00
FluxCD
4d5b5f1aee
chore: update prop image to git.mcintire.me/graham/prop:main-1776811628-04aa198 [skip ci]
2026-04-21 22:50:40 +00:00
FluxCD
a232670139
chore: update prop image to git.mcintire.me/graham/prop:main-1776810886-4d07f89 [skip ci]
2026-04-21 22:37:16 +00:00
FluxCD
3217b011a4
chore: update prop image to git.mcintire.me/graham/prop:main-1776810088-d3ca2e5 [skip ci]
2026-04-21 22:23:56 +00:00
FluxCD
1c1a253fbe
chore: update prop image to git.mcintire.me/graham/prop:main-1776809787-ff950f9 [skip ci]
2026-04-21 22:18:53 +00:00
FluxCD
944c2dc1d4
chore: update prop image to git.mcintire.me/graham/prop:main-1776809397-49d0594 [skip ci]
2026-04-21 22:12:43 +00:00
FluxCD
7e53f6f890
chore: update prop image to git.mcintire.me/graham/prop:main-1776809188-e9a3862 [skip ci]
2026-04-21 22:10:41 +00:00
FluxCD
a0608dfa2f
chore: update prop image to git.mcintire.me/graham/prop:main-1776807571-26ef50c [skip ci]
2026-04-21 21:42:32 +00:00
FluxCD
e73f097b75
chore: update prop image to git.mcintire.me/graham/prop:main-1776807271-8ec5d89 [skip ci]
2026-04-21 21:38:29 +00:00
FluxCD
6c57caa902
chore: update prop image to git.mcintire.me/graham/prop:main-1776806062-c56bb91 [skip ci]
2026-04-21 21:28:25 +00:00
FluxCD
02bf34db9e
chore: update prop image to git.mcintire.me/graham/prop:main-1776798788-48d30f1 [skip ci]
2026-04-21 19:16:25 +00:00
FluxCD
e9b804f628
chore: update prop image to git.mcintire.me/graham/prop:main-1776786279-38ef716 [skip ci]
2026-04-21 15:47:20 +00:00
FluxCD
6c977e250d
chore: update prop image to git.mcintire.me/graham/prop:main-1776785420-d61fbd3 [skip ci]
2026-04-21 15:34:05 +00:00
FluxCD
2f5b37f7c5
chore: update prop image to git.mcintire.me/graham/prop:main-1776783205-7b78a25 [skip ci]
2026-04-21 14:55:46 +00:00
FluxCD
e5991d324a
chore: update prop image to git.mcintire.me/graham/prop:main-1776782574-71c134d [skip ci]
2026-04-21 14:46:43 +00:00
FluxCD
61da51c03f
chore: update prop image to git.mcintire.me/graham/prop:main-1776781734-f4f44b5 [skip ci]
2026-04-21 14:31:32 +00:00
FluxCD
360a0e32c1
chore: update prop image to git.mcintire.me/graham/prop:main-1776781470-33be6d0 [skip ci]
2026-04-21 14:27:27 +00:00
FluxCD
9ced0eab87
chore: update prop image to git.mcintire.me/graham/prop:main-1776780470-fbbb8b2 [skip ci]
2026-04-21 14:16:16 +00:00
FluxCD
cd2962687a
chore: update prop image to git.mcintire.me/graham/prop:main-1776777603-cc9220b [skip ci]
2026-04-21 13:23:57 +00:00
FluxCD
cb9d9fb30b
chore: update prop image to git.mcintire.me/graham/prop:main-1776752742-c96bdca [skip ci]
2026-04-21 06:35:16 +00:00
FluxCD
e3d7f5b2c2
chore: update prop image to git.mcintire.me/graham/prop:main-1776722189-30525a1 [skip ci]
2026-04-20 21:59:07 +00:00
FluxCD
ff241afa6e
chore: update prop image to git.mcintire.me/graham/prop:main-1776721927-aebf391 [skip ci]
2026-04-20 21:55:01 +00:00
FluxCD
9815bd691e
chore: update prop image to git.mcintire.me/graham/prop:main-1776721607-e641fce [skip ci]
2026-04-20 21:49:58 +00:00
FluxCD
796fb22d4b
chore: update prop image to git.mcintire.me/graham/prop:main-1776721085-ee732d0 [skip ci]
2026-04-20 21:45:54 +00:00
FluxCD
00101ca928
chore: update prop image to git.mcintire.me/graham/prop:main-1776720629-bd3b114 [skip ci]
2026-04-20 21:34:50 +00:00
FluxCD
77b54a1642
chore: update prop image to git.mcintire.me/graham/prop:main-1776719536-9bf9634 [skip ci]
2026-04-20 21:14:47 +00:00
FluxCD
6463b7261f
chore: update prop image to git.mcintire.me/graham/prop:main-1776715484-c78cb97 [skip ci]
2026-04-20 20:07:30 +00:00
FluxCD
c10cf176e7
chore: update prop image to git.mcintire.me/graham/prop:main-1776714901-52ce44d [skip ci]
2026-04-20 19:57:23 +00:00
FluxCD
7244a59a21
chore: update prop image to git.mcintire.me/graham/prop:main-1776712929-ddedd24 [skip ci]
2026-04-20 19:25:01 +00:00
FluxCD
444de00ef7
chore: update prop image to git.mcintire.me/graham/prop:main-1776712357-a7bea4a [skip ci]
2026-04-20 19:14:53 +00:00
FluxCD
26640b1204
chore: update prop image to git.mcintire.me/graham/prop:main-1776711181-608fd32 [skip ci]
2026-04-20 18:55:45 +00:00
FluxCD
0c1d90ef58
chore: update prop image to git.mcintire.me/graham/prop:main-1776710724-f3fab30 [skip ci]
2026-04-20 18:47:39 +00:00
FluxCD
818b270a0a
chore: update prop image to git.mcintire.me/graham/prop:main-1776710473-1a1ba46 [skip ci]
2026-04-20 18:44:39 +00:00
