Root cause of the deadlock prod hit today:
- prop-backfill's podAntiAffinity matched `app: prop`, which applies to
backfill pods themselves as well as hot pods.
- The hot Deployment runs maxSurge=1/maxUnavailable=1 with HPA min 2 /
max 5; when HPA was pegged at max and a rollout began, the old
ReplicaSet's requiredDuringScheduling/IgnoredDuringExecution rule
stayed pinned on every already-running hot pod.
- backfill's rule then excluded new hot pods from every physical host
that had an app=prop pod on it, which was all of them, and the
rollout couldn't progress because no old hot pod could come down
until a new one went Ready first.
Narrowing the selector to `{app: prop, tier: hot}` means:
- backfill never constrains itself (it carries tier=backfill).
- Hot pods carry no anti-affinity of their own, so HPA can continue
past the 4-physical-host count by co-locating when it has to.
- Backfill still lands on whichever physical host hot pods haven't
filled up, which is the original intent.
Removes the OTLP span exporter (tracing-opentelemetry, opentelemetry,
opentelemetry_sdk, opentelemetry-otlp) and its transitive tonic/prost/
h2 tree — 56 crates off the build graph, ~3-5 min off a cold build on
the forgejo runner.
The fmt layer (JSON to stdout via tracing-subscriber) is unchanged;
structured logs still reach the Loki collector through kubectl logs,
and /metrics on :9100 still serves Prometheus. Traces were a nice-to-
have that nothing on call was actually reading.
telemetry::init keeps its signature (service_name + TelemetryGuard)
so re-adding OTLP later is a single-module change. k8s manifests drop
the now-unused OTEL_EXPORTER_OTLP_ENDPOINT env var.
Adds HorizontalPodAutoscaler resources for the three load-responsive
Deployments:
- prop (web): min 2, max 5, averageValue 800m CPU
- prop-grid-rs: min 1, max 4, averageValue 400m
- hrrr-point-rs: min 1, max 4, averageValue 400m
- prop-backfill is intentionally unmanaged (queue-driven, not CPU-bound)
The scale-up policy reacts in 30 s with up-to-2-pods or +100 % per
window; scale-down stabilises over 10 min so the hourly propagation
chain doesn't churn replicas right after the :05 cron.
Removes `spec.replicas` from the HPA-managed Deployments so Flux's
periodic reconcile stops fighting HPA's live scaling decisions.
Drops `podAntiAffinity` from prop, prop-grid-rs, and hrrr-point-rs
so HPA can actually reach its maxReplicas when one physical host has
capacity for two pods; the scheduler picks whatever fits.
Updates runbook + project memory to drop the `talos5` hostname —
Postgres is now addressed strictly by IP (10.0.15.30).
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
podAntiAffinity on app=prop already keeps the backfill pod on its own
physical host, so the dedicated nodeSelector + taint were redundant.
Removing them lets talos4 be used as a normal worker for anything else
the cluster wants to place there. The live taint and label on talos4
were cleared alongside this commit.
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the
fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm
(Req exponential backoff up to 32s) piled concurrent sleeps on top of
live workers and tripped Bandit's acceptor + Oban's leader heartbeat.
- Raise default interval to 1500ms (~2.7 req/sec fleet).
- Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps).
- prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the
release CLI round-trips through distributed Erlang and 15s wasn't
enough under load, restart-looping every ~10 min.
The exec probe runs `/app/bin/microwaveprop rpc ':ok'` which spawns a
release CLI and round-trips through distributed Erlang. Under an IEM
429 retry storm (WeatherFetchWorker sleeps 1-16 s between attempts),
the BEAM scheduler stays responsive but probe latency spikes past 5 s
and the kubelet SIGKILLs the pod every ~10 min. 15 s is still well
below the 30 s period so the probe still catches a genuinely wedged
VM inside one cycle.
Every prop pod restart was dumping a pair of ArgumentError stack
traces from grpcbox_channel / grpcbox_subchannel terminate/3:
:ets.select(:gproc, …)
* 1st argument: the table identifier does not refer to an
existing ETS table
This is a known grpcbox shutdown race: the OTLP/gRPC transport's
terminate callback tries to look itself up in gproc's ETS just
after gproc's supervisor has already torn the table down. Noisy,
not harmful — but one ArgumentError per restart × N pods adds up.
The OTLP/HTTP transport (port 4318 on the same otel-collector
Service) has no gproc/ETS channel machinery, so shutdown is
quiet. Spans are identical — our collector already accepts both.
- `config/runtime.exs`: default `otlp_protocol: :http_protobuf`,
with an `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` escape hatch.
- `k8s/deployment.yaml`: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
:4318 instead of :4317. The Rust workers stay on :4317 — their
native opentelemetry-otlp transport doesn't share the grpcbox
bug.
