Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
Root cause of the deadlock prod hit today:
- prop-backfill's podAntiAffinity matched `app: prop`, which applies to
backfill pods themselves as well as hot pods.
- The hot Deployment runs maxSurge=1/maxUnavailable=1 with HPA min 2 /
max 5; when HPA was pegged at max and a rollout began, the old
ReplicaSet's requiredDuringScheduling/IgnoredDuringExecution rule
stayed pinned on every already-running hot pod.
- backfill's rule then excluded new hot pods from every physical host
that had an app=prop pod on it, which was all of them, and the
rollout couldn't progress because no old hot pod could come down
until a new one went Ready first.
Narrowing the selector to `{app: prop, tier: hot}` means:
- backfill never constrains itself (it carries tier=backfill).
- Hot pods carry no anti-affinity of their own, so HPA can continue
past the 4-physical-host count by co-locating when it has to.
- Backfill still lands on whichever physical host hot pods haven't
filled up, which is the original intent.
Removes the OTLP span exporter (tracing-opentelemetry, opentelemetry,
opentelemetry_sdk, opentelemetry-otlp) and its transitive tonic/prost/
h2 tree — 56 crates off the build graph, ~3-5 min off a cold build on
the forgejo runner.
The fmt layer (JSON to stdout via tracing-subscriber) is unchanged;
structured logs still reach the Loki collector through kubectl logs,
and /metrics on :9100 still serves Prometheus. Traces were a nice-to-
have that nothing on call was actually reading.
telemetry::init keeps its signature (service_name + TelemetryGuard)
so re-adding OTLP later is a single-module change. k8s manifests drop
the now-unused OTEL_EXPORTER_OTLP_ENDPOINT env var.
Adds HorizontalPodAutoscaler resources for the three load-responsive
Deployments:
- prop (web): min 2, max 5, averageValue 800m CPU
- prop-grid-rs: min 1, max 4, averageValue 400m
- hrrr-point-rs: min 1, max 4, averageValue 400m
- prop-backfill is intentionally unmanaged (queue-driven, not CPU-bound)
The scale-up policy reacts in 30 s with up-to-2-pods or +100 % per
window; scale-down stabilises over 10 min so the hourly propagation
chain doesn't churn replicas right after the :05 cron.
Removes `spec.replicas` from the HPA-managed Deployments so Flux's
periodic reconcile stops fighting HPA's live scaling decisions.
Drops `podAntiAffinity` from prop, prop-grid-rs, and hrrr-point-rs
so HPA can actually reach its maxReplicas when one physical host has
capacity for two pods; the scheduler picks whatever fits.
Updates runbook + project memory to drop the `talos5` hostname —
Postgres is now addressed strictly by IP (10.0.15.30).
