Ecto auto-instrumentation created one span per query, so Oban jobs
running thousands of DB ops produced traces with 8k+ child spans.
Tempo's compactor couldn't ingest them — it stalled long enough to
miss heartbeats, got auto-forgotten from the ring, and crash-looped.
The Elixir grpcbox exporter then blew up trying to reconnect.
Stops the cascade: OTLP endpoint removed from prop and prop-backfill
Deployments, opentelemetry_ecto removed from deps, Phoenix/Oban/Bandit
auto-instrumentation retained. Rust workers still export to Tempo.
Shared NFS idx cache at /data/hrrr_idx lets all prop-grid-rs and
hrrr-point-rs replicas deduplicate redundant NOAA S3 idx fetches.
Opt-in via HRRR_IDX_CACHE_DIR; falls back to in-memory-only when unset.
Also scales prop-grid-rs to 3 replicas (no talos5 pinning),
hrrr-point-rs to 2 replicas, and drops hot pod replicas 4→3 so the
physical-host anti-affinity still fits with talos5 cordoned off for
Postgres.
Adds the OTel deps (opentelemetry + exporter + phoenix / ecto / oban /
bandit auto-instrumentation helpers) and attaches them in
Application.start/2 so the existing Phoenix, Ecto, Oban, and Bandit
telemetry events flow as OTLP spans without any call-site changes.
Exporter config is gated on OTEL_EXPORTER_OTLP_ENDPOINT in
config/runtime.exs — set in the k8s manifests to the cluster
collector (otel-collector.observability.svc.cluster.local:4317).
When unset we switch traces_exporter to :none so nothing is shipped;
dev/test stays quiet.
Resource attributes tag spans with service.name=microwaveprop and
service.namespace=prop, matching the Rust workers' attribute shape
so Tempo can group the full hourly chain across both languages.
Both the main prop deployment and the backfill deployment get the
env; backfill is still a full BEAM node running enrichment workers,
so its Oban/Ecto spans are worth seeing too.
Both the hot (`prop`) Deployment and the `prop-backfill` Deployment
labelled their pods with `app: prop` so both joined the same libcluster
Erlang cluster. The `prop` Service selector was `app: prop` too — which
meant the backfill pod (PHX_SERVER=false, no listener on :5000) was in
the Service endpoint pool.
With sessionAffinity: ClientIP, Cloudflared's source IP got hashed to
the backfill endpoint after a backfill pod roll and every request
from that client returned 502 (connection refused on :5000).
Fix: add `tier: hot` to hot-pod template labels; narrow the Service
selector to `{app: prop, tier: hot}`. Backfill keeps `tier: backfill`
and is cleanly excluded. libcluster selector (still `app: prop`) is
unchanged so backfill stays in the Erlang cluster for Oban queue
leader election and Oban.Pro's Smart engine.
talos5 is a NUC7i5 with 32GB RAM and a Samsung 980 Pro NVMe. Adding
a 4th hot-path replica lets the anti-affinity rule place one pod
per physical host across node1/2/3/5 and gives the hourly f00-f18
grid chain 4-way parallel fan-out, with the NVMe-backed pod
handling its share of grib extraction.
