The 'Fetching HRRR atmospheric data' spinner stayed up forever for
contacts stuck in :queued without a matching hrrr_fetch_tasks row.
Ways a contact lands in that state:
- HrrrPointEnqueuer.enqueue/1 raises, the rescue swallows the error
and returns {:ok, 0}, but the caller still marked the contact
:queued before it noticed the failure.
- Rust worker marks a (valid_time) task :done with a points array
that never included this contact's point, so no profile is ever
written for it.
- hrrr_fetch_tasks row is cleaned up by retention before the contact
page loads.
Drop the hrrr_status != :queued short-circuit and always call the
enqueuer when the profile is nil. The ON CONFLICT clause in
HrrrPointEnqueuer.enqueue/1 unions points into the existing row,
resets :done/:failed rows back to :queued with attempt=0, and no-ops
when the point is already present — so re-entering on every page view
is cheap and self-healing.
Clicks on the map were hanging for 1-3s, triggering the Phoenix topbar
progress bar visible to the user. Three hot-path improvements:
1. ScoresFile.read_point now uses a two-range :file.pread/2 — reads the
33-byte header + one cell byte (~60 bytes) instead of the full ~93 KB
file. point_forecast walks 19 forecast hours per click, so the old
full-read path was the dominant cost on cold cache.
2. ProfilesFile.read/1 and list_valid_times/0 cached via
Microwaveprop.Cache (5s TTL). Decoded profile maps are ~10 MB each
(92k cells); timeline scrub that re-clicks the same valid_time
within seconds now hits ETS instead of re-gunzipping + decoding.
Cache keys include base_dir so test setup that swaps dirs doesn't
see stale entries. Writes + prune + retain_window invalidate.
3. Endpoint log_level now filters /metrics the same way it already
filters /health — Prometheus scrapes every 5s and was producing
visible log spam.
Pod-local ETS is right for this workload (per-click read, tiny working
set); Valkey / shared memstore would not help here since each pod needs
its own fast-path lookup. File a follow-up if cross-pod score lookups
ever show up in flame graphs.
Three small wins from a telemetry audit:
1. Propagation.scores_at/3 wrapped every call in Instrument.span, firing
two handler dispatches that dominated the ~10µs ETS lookup on cache
hits. The map's LiveView fires this on every pan + point-click, so
hot-path latency was mostly telemetry. Span now wraps only the miss
branch (where disk IO makes the duration signal meaningful); hit path
still emits the cheap hit/miss counter the cache-ratio panel reads.
2. Oban queue-depth poller: 10s → 30s. Every replica independently
GROUP-BYs oban_jobs, so each extra replica paid ~3 redundant queries
per minute for a gauge that moves on the hour-scale anyway.
3. Dropped summary(phoenix.endpoint.start.system_time): summarizing a
wall-clock timestamp produces no useful aggregate — stop.duration
below it is the meaningful signal.
Contacts lacking pos2 had terrain/radar/mechanism stuck on :pending, so
BackfillEnqueueWorker re-scanned them every cron cycle. Mark these
statuses :unavailable so the enqueue filter excludes them; a later pos2
edit resets them back to :pending via Radio.reset_enrichment_statuses/2.
BackfillEnqueueWorker re-enqueued every :queued contact each cycle, and
RadarFrameWorker had no unique clause, so the same 5-min NEXRAD frame
got scheduled repeatedly with overlapping contact lists. Prod had 9,134
duplicate available jobs across 1,061 distinct frames — 89% waste.
Two coupled changes so contacts aren't stranded by the dedup:
1. Worker now queries eligible contacts by the frame's 5-min bucket
instead of the args' contact_ids list. A contact that flips to
:queued between the enqueue and the job running gets picked up.
2. unique: [frame_ts] on states [available, scheduled, retryable].
:executing intentionally excluded — a contact that flips to :queued
mid-execution can still enqueue a follow-up instead of getting
stuck behind a running job that missed it.
HRRR publishes ~2h after the cycle hour, so the hourly seeder uses
run_time = now - 2h. That puts the f00 analysis file's valid_time
right at the 2h prune cutoff — the file was being deleted within
minutes of the Rust analysis step writing it, leaving /weather and
/map point-detail with no data.
A 3h cutoff keeps the current analysis alive until the next hourly
run supersedes it; forecast files (valid_time >> cutoff) are
unaffected.
Second row in the Rust worker table pulls from hrrr_fetch_tasks so
per-QSO HRRR backfill progress is visible alongside prop-grid-rs.
Same columns (Running / Queued / Retrying / Done 1h), 2s cache, same
refresh debounce as the grid_tasks panel.
Ecto auto-instrumentation created one span per query, so Oban jobs
running thousands of DB ops produced traces with 8k+ child spans.
Tempo's compactor couldn't ingest them — it stalled long enough to
miss heartbeats, got auto-forgotten from the ring, and crash-looped.
The Elixir grpcbox exporter then blew up trying to reconnect.
Stops the cascade: OTLP endpoint removed from prop and prop-backfill
Deployments, opentelemetry_ecto removed from deps, Phoenix/Oban/Bandit
auto-instrumentation retained. Rust workers still export to Tempo.
Shared NFS idx cache at /data/hrrr_idx lets all prop-grid-rs and
hrrr-point-rs replicas deduplicate redundant NOAA S3 idx fetches.
Opt-in via HRRR_IDX_CACHE_DIR; falls back to in-memory-only when unset.
Also scales prop-grid-rs to 3 replicas (no talos5 pinning),
hrrr-point-rs to 2 replicas, and drops hot pod replicas 4→3 so the
physical-host anti-affinity still fits with talos5 cordoned off for
Postgres.
CommonVolumeRadarWorker was enqueuing one job per contact — every
job fetched + decoded the ~5 MB n0q PNG for its own 5-min frame.
At current backlog depth that's 17,588 contacts across just 1,747
distinct 5-min frames, i.e. ~10 contacts per frame paying for the
same PNG decode over and over.
New RadarFrameWorker takes a batch of contact_ids sharing one frame,
fetches + decodes the frame ONCE, then walks the in-memory pixel
buffer per contact. build_radar_jobs/1 in ContactWeatherEnqueueWorker
now groups the input contacts by their rounded 5-min timestamp and
emits one RadarFrameWorker job per frame instead of one
CommonVolumeRadarWorker per contact.
process_frame/4 is public so the same code path is used both by
perform/1 (production fetch) and tests (pre-decoded pixel buffer);
keeps the happy-path test hermetic without hand-crafting a PNG.
CommonVolumeRadarWorker stays in the tree — still used from the
single-contact submit path where batching is pointless and the
aggregate_stats/5 pure helper is a dependency of the new batched
worker.
Expected impact on the backfill: ~10x fewer fetch + decode cycles,
so the 17k queued-contact backlog drains in minutes instead of ~50.
Adds the OTel deps (opentelemetry + exporter + phoenix / ecto / oban /
bandit auto-instrumentation helpers) and attaches them in
Application.start/2 so the existing Phoenix, Ecto, Oban, and Bandit
telemetry events flow as OTLP spans without any call-site changes.
Exporter config is gated on OTEL_EXPORTER_OTLP_ENDPOINT in
config/runtime.exs — set in the k8s manifests to the cluster
collector (otel-collector.observability.svc.cluster.local:4317).
When unset we switch traces_exporter to :none so nothing is shipped;
dev/test stays quiet.
Resource attributes tag spans with service.name=microwaveprop and
service.namespace=prop, matching the Rust workers' attribute shape
so Tempo can group the full hourly chain across both languages.
Both the main prop deployment and the backfill deployment get the
env; backfill is still a full BEAM node running enrichment workers,
so its Oban/Ecto spans are worth seeing too.
New telemetry module wires tracing-subscriber to both a local JSON
fmt layer (keeps kubectl-logs output identical) and an
OpenTelemetry OTLP/gRPC exporter, activated only when
OTEL_EXPORTER_OTLP_ENDPOINT is set. The returned TelemetryGuard
holds the SdkTracerProvider until process shutdown so queued spans
flush before exit.
Both bin targets (worker, hrrr_point_worker) now call
telemetry::init(service_name) at startup; service_name becomes the
OTel service.name attribute so Tempo groups spans per binary.
Tracing instrumentation on the three main work units:
- pipeline::run_chain_step (forecast f01..f18)
- pipeline::run_analysis_step (analysis f00)
- hrrr_points::process_batch (per-QSO point drain)
k8s manifests set OTEL_EXPORTER_OTLP_ENDPOINT to the cluster
collector at otel-collector.observability.svc.cluster.local:4317.
Backend wiring lives in the vntx-infra repo.
The commercial schema uses uuid primary keys everywhere in this project
(all schemas: `@primary_key {:id, :binary_id, autogenerate: true}`), but
the Rust port bound SELECTs against commercial_links.id and
commercial_samples.link_id as i64. The f00 analysis step failed on first
contact with:
error occurred while decoding column 0: mismatched types; Rust type
`i64` (as SQL type `INT8`) is not compatible with SQL type `UUID`
Switched LinkLookupEntry.link_id and fetch_per_link_degradation's
parameter to uuid::Uuid, updated the sqlx::query_as tuple type, and
patched the test helper to build distinct UUIDs from a byte tag so the
no-range / aggregation / rounding tests still exercise the same
identity semantics without needing a real DB column.
Three fixes after hrrr-point-rs restarts on the first live drain:
1. Type mismatch (hard error): the `profile` column on `hrrr_profiles`
is `jsonb[]` (one element per pressure level), but the Rust worker
was binding a single jsonb array-of-objects cast as `::jsonb`. Every
insert failed with `column "profile" is of type jsonb[] but
expression is of type jsonb`. Switched to
`Vec<sqlx::types::Json<Value>>`, dropped the explicit `::jsonb`
cast, and enabled sqlx's `json` feature so the encoder maps the
array into `_jsonb` correctly.
2. Silent zero-inserts: four consecutive 2019-09-22 batches completed
with `profiles_inserted: 0` and no diagnostic. Most likely the
upstream archive doesn't keep cycles that old, but without a log
it looks identical to a snap-mismatch bug. Added a WARN when both
surface and pressure grids come back empty so fetch-miss vs.
projection-miss is distinguishable.
3. OOMKilled: the pod took four OOM restarts in ten minutes at 1 Gi.
A single CONUS decode holds the ~40 MB blob, the ~200 MB wgrib2
working set, AND the full 92k-cell merged map until all requested
points drain. 1 Gi has no headroom. Bumped to 2 Gi, matching the
rest of the per-container budgets in this namespace.
Two concurrent forecast tasks plus the NFS score-file write cache (23
files x ~2 MB) plus the analysis step's wgrib2 peak repeatedly crossed
the 3 Gi cgroup limit. Single-lane parallelism per pod keeps steady RSS
under 2 Gi; with 2 replicas that still gives 2 concurrent tasks
cluster-wide.
PROP_GRID_RS_PG_CONNS follows the parallelism+2 formula, dropping from
4 to 3.
Mirrors the verbose wait-for-docker block from build.yaml so future "failed
to connect to docker" failures print socket permissions, docker-related
env vars, and full `docker info` output instead of a single unhelpful line.
The FROM ${WGRIB2_IMAGE} AS wgrib2-src pattern required docker buildx
to pull a private-registry image mid-build. That pull started failing
consistently after Stream C landed — docker login on the runner
succeeds but buildx's implicit FROM pull can't always reach the same
credentials (buildx instance context vs the host daemon).
Rebuild wgrib2 + NCEPLIBS-g2c from source in a dedicated builder stage,
mirroring the Elixir Dockerfile's self-contained wgrib2-builder. Adds
~5 min per CI run — worth the end of a 3-run CI-failure streak.
The docker buildx build step has failed twice since Stream C added the
hrrr_point_worker binary. Dropping the target/ BuildKit cache mount
eliminates the most likely cause (disk pressure inside a persistent
cache shared between the two bin builds). Full rebuild from scratch
adds ~3 min to the build but the failure mode is worth more than the
cache.
Also dropped the empty unit test scaffolding in hrrr_points.rs; the
module is thin glue over fetcher+decoder which already carry coverage.
Integration tests for process_batch/upsert_profile are gated on a live
DB + HRRR mirror and live in tests/ (when added).
Two bugs surfacing 13 h after Stream A cutover:
1. prop-grid-rs pods OOMKilled 16–18× overnight on the analysis step.
f00's native-level GRIB2 (~530 MB + wgrib2 working set) runs on top
of 2 concurrent forecast tasks, briefly reaching ~2 Gi. 2 Gi limit
was too tight — 3 Gi plus parallelism 3→2 gives the analysis step
room without eliminating forecast-lane headroom.
2. hrrr-point-rs pod is CrashLoopBackOff because its container is
running image main-1776640915-65f7963 (pre-Stream-C) which doesn't
contain the hrrr_point_worker binary. The grid-rs CI run for
commit 4fefb81 failed and no newer image got published. Touch the
Dockerfile to re-trigger the workflow so flux picks up a new tag
with both binaries. No functional Dockerfile change, just a
docstring update so the path-filter kicks.
Forecast + analysis both claim per FOR UPDATE SKIP LOCKED, so
reducing per-pod parallelism doesn't break the chain — the two
replicas cover 4 slots cluster-wide, which still drains f01..f18 in
~5 min.
Phase 3 Stream C Rust side. Completes the HrrrFetchWorker port.
Pipeline:
- db::claim_next_hrrr_task — FOR UPDATE SKIP LOCKED on hrrr_fetch_tasks,
newest valid_time first. Accepts the points JSONB directly.
- hrrr_points::process_batch — fetch surface + pressure GRIB2 once
per task (tokio::try_join), decode via the existing wgrib2 plumbing,
then for each requested point pull the cell and UPSERT INTO
hrrr_profiles (conflict on lat/lon/valid_time).
- db::complete_hrrr_task / fail_hrrr_task — status transitions; Elixir
backfill re-enqueues failed rows on next /30-min scan.
Shipping pieces:
- new bin src/bin/hrrr_point_worker.rs
- new module src/hrrr_points.rs (process_batch, upsert_profile)
- new Cargo [[bin]] entry; Dockerfile builds both binaries in one stage
and ships them in the runtime image so a single CI pipeline covers
the whole cluster
- k8s/deployment-hrrr-point-rs.yaml (1 replica, 1 Gi limit, anti-affinity
against prop-grid-rs so chain + point work don't fight for wgrib2
slots). Uses the same image; command: override picks the right binary.
- kustomization.yaml: include the new deployment so flux applies it
- deployment-grid-rs.yaml: bump readiness initialDelaySeconds 3→15 +
failureThreshold 3→6 so a slow DB connect during startup can't race
the first probe
119 Rust tests green.
