prop-grid-rs already exposed prop_grid_rs_chain_step_duration_seconds
+ tasks_in_flight on :9100 but the comment still pointed at the old
prom server (10.0.15.25). hrrr-point-rs had no metrics listener at all.
- metrics.rs: new POINT_BATCH_DURATION / POINT_BATCHES_TOTAL /
POINTS_PROCESSED_TOTAL series + record_point_batch helper, plus unit
tests serialized via a static Mutex (the prometheus crate's global
registry races otherwise)
- hrrr_point_worker.rs: spawn metrics::serve on METRICS_ADDR (default
0.0.0.0:9100), wrap process_batch in InFlightGuard, record outcome +
per-point counts on every batch
- deployment-hrrr-point-rs.yaml: prometheus.io/{scrape,port,path,job}
annotations, METRICS_ADDR env, ports.metrics, /health readiness probe
- deployment-grid-rs.yaml: refresh stale 10.0.15.25 comment, add
prometheus.io/job for clean dashboard up{} selectors
Also fix Microwaveprop.PromExTest: `assert plugins != []` triggered
Elixir 1.19 type-checker warning ("non_empty_list != empty_list always
true"); replaced with membership assertions for the InstrumentPlugin +
Plugins.Beam.
The image tags here are placeholders — the live tag is set by the
ArgoCD Application's spec.source.kustomize.images field, same as
before. Only the registry hostname and path changed.
Pull secret 'forgejo-registry' in the prop namespace now carries
both git.mcintire.me and codeberg.org auths during the transition.
Removes the OTLP span exporter (tracing-opentelemetry, opentelemetry,
opentelemetry_sdk, opentelemetry-otlp) and its transitive tonic/prost/
h2 tree — 56 crates off the build graph, ~3-5 min off a cold build on
the forgejo runner.
The fmt layer (JSON to stdout via tracing-subscriber) is unchanged;
structured logs still reach the Loki collector through kubectl logs,
and /metrics on :9100 still serves Prometheus. Traces were a nice-to-
have that nothing on call was actually reading.
telemetry::init keeps its signature (service_name + TelemetryGuard)
so re-adding OTLP later is a single-module change. k8s manifests drop
the now-unused OTEL_EXPORTER_OTLP_ENDPOINT env var.
Adds HorizontalPodAutoscaler resources for the three load-responsive
Deployments:
- prop (web): min 2, max 5, averageValue 800m CPU
- prop-grid-rs: min 1, max 4, averageValue 400m
- hrrr-point-rs: min 1, max 4, averageValue 400m
- prop-backfill is intentionally unmanaged (queue-driven, not CPU-bound)
The scale-up policy reacts in 30 s with up-to-2-pods or +100 % per
window; scale-down stabilises over 10 min so the hourly propagation
chain doesn't churn replicas right after the :05 cron.
Removes `spec.replicas` from the HPA-managed Deployments so Flux's
periodic reconcile stops fighting HPA's live scaling decisions.
Drops `podAntiAffinity` from prop, prop-grid-rs, and hrrr-point-rs
so HPA can actually reach its maxReplicas when one physical host has
capacity for two pods; the scheduler picks whatever fits.
Updates runbook + project memory to drop the `talos5` hostname —
Postgres is now addressed strictly by IP (10.0.15.30).
Root cause of the blank /map analysis breakdown: the skippy.w5isp.com
caching proxy was unreachable from the cluster, so every f00 analysis
step failed at the idx fetch and Rust never wrote a ProfilesFile. The
/map point-detail panel silently degraded to empty factors because
no profile ever lands on `/data/scores/profiles/`.
HrrrClient now takes an optional fallback base. When the primary base
exhausts its retries (5× exponential backoff), fetches retry once
against the fallback — defaulted to the NOAA S3 public bucket. Forecast,
analysis, and native-duct paths all go through the same
fetch_blob_with_fallback helper so a proxy outage degrades to direct
S3 instead of stalling.
Moved the native-URL builder out of native_duct into fetcher so the
single helper can build both primary and fallback URLs for the native
hybrid-sigma product.
Shared NFS idx cache at /data/hrrr_idx lets all prop-grid-rs and
hrrr-point-rs replicas deduplicate redundant NOAA S3 idx fetches.
Opt-in via HRRR_IDX_CACHE_DIR; falls back to in-memory-only when unset.
Also scales prop-grid-rs to 3 replicas (no talos5 pinning),
hrrr-point-rs to 2 replicas, and drops hot pod replicas 4→3 so the
physical-host anti-affinity still fits with talos5 cordoned off for
Postgres.
New telemetry module wires tracing-subscriber to both a local JSON
fmt layer (keeps kubectl-logs output identical) and an
OpenTelemetry OTLP/gRPC exporter, activated only when
OTEL_EXPORTER_OTLP_ENDPOINT is set. The returned TelemetryGuard
holds the SdkTracerProvider until process shutdown so queued spans
flush before exit.
Both bin targets (worker, hrrr_point_worker) now call
telemetry::init(service_name) at startup; service_name becomes the
OTel service.name attribute so Tempo groups spans per binary.
Tracing instrumentation on the three main work units:
- pipeline::run_chain_step (forecast f01..f18)
- pipeline::run_analysis_step (analysis f00)
- hrrr_points::process_batch (per-QSO point drain)
k8s manifests set OTEL_EXPORTER_OTLP_ENDPOINT to the cluster
collector at otel-collector.observability.svc.cluster.local:4317.
Backend wiring lives in the vntx-infra repo.
Two concurrent forecast tasks plus the NFS score-file write cache (23
files x ~2 MB) plus the analysis step's wgrib2 peak repeatedly crossed
the 3 Gi cgroup limit. Single-lane parallelism per pod keeps steady RSS
under 2 Gi; with 2 replicas that still gives 2 concurrent tasks
cluster-wide.
PROP_GRID_RS_PG_CONNS follows the parallelism+2 formula, dropping from
4 to 3.
Two bugs surfacing 13 h after Stream A cutover:
1. prop-grid-rs pods OOMKilled 16–18× overnight on the analysis step.
f00's native-level GRIB2 (~530 MB + wgrib2 working set) runs on top
of 2 concurrent forecast tasks, briefly reaching ~2 Gi. 2 Gi limit
was too tight — 3 Gi plus parallelism 3→2 gives the analysis step
room without eliminating forecast-lane headroom.
2. hrrr-point-rs pod is CrashLoopBackOff because its container is
running image main-1776640915-65f7963 (pre-Stream-C) which doesn't
contain the hrrr_point_worker binary. The grid-rs CI run for
commit 4fefb81 failed and no newer image got published. Touch the
Dockerfile to re-trigger the workflow so flux picks up a new tag
with both binaries. No functional Dockerfile change, just a
docstring update so the path-filter kicks.
Forecast + analysis both claim per FOR UPDATE SKIP LOCKED, so
reducing per-pod parallelism doesn't break the chain — the two
replicas cover 4 slots cluster-wide, which still drains f01..f18 in
~5 min.
Phase 3 Stream C Rust side. Completes the HrrrFetchWorker port.
Pipeline:
- db::claim_next_hrrr_task — FOR UPDATE SKIP LOCKED on hrrr_fetch_tasks,
newest valid_time first. Accepts the points JSONB directly.
- hrrr_points::process_batch — fetch surface + pressure GRIB2 once
per task (tokio::try_join), decode via the existing wgrib2 plumbing,
then for each requested point pull the cell and UPSERT INTO
hrrr_profiles (conflict on lat/lon/valid_time).
- db::complete_hrrr_task / fail_hrrr_task — status transitions; Elixir
backfill re-enqueues failed rows on next /30-min scan.
Shipping pieces:
- new bin src/bin/hrrr_point_worker.rs
- new module src/hrrr_points.rs (process_batch, upsert_profile)
- new Cargo [[bin]] entry; Dockerfile builds both binaries in one stage
and ships them in the runtime image so a single CI pipeline covers
the whole cluster
- k8s/deployment-hrrr-point-rs.yaml (1 replica, 1 Gi limit, anti-affinity
against prop-grid-rs so chain + point work don't fight for wgrib2
slots). Uses the same image; command: override picks the right binary.
- kustomization.yaml: include the new deployment so flux applies it
- deployment-grid-rs.yaml: bump readiness initialDelaySeconds 3→15 +
failureThreshold 3→6 so a slow DB connect during startup can't race
the first probe
119 Rust tests green.
Three independent improvements in one commit:
1. Parallel band scoring (pipeline.rs):
- rayon par_iter across 23 bands replaces the serial for loop.
- All band files now write via try_join_all over spawn_blocking
instead of serializing one-at-a-time on NFS. Chain per-step
drops from ~60s to ~15-25s on a 4-core box.
2. Prometheus metrics (new metrics.rs):
- axum server on METRICS_ADDR (default :9100) exposes /metrics and
/health. Scraped by existing Prometheus via annotation.
- Histograms: chain step duration (by outcome), decode duration.
- Gauge: tasks in flight (RAII guarded so panics don't leak).
- Counter: chain steps by outcome.
- worker.rs records step duration and wraps each run in an
InFlightGuard.
3. HA + ordering fixes:
- deployment-grid-rs.yaml: replicas 1→2, required podAntiAffinity
spreads them across hosts, nodeAffinity prefers talos5 for one.
PROP_GRID_RS_PARALLELISM 4→3 per pod (6 concurrent across
replicas; smaller secondary-node footprint).
- Readiness probe on /health so rollouts wait for the runtime to
be alive.
- claim_next ordering: run_time ASC → DESC so the newest hourly
run drains before any stragglers from a broken prior run. Within
a run, forecast_hour ASC keeps nearest-first.
- grid_task_enqueuer sets kind="forecast" explicitly and updates
conflict_target to the new 3-column unique index introduced by
migration 20260419222624.
Replaces the serial claim→process loop with a JoinSet of N worker tasks,
each running the same claim→fetch→decode→score→complete flow against
grid_tasks. FOR UPDATE SKIP LOCKED already prevents contention between
workers so no additional coordination is needed.
talos5 is 4c/4t (i5-7260U); PROP_GRID_RS_PARALLELISM=4 turns the full
f01..f18 chain from ~18 min serial into roughly ~5 min. Per-task peak
memory after the streamed-bands refactor is ~250 Mi, so 4 concurrent is
~1 Gi — well under the 2 Gi container limit.
Postgres pool sized to parallelism + 2 so NOTIFY and an opportunistic
claim retry share the pool without blocking a worker transaction.
Rust `prop-grid-rs` has been validated end-to-end on talos5 on the
streamed-bands image (main-1776635096-6d91461): real chain steps
complete in ~7 s each and the peak-heap refactor landed.
Changes:
- `PropagationGridWorker.seed_chain` now enqueues a single f00 Oban
job instead of f00–f18. The f00 analysis-hour keeps its enrichment
(native-level duct, NEXRAD composite, commercial-link boost,
ProfilesFile write) and the GridCache broadcast.
- `GridTaskEnqueuer.seed/1` is called again so grid_tasks gets the
18 forecast-hour rows the Rust worker claims.
- Test asserts only one Oban job is enqueued and the matching 18
grid_tasks rows land in the DB.
- `deployment-grid-rs.yaml` flips `PROP_SCORES_DIR` from
`/data/scores_shadow` to `/data/scores` so Rust writes to the live
score tree. No file collision — Elixir only writes the f00 files.
The hot pod memory shrink (6 Gi → 1 Gi) is deferred until one full
hourly cycle runs clean under the new split.
1 Gi OOM'd during a real chain step. Each forecast hour loads both a
surface and pressure-level HRRR GRIB2 (~40 MB combined), decodes via
wgrib2 subprocess (which allocates its own buffers), scores 92k grid
points × 23 bands (~2.1M intermediates), and writes 23 score files.
2 Gi absorbs the spike; still ~3× smaller than the Elixir-per-pod
budget we're replacing.
The real worker binary OOM-killed at 512 Mi on startup. The plan's
512 Mi assumption was speculative — actual Rust RSS at boot includes
sqlx pool buffers, reqwest connection pool, tokio runtime, and
tracing JSON subscriber. 1 Gi leaves room for all of that plus the
steady-state ~500 Mi during a chain step.
Mirrors the Elixir wiring:
- ImageRepository + ImagePolicy scan git.mcintire.me/graham/prop-grid-rs
for main-<ts>-<sha> tags and pick the highest timestamp.
- ImageUpdateAutomation already covers ./k8s via the Setters strategy,
so the new setter marker on deployment-grid-rs.yaml means flux will
rewrite it on every CI push and commit the update back to main.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
