prop-grid-rs already exposed prop_grid_rs_chain_step_duration_seconds
+ tasks_in_flight on :9100 but the comment still pointed at the old
prom server (10.0.15.25). hrrr-point-rs had no metrics listener at all.
- metrics.rs: new POINT_BATCH_DURATION / POINT_BATCHES_TOTAL /
POINTS_PROCESSED_TOTAL series + record_point_batch helper, plus unit
tests serialized via a static Mutex (the prometheus crate's global
registry races otherwise)
- hrrr_point_worker.rs: spawn metrics::serve on METRICS_ADDR (default
0.0.0.0:9100), wrap process_batch in InFlightGuard, record outcome +
per-point counts on every batch
- deployment-hrrr-point-rs.yaml: prometheus.io/{scrape,port,path,job}
annotations, METRICS_ADDR env, ports.metrics, /health readiness probe
- deployment-grid-rs.yaml: refresh stale 10.0.15.25 comment, add
prometheus.io/job for clean dashboard up{} selectors
Also fix Microwaveprop.PromExTest: `assert plugins != []` triggered
Elixir 1.19 type-checker warning ("non_empty_list != empty_list always
true"); replaced with membership assertions for the InstrumentPlugin +
Plugins.Beam.
The image tags here are placeholders — the live tag is set by the
ArgoCD Application's spec.source.kustomize.images field, same as
before. Only the registry hostname and path changed.
Pull secret 'forgejo-registry' in the prop namespace now carries
both git.mcintire.me and codeberg.org auths during the transition.
Removes the OTLP span exporter (tracing-opentelemetry, opentelemetry,
opentelemetry_sdk, opentelemetry-otlp) and its transitive tonic/prost/
h2 tree — 56 crates off the build graph, ~3-5 min off a cold build on
the forgejo runner.
The fmt layer (JSON to stdout via tracing-subscriber) is unchanged;
structured logs still reach the Loki collector through kubectl logs,
and /metrics on :9100 still serves Prometheus. Traces were a nice-to-
have that nothing on call was actually reading.
telemetry::init keeps its signature (service_name + TelemetryGuard)
so re-adding OTLP later is a single-module change. k8s manifests drop
the now-unused OTEL_EXPORTER_OTLP_ENDPOINT env var.
Adds HorizontalPodAutoscaler resources for the three load-responsive
Deployments:
- prop (web): min 2, max 5, averageValue 800m CPU
- prop-grid-rs: min 1, max 4, averageValue 400m
- hrrr-point-rs: min 1, max 4, averageValue 400m
- prop-backfill is intentionally unmanaged (queue-driven, not CPU-bound)
The scale-up policy reacts in 30 s with up-to-2-pods or +100 % per
window; scale-down stabilises over 10 min so the hourly propagation
chain doesn't churn replicas right after the :05 cron.
Removes `spec.replicas` from the HPA-managed Deployments so Flux's
periodic reconcile stops fighting HPA's live scaling decisions.
Drops `podAntiAffinity` from prop, prop-grid-rs, and hrrr-point-rs
so HPA can actually reach its maxReplicas when one physical host has
capacity for two pods; the scheduler picks whatever fits.
Updates runbook + project memory to drop the `talos5` hostname —
Postgres is now addressed strictly by IP (10.0.15.30).
Root cause of the blank /map analysis breakdown: the skippy.w5isp.com
caching proxy was unreachable from the cluster, so every f00 analysis
step failed at the idx fetch and Rust never wrote a ProfilesFile. The
/map point-detail panel silently degraded to empty factors because
no profile ever lands on `/data/scores/profiles/`.
HrrrClient now takes an optional fallback base. When the primary base
exhausts its retries (5× exponential backoff), fetches retry once
against the fallback — defaulted to the NOAA S3 public bucket. Forecast,
analysis, and native-duct paths all go through the same
fetch_blob_with_fallback helper so a proxy outage degrades to direct
S3 instead of stalling.
Moved the native-URL builder out of native_duct into fetcher so the
single helper can build both primary and fallback URLs for the native
hybrid-sigma product.
Shared NFS idx cache at /data/hrrr_idx lets all prop-grid-rs and
hrrr-point-rs replicas deduplicate redundant NOAA S3 idx fetches.
Opt-in via HRRR_IDX_CACHE_DIR; falls back to in-memory-only when unset.
Also scales prop-grid-rs to 3 replicas (no talos5 pinning),
hrrr-point-rs to 2 replicas, and drops hot pod replicas 4→3 so the
physical-host anti-affinity still fits with talos5 cordoned off for
Postgres.
New telemetry module wires tracing-subscriber to both a local JSON
fmt layer (keeps kubectl-logs output identical) and an
OpenTelemetry OTLP/gRPC exporter, activated only when
OTEL_EXPORTER_OTLP_ENDPOINT is set. The returned TelemetryGuard
holds the SdkTracerProvider until process shutdown so queued spans
flush before exit.
Both bin targets (worker, hrrr_point_worker) now call
telemetry::init(service_name) at startup; service_name becomes the
OTel service.name attribute so Tempo groups spans per binary.
Tracing instrumentation on the three main work units:
- pipeline::run_chain_step (forecast f01..f18)
- pipeline::run_analysis_step (analysis f00)
- hrrr_points::process_batch (per-QSO point drain)
k8s manifests set OTEL_EXPORTER_OTLP_ENDPOINT to the cluster
collector at otel-collector.observability.svc.cluster.local:4317.
Backend wiring lives in the vntx-infra repo.
Three fixes after hrrr-point-rs restarts on the first live drain:
1. Type mismatch (hard error): the `profile` column on `hrrr_profiles`
is `jsonb[]` (one element per pressure level), but the Rust worker
was binding a single jsonb array-of-objects cast as `::jsonb`. Every
insert failed with `column "profile" is of type jsonb[] but
expression is of type jsonb`. Switched to
`Vec<sqlx::types::Json<Value>>`, dropped the explicit `::jsonb`
cast, and enabled sqlx's `json` feature so the encoder maps the
array into `_jsonb` correctly.
2. Silent zero-inserts: four consecutive 2019-09-22 batches completed
with `profiles_inserted: 0` and no diagnostic. Most likely the
upstream archive doesn't keep cycles that old, but without a log
it looks identical to a snap-mismatch bug. Added a WARN when both
surface and pressure grids come back empty so fetch-miss vs.
projection-miss is distinguishable.
3. OOMKilled: the pod took four OOM restarts in ten minutes at 1 Gi.
A single CONUS decode holds the ~40 MB blob, the ~200 MB wgrib2
working set, AND the full 92k-cell merged map until all requested
points drain. 1 Gi has no headroom. Bumped to 2 Gi, matching the
rest of the per-container budgets in this namespace.
Phase 3 Stream C Rust side. Completes the HrrrFetchWorker port.
Pipeline:
- db::claim_next_hrrr_task — FOR UPDATE SKIP LOCKED on hrrr_fetch_tasks,
newest valid_time first. Accepts the points JSONB directly.
- hrrr_points::process_batch — fetch surface + pressure GRIB2 once
per task (tokio::try_join), decode via the existing wgrib2 plumbing,
then for each requested point pull the cell and UPSERT INTO
hrrr_profiles (conflict on lat/lon/valid_time).
- db::complete_hrrr_task / fail_hrrr_task — status transitions; Elixir
backfill re-enqueues failed rows on next /30-min scan.
Shipping pieces:
- new bin src/bin/hrrr_point_worker.rs
- new module src/hrrr_points.rs (process_batch, upsert_profile)
- new Cargo [[bin]] entry; Dockerfile builds both binaries in one stage
and ships them in the runtime image so a single CI pipeline covers
the whole cluster
- k8s/deployment-hrrr-point-rs.yaml (1 replica, 1 Gi limit, anti-affinity
against prop-grid-rs so chain + point work don't fight for wgrib2
slots). Uses the same image; command: override picks the right binary.
- kustomization.yaml: include the new deployment so flux applies it
- deployment-grid-rs.yaml: bump readiness initialDelaySeconds 3→15 +
failureThreshold 3→6 so a slow DB connect during startup can't race
the first probe
119 Rust tests green.
