`Pskr.Recalibrator.run/0` reads `pskr_calibration_samples`, bins
each sample per (band × feature), and writes spot-density stats to
`pskr_feature_bins` so an operator can read whether a feature
actually discriminates propagation at the threshold granularity
the scorer uses.
Bins, not regression: `BandConfig` already encodes scoring as
discrete thresholds, so the bin output matches that shape and an
operator can copy adjusted thresholds directly without translating
from regression coefficients.
Self-healing: corpus too thin ⇒ run row written with status
`skipped_insufficient_data` and the analysis is a no-op until next
fire. `min_total_samples = 1000` (≈ 4-5 days of CONUS PSKR
activity); per-band threshold is 100. Both surface in the run row's
`notes`.
Auto-applies nothing. Weight changes still go through human review
of `BandConfig.@band_configs` and a code commit. The recalibrator
is a read-only analyst that stays out of the production scoring
path.
Features binned (matching the scorer's discriminating fields):
* pwat_mm — humidity U-shape candidate
* hpbl_m — boundary layer (mechanism vs scoring re-eval)
* min_refractivity_gradient — refractivity threshold validation
* surface_pressure_mb — pressure-front proxy
* kp_index — aurora boost magnitude tuning
Schema: two tables.
* `pskr_recalibration_runs` — one row per fire with corpus
stats, status, notes
* `pskr_feature_bins` — one row per (run, band, feature, bin)
with sample_count, spot_count_total/avg/p50/p90
Cron: `0 4 * * 0` (Sundays 04:00 UTC, off-peak, post-climatology).
Manual reruns enqueue with no args.
Tests cover the empty-corpus skip path, sub-threshold totals,
per-band threshold gating, the actual bin emission, nil-feature
handling, spot-count averaging, and the always-records-a-run
audit invariant. 8 new tests, 3282 total passing.
Backfill pipeline untouched.
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:
* `pskr_spots_hourly` — observed spot density (truth signal)
* `hrrr_profiles` nearest match at the cell midpoint at hour
boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
ducting flag)
* `geomagnetic_observations` latest Kp at the hour (space weather)
Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.
Components:
* Migration `20260504210756_create_pskr_calibration_samples` —
new table + unique index on (hour, band, midpoint_lat, lon),
plus a midpoint spatial index on `pskr_spots_hourly` to keep
the join cheap.
* `Pskr.CalibrationSample` — schema mirror of the table with
the same `(hour, band, midpoint_lat, lon)` unique constraint.
* `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
for the hour, snaps midpoints to the propagation grid (0.125°),
builds an in-memory HRRR index over a ±0.07°/±60 min window,
and bulk-upserts samples. Idempotent — re-runs upsert.
* `Workers.PskrCalibrationWorker` — Oban cron entry on the
`:backfill_enqueue` queue. Default args target the previous
full hour; explicit `hour_utc` arg reruns any hour.
* Cron `25 * * * *` — fires past HRRR analysis publish window
(~HH:50→HH:05) and PSKR aggregator's 60s flush.
Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.
Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).
Backfill pipeline untouched — none of these changes feed into
contact enrichment.
Both RTMA and METAR5 schemas + clients + workers were defined but
never had a consumer. `find_nearest_rtma/3` and `recent_surface_obs/3`
existed in `Microwaveprop.Weather` with zero callers cluster-wide;
RtmaFetchWorker had test coverage but was never enqueued anywhere;
no IEM 5-min ASOS fetcher was ever written. The tables sat empty in
prod and the recalibrate audit was permanently flagging them BROKEN
even though the empty state was the steady state.
Drop the lot — schemas, clients, workers, queue config, Req.Test
stubs, audit checks, and the per-table unique tests in the
observation-changesets suite. New migration drops the now-unreferenced
`rtma_observations` and `metar_5min_observations` tables (both 0
rows in prod). Trim the `:rtma` slot out of `backfill_only_queues`
in runtime.exs so Oban Pro's Smart engine stops trying to manage a
queue with no producer.
Audit thresholds reset on the surviving NARR check: drop the
"WARN < 5000 absolute rows" rule that didn't account for
NarrFetchWorker's 0.13° spatial dedup, and point remediation at
the existing BackfillEnqueueWorker cron instead of the dev-only
`mix narr.backfill` task.
scripts/recalibrate_algo.py: ROW_COUNT_SOURCES + DATA_GAP_SQL +
DATA_GAP_RULES all stop referencing the dropped tables.
The hrrr_climatology table previously rebuilt only when an operator
remembered to run `mix hrrr_climatology` from a workstation, which
the prod environment can't do. AdminTaskWorker already exposes the
same logic as a "climatology" task, so a one-line cron entry
(02:30 UTC daily) lets prod self-heal: empty/stale rows refold from
the hrrr_profiles archive within 24h, no manual intervention.
Recalibration audit remediation now points at the cron path instead
of the dev-only mix task.
Both `prop` and `prop-backfill` deployments use `app=prop` for the
libcluster selector, so they share a :global namespace. During a
rollout the backfill pod can win FCFS leader election and end up
running the real-time MQTT client on the same pod that's churning
through batch enrichment jobs. Worse, the backfill pod was still
running the pre-IPv4-fix image and surfacing :einval on every
30-second retry.
Gate `pskr_mqtt_enabled` off when PROP_ROLE=backfill. Explicit
PSKR_MQTT_ENABLED=true still wins if you ever want to flip it on
for a dedicated MQTT pod.
Subscribes to mqtt.pskreporter.info:1883 over plain TCP and folds
every reception report into hourly aggregates per
(band, sender_grid_4, receiver_grid_4). Each spot is the empirical
"propagation actually occurred on this path right now" signal we'll
correlate against HRRR atmospheric state to recalibrate the scoring
weights. Aggregating at the path-hour bucket collapses 50-reporter
QSOs to one row instead of 50.
Topic filter anchors on USA (DXCC 291) on either sender or receiver
side so the broker pre-filters out everything outside our HRRR
domain. Bands subscribed: VHF and up (6m, 2m, 70cm, 23cm, +
microwave) — HF is dominated by ionospheric propagation, which this
project doesn't model.
Pskr.Client cluster-elects a singleton via :global.register_name —
all replicas start the GenServer but only one connects to MQTT;
the rest stay in :standby and watch for the leader's nodedown to
re-run election. Off in dev/test, on by default in prod
(PSKR_MQTT_ENABLED=false as kill-switch).
Pskr.Aggregator buffers in memory keyed by Pskr.path_key/1 and
flushes every 60s via Repo.insert_all/3 with an additive
ON CONFLICT (count summed, SNR envelope by GREATEST/LEAST,
modes unioned via unnest+DISTINCT). Idempotent across overlapping
flushes and across leader handover.
Dockerfile sets BUILD_WITHOUT_QUIC=1 — emqtt's transitive quicer
dep wants CMake + OpenSSL headers we'd otherwise have to add to
the builder image just to never use QUIC over plain MQTT. Base
image is unchanged; the new dep compiles cleanly into the existing
prop-base runtime.
Each cached HRRR valid_time is ~32 MiB compressed × cap of 8 = up to
256 MiB per pod. With 4 hot pods + 1 backfill that's ~1.25 GiB of
cluster RAM holding identical content. Replacing the per-pod ETS
replicas with a single shared Valkey copy collapses that to ~256 MiB
total (off-pod) and makes the BEAM heap on each replica meaningfully
smaller — directly addresses the headroom the 2026-05-03 OOM cascade
ate into.
* New `Microwaveprop.Valkey`: single Redix connection (named conn,
sync_connect: false, exit_on_disconnection: false). `start_link/0`
returns `:ignore` when VALKEY_URL is unset so dev/test boot without
Valkey. Helpers wrap GET/MGET/SET-with-TTL/ZADD/ZREVRANGE/SCAN with
`:erlang.term_to_binary` round-tripping under the safe atom flag.
* `Microwaveprop.Weather.GridCache` now branches on
`Valkey.configured?/0`. Public API is identical so callers (Weather,
GridCachePruneWorker, MapLive) don't change. ETS path is preserved
unchanged for dev/test.
* Storage layout when on Valkey:
prop:wg:vts ZSET of valid_time iso8601 strings
prop:wg:<vt>:<lat>:<lon> one chunk per 5°×5° spatial bucket
fetch_bounds MGETs only the chunks intersecting the viewport, so a
regional view is 1-4 round-trips instead of pulling the full grid.
* Per-key TTL = 8 h (matches the ETS valid_time cap). Eviction is
automatic; prune_keep_latest is a no-op on Valkey, prune_older_than
uses ZRANGEBYSCORE + DEL.
* All Valkey errors fall back to disk (ScalarFile.read_bounds), logged
as warnings — page works through Valkey outages, no user-visible
break beyond a one-shot latency bump.
ScoreCache, Microwaveprop.Cache, NexradCache, telemetry/Postgres
protocol/libcluster registry tables remain ETS — they're sub-ms
hot-path lookups where a network hop would be felt.
VALKEY_URL must be present in prop-secrets for production rollout
(applied out-of-band, secret.yaml is git-ignored).
Future-proof the queue config so a "let's bump concurrency for
throughput" change has the per-pod RSS math in front of it. Includes
the actual memory limit (6 GiB), observed steady state (3-5 GiB), the
~500 MiB peak per path-compute, and the explicit recommendation to add
hot replicas via HPA or offload HRRR fetch to hrrr-point-rs before
touching this slot count.
The :terrain queue at 3 slots/pod was sized for the lightweight per-QSO
TerrainProfileWorker, but RoverPathProfileWorker now runs the full
PathCompute pipeline (terrain + 9 HRRR profiles + sounding + ionosphere
+ scoring + loss + forecast) — same heap profile as :propagation, which
sits at 1 slot precisely to avoid OOM.
A backfill_paths flood today queued ~200 rover-path jobs. With 4 hot
pods × 3 slots, the cluster stacked 15 concurrent path-computes and
two pods OOM-killed in a loop, taking the libcluster ring with them
(visible as the "unable to connect" warnings on the surviving pods).
New :rover_path queue at 1 slot/pod caps cluster-wide concurrency at
(hot_replicas + 1 backfill), drains the existing 200-job backlog
inside ~30 min, and keeps :terrain free for the cheap workers it was
sized for.
Adds RoverPlanning.backfill_paths/0 that walks every mission and
re-enqueues RoverPathProfileWorker jobs for any (rover × station) pair
whose Path is missing or not :complete. The path-profile worker
short-circuits on :complete, so this is idempotent.
Wires a new RoverMissionBackfillWorker into both dev (config.exs) and
prod (runtime.exs) crons at HH:30 — heals missions stuck in :pending
or :failed from transient elevation-API errors or interrupted
create_mission enqueues without operator action.
- Empty-string env var (k8s secret resolves to '') was passing the nil
check and handing Postgrex an empty URL — now treated the same as unset.
- Drop runtime.exs warning; keep the single warning in application.ex
where the supervisor decision is made.
Wire up Microwaveprop.AprsRepo as an optional secondary Ecto repo that
points at the sister aprs.me Postgres for read-only access. This is the
foundation for upcoming 144 MHz calibration work that uses verified APRS
RF hops as ground truth.
* lib/microwaveprop/aprs_repo.ex declares a read_only repo; no schemas,
no migrations, no writes — Microwaveprop never persists APRS data.
* Application supervisor starts the repo only when configured (dev/test
via config blocks, prod via APRS_DATABASE_URL). A missing config logs
a warning and skips startup so the main app still boots.
* config/dev.exs + config/test.exs add local aprsme_dev / aprsme_test
endpoints. Test pool uses Ecto SQL Sandbox; tests that don't touch
APRS data don't need to check the connection out.
* config/runtime.exs guards the prod block on APRS_DATABASE_URL —
unset env logs a warning and leaves the repo unconfigured.
The hourly :35 fire was reliably being missed in production: the prop
deployment is pinned to a single replica, and frequent ArgoCD deploys
restart the pod often enough that some fires land in the swap window.
Between 13:35Z and 17:25Z today the cron fired exactly zero times.
Switch to :05/:25/:45. Each fire calls seed_current_hour which is
idempotent on the (run_time, forecast_hour, kind, source) unique
index, so duplicate inserts in the same hour are no-ops at the DB
level and don't add cost on the happy path. Three shots per hour
gives two retry windows when a deploy kills the pod mid-cron-tick.
The HRDPS pipeline as written was unusable in production: each chain
step took ~5 hours (not 30-90s as the dev-bench claim) because every
`wgrib2 -lon` batch redundantly JPEG2000-decodes ~150 records. With 57
batches × 18 forecast hours, a single cycle would take days to drain.
The /weather Canadian view only needs the current hour, so the cheapest
fix is to stop seeding the rest of the chain entirely:
* Rust grid: HRDPS_STEP=0.5 (was 0.125) drops point count 16× to ~3.5k.
Coarser cells than HRRR, but /weather gets visible Canadian coverage
inside one chain step (~20 min) instead of never.
* GridTaskEnqueuer.seed_current_hour/2 inserts exactly one forecast row
whose valid_time is closest to `now`. fh is clamped to 1..18 so the
Rust F00Reserved branch never fires.
* HrdpsGridWorker switches to seed_current_hour and drops the
6-hour-cycle cron in favor of HH:35 hourly fires. Reseeds are
idempotent on the 4-column unique key.
Also fixes a pre-existing credo "nested too deep" in
ScalarFile.read_point by extracting lookup_in_chunk + hit_or_false
helpers — happened to be on the read path that surfaces this work, so
folding it into the same commit.
Stages 6, 7, 10 of the HRDPS plan, plus the Elixir read-side merge
that makes Rust's `.hrdps.prop` writes show up on /map.
Read side:
- `ScoresFile.path_for_hrdps/2` + `read_hrdps/2` — companions to the
HRRR equivalents. Same decode path; different filename.
- `ScoresFile.read_bounds/3` now reads HRRR + HRDPS files and
concatenates the cells. Files are disjoint by construction
(Grid.hrdps_only_points excludes CONUS) so no de-dup needed.
- `ScoresFile.read_point/4` checks `.prop`, then `.hrdps.prop`, then
legacy `.ntms` — first hit wins. Cells outside their owning region
read as no-data in the other file's grid, so the order doesn't
matter for correctness.
- `Propagation.warm_cache_and_broadcast/2` reads both files and
merges before broadcasting to the cluster ScoreCache. A single
missing file (HRDPS pre-cycle, HRRR briefly absent) is now OK —
the cache warms with whichever side is available.
Activation:
- runtime.exs cron: HRDPS seed worker fires at HH:35 of each cycle
hour (05:35Z, 11:35Z, 17:35Z, 23:35Z) — 5h after cycle, 30 min
staggered from HRRR's */15 schedule.
- HrdpsGridWorker moduledoc updated: no longer dormant; Rust
prop-grid-rs handles source='hrdps' rows.
- map_live North America bbox extended to (24.5, 60.0, -141.0, -52.0)
for the out-of-coverage banner check; visitors anywhere in CONUS
or the Canadian extent now stay banner-free.
Cleanup:
- mix hrdps.probe deleted — its risks are retired and the production
HrdpsClient covers the same surface.
What's still in motion (not blocking the Canadian /map):
- Per-valid_time profile + scalar artifacts for HRDPS (would let
/weather show Canadian temperature/refractivity layers). Rust
pipeline currently skips those so the HRRR-written companion files
aren't clobbered — separate stage when /weather UX is decided.
- FreshnessMonitor HRDPS staleness tracking — HRDPS has its own cron
and a different cadence than HRRR; HRRR-stale logic doesn't apply
cleanly. Defer until prod tells us what alarms we actually want.
Stage 3 (Elixir foundation) of the HRDPS plan. Lands the scaffolding
that lets the Rust prop-grid-rs HRDPS branch be wired up cleanly when
it ships, without further DB or Elixir-side changes.
What ships:
- `grid_tasks.source` column (default 'hrrr', backfill is a no-op).
Replaces the (run_time, forecast_hour, kind) unique index with a
4-column key (..., source) so HRRR and HRDPS analysis/forecast rows
for the same cycle coexist.
- `Grid.hrdps_only_points/0` — Canadian cells inside HRDPS bbox
(49-60°N, -141 to -52°W) but outside HRRR's CONUS bbox. Disjoint
from `conus_points/0` by construction so the two grids never
double-write the same (lat, lon).
- `Grid.point_source/1` — `:hrrr | :hrdps | :neither` classification
for a (lat, lon) pair. Routes per-QSO enrichment to the right NWP
source.
- `GridTaskEnqueuer.seed_with_analysis/2` and `seed/2` accept a
`:source` option; defaults to "hrrr" so existing callers are
unchanged.
- `HrdpsGridWorker` cycle seeder mirroring PropagationGridWorker's
shape but for HRDPS's 4×/day cadence. `:hrdps` queue added to base
Oban config.
What's deferred:
- The Rust `prop-grid-rs` worker doesn't yet branch on
`task.source`. Until that ships, `HrdpsGridWorker` stays out of
runtime.exs cron — it would just produce rows that the Rust worker
mishandles as HRRR. Module doc explicitly notes the dormant state.
- Map overlay extension and FreshnessMonitor HRDPS staleness deferred
to follow-up sessions once real data flows.
Activation path documented in the updated plan file.
Coverage cap recorded as Stage 8 decision: 60°N for v1 (SRTM stops
there). Arctic CDEM deferred to a follow-up plan.
- Local prominence: each cell's elevation vs. its 8-neighbor ring
(~1.3 km radius) feeds a small additive bonus so broad hilltops
rank above isolated SRTM voxels
- Road proximity: one Overpass call per Calculate fetches drivable
ways inside the bbox; cells beyond ~0.5 km of any road get a
light dB penalty, gated off in tests
NOAA publish latency for HRRR pressure-level f18 straddles ~50-65 min
after cycle hour. The HH:05 cron frequently probes a (now-1h) cycle
that's not yet on S3 and falls back to (now-2h). Until today the next
re-attempt was a full hour later, leaving weather visibly stale.
Quadrupling the schedule lets pick_run_time/1 upgrade to the freshest
cycle within 15 min of it publishing. seed_with_analysis is idempotent
(on_conflict: :nothing), so re-fires against an already-seeded run_time
are a single cheap INSERT batch.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
No translations are actually maintained and no UI is shipped in a
non-English locale. Replaced the handful of gettext(...) calls in
core_components + layouts with their literal English strings, and
rewrote translate_error/1 as a small %{key} interpolator over the
Ecto changeset error tuple.
Removes gettext + expo (compile dep). sutra_ui stays — it's a
transitive dep of live_table's TableComponent.
Replaces the SVG schematic on /eme with a textured WebGL scene: NASA
Blue Marble Earth, real-size Moon, outbound beam, dashed return ray
to the sub-lunar point, and a red shading on the anti-moon hemisphere
that marks who can't see the Moon (outside the bounce footprint).
Three.js (~680 kB) now lives in its own chunk; esbuild --splitting
keeps it out of the main bundle so it only loads when a user hits
/eme. Main app.js drops from 1.6 MB to 922 kB.
Also swaps the Elevation/SNR rows so the badge comes before the number.
Hot pods were running weather/narr/rtma/nexrad/contact_import
alongside LiveView and the hourly propagation chain. The weather
queue in particular hammers IEM with ASOS backfills; each Req
retry against a 429 stacks an in-worker Process.sleep, and enough
concurrent retries saturated the Ecto pool. Once DB checkouts
started queueing past 15 s the cascade was always the same:
Postgrex client timeouts → Oban.Notifier/Peer 5 s health checks
failing → /health readiness timeout → kubelet flipping the pod
out of the service endpoint set. One hot replica accumulated 11
restarts over 16 h.
Split the shared queue block: anything that's truly per-QSO
enrichment on demand (terrain, iemre, radar, mechanism, gefs,
ionosphere, space_weather, admin, exports, backfill_enqueue) stays
on both tiers, while weather/narr/rtma/nexrad/contact_import run
only on prop-backfill. The hot pods mark those as paused so the
queue entries still exist cluster-wide (Oban Pro's Smart engine
rejects `limit: 0`). Cluster capacity for the backfill queues is
unchanged; they just execute on the dedicated t620.
Every prop pod restart was dumping a pair of ArgumentError stack
traces from grpcbox_channel / grpcbox_subchannel terminate/3:
:ets.select(:gproc, …)
* 1st argument: the table identifier does not refer to an
existing ETS table
This is a known grpcbox shutdown race: the OTLP/gRPC transport's
terminate callback tries to look itself up in gproc's ETS just
after gproc's supervisor has already torn the table down. Noisy,
not harmful — but one ArgumentError per restart × N pods adds up.
The OTLP/HTTP transport (port 4318 on the same otel-collector
Service) has no gproc/ETS channel machinery, so shutdown is
quiet. Spans are identical — our collector already accepts both.
- `config/runtime.exs`: default `otlp_protocol: :http_protobuf`,
with an `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` escape hatch.
- `k8s/deployment.yaml`: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
:4318 instead of :4317. The Rust workers stay on :4317 — their
native opentelemetry-otlp transport doesn't share the grpcbox
bug.
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
The ingress + Grafana pipeline parses JSON-formatted log lines; Elixir's
default text formatter was the odd format everywhere else, forcing Loki
to fall back to free-text parsing. Swap the `:default` :logger handler's
formatter to `LoggerJSON.Formatters.Basic` when `LOG_FORMAT=json`.
Defaults: `LOG_FORMAT=text` for dev/test (no behaviour change), `json`
for prod. Whitelisted metadata covers the fields we actually correlate
on — `request_id`, `trace_id`, `worker`, `queue`, `job_id`.
OTP 28 / Elixir 1.19 exposes `Logger.Formatter.new/1` but ships no
JSON encoder of its own, so `logger_json` (~> 7.0) carries the rendering.
The test pokes the formatter directly rather than round-tripping through
ExUnit's capture_log, which replaces the handler and so can't see the
installed formatter's output.
- HrrrClient idx-cache test only invalidated the surface idx URL, but
fetch_profile also fetches a pressure idx. Previous runs' state for
the pressure key decided whether the counter landed at 1 (prior run
cached it, pass) or 2 (cold, fail). Invalidate both + assert 2 total
fetches to reflect the actual code path.
- CsvImportTest deadlocked against other async DataCase tests when
inline Oban child jobs upserted iemre_observations/terrain_profiles
with a shared conflict target. Flip to async: false — same fix as
ContactWeatherEnqueueWorkerTest earlier this session.
- PromEx.Plugins.Oban runs a 5s telemetry_poller that queries the DB,
but its poller PID has no sandbox connection in test and crashed
with DBConnection.OwnershipError on every tick, spamming the log.
Gate the plugin on a config flag and skip it in config/test.exs;
prod behaviour unchanged.
The remote_ip plug previously trusted Cf-Connecting-Ip and X-Forwarded-For
on every request, so any client reaching a pod directly (cluster-internal,
kubectl port-forward, misconfigured Service) could spoof conn.remote_ip,
which flows into session storage and logs.
Now only honour those headers when the immediate TCP peer sits inside a
configured CIDR list. Default list covers loopback, RFC1918, CGNAT,
link-local, and IPv6 ULA — matches the k8s pod/service network.
Override via config :microwaveprop, :trusted_proxies, or per-environment
via the TRUSTED_PROXY_CIDRS env var in runtime.exs.
Also refresh the stale "nginx/dokku proxy" moduledoc — deployment has
been k8s + Cloudflare for a while.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
Periodic 60s sweep re-warms ScoreCache from /data/scores whenever
{band, valid_time} pairs are on disk but absent in local ETS.
Covers the three silent-fail modes of the NOTIFY/LISTEN path:
dropped notifies on reconnect, missing listener, and stale NFS
reads. Each pod reconciles its own cache — no PubSub fan-out,
no coordination required. Rescue on File.Error handles the
rare window where an atomic-rename write races the reader.
docs/runbook_propagation_pipeline.md names every failure mode
in the Rust↔Elixir seam and what recovers it.
Every Sent log line now carries method and path so noisy traffic
(probes, scrapers, redirects) is identifiable without a tracer.
RemoteIp plug already sets remote_ip metadata; add method and
request_path alongside it and register the extra keys with the
default formatter.
Unlimited BackfillEnqueueWorker runs loaded all ~36k enrichable
contacts and hit Oban's ~17 min execution timeout, discarding the
parent job every cron tick. Enrichment was landing partially
(jobs enqueued before the timeout) but the loop never completed
and no success signal was emitted. A per-tick limit of 2000 keeps
each run well under any timeout while still draining the backlog
at 4000/hour across the */30 schedule.
Adds the OTel deps (opentelemetry + exporter + phoenix / ecto / oban /
bandit auto-instrumentation helpers) and attaches them in
Application.start/2 so the existing Phoenix, Ecto, Oban, and Bandit
telemetry events flow as OTLP spans without any call-site changes.
Exporter config is gated on OTEL_EXPORTER_OTLP_ENDPOINT in
config/runtime.exs — set in the k8s manifests to the cluster
collector (otel-collector.observability.svc.cluster.local:4317).
When unset we switch traces_exporter to :none so nothing is shipped;
dev/test stays quiet.
Resource attributes tag spans with service.name=microwaveprop and
service.namespace=prop, matching the Rust workers' attribute shape
so Tempo can group the full hourly chain across both languages.
Both the main prop deployment and the backfill deployment get the
env; backfill is still a full BEAM node running enrichment workers,
so its Oban/Ecto spans are worth seeing too.
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an
MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer,
AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime).
Net effect: 30 GRIB2 decodes/hour of dead work on hot pods.
Removed:
- MrmsFetchWorker, MrmsClient, MrmsCache and their tests
- cron entries in config.exs, dev.exs, runtime.exs
- mrms_req_options stub in test.exs
- MrmsCache from supervision tree in application.ex
- stale MRMS references in PropagationGridWorker docstring and test
Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks
the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port)
follow-on work.
Oban Pro 2.21.1's Smart engine rejects `local_limit: 0` with
"local_limit must be greater than 0", OOMing the backfill pod on every
boot since the upgrade. Switch the disable pattern to
`[local_limit: 1, paused: true]` so the queue still exists (satisfying
the validator) but processes nothing. Verified under PROP_ROLE=backfill
the four hot queues (propagation, commercial, solar, enqueue) resolve
to paused=true and the others keep their normal concurrency.
config.exs registers Oban queues + the Cron plugin at compile time,
and Config deep-merges with runtime.exs by key. Simply omitting
propagation/commercial/solar from runtime.exs left them inheriting
from config.exs, so backfill pods were still processing hot-path
queues and holding the Cron plugin. Override the hot queues to
limit: 0 and re-register Cron with an empty crontab in backfill
mode so the compile-time defaults get neutralised.
PROP_ROLE=backfill drops the hot-path propagation/commercial/solar
queues and the Cron plugin. Paired with a dedicated Deployment
(nodeSelector: prop-backfill=true, 4Gi memory, no web) this lets 8GB
T620 workers drain the hrrr/weather/terrain/iemre/narr backfill
queues without competing with the hourly grid chain.
Two independent wins the latest prod telemetry pointed at:
1. Iowa Environmental Mesonet rate limiter. IEM throttles per source
IP across all in-flight requests, so dropping the `:weather` Oban
queue to 1/pod wasn't enough — workers were still issuing back-to-
back requests inside each job and drawing a steady stream of HTTP
429s (1,396 retryable on the queue). A GenServer-based token bucket
serialises acquire() with a 700ms min gap per pod; three pods give
~4 req/sec cluster-wide, well under the observed 429 threshold.
Wrapped around every IemClient fetch.
2. Parallel HRRR surface + pressure grid fetch. The two products live
in separate wrfsfcf / wrfprsf GRIB files, so their fetch → wgrib2
→ decode pipelines are independent; running them sequentially was
adding ~30s per forecast hour for no reason. Task.async the pair,
merge at the end. Halves per-fh wall time and should unblock some
of the missing-hour cases we saw on the 14:00Z chain.
Four changes sized by measured prod telemetry (83m of spans):
1. propagation queue: 2 → 1 slot per pod. Two concurrent forecast-hour
steps per pod stacked HRRR grid + native duct grid + scored band
map into ~5-6 GiB RSS, OOM-killing every ~15 min. 3-way parallelism
cluster-wide still finishes the chain inside the hourly interval.
2. weather queue: 3 → 1 slot per pod. ASOS backfill was 429-thrashing
IEM (1,296 retryable jobs; logs were nothing but 429 backoffs).
3. PropagationGridWorker: skip native-level duct fetch on f01..f18.
At ~7-11 min/fh and 18 forecast hours, this was the largest single
cost per chain. Forecast hours fall back to
derived[:min_refractivity_gradient] from the pressure-level
profile. f00 still gets full native-level duct analysis.
4. HrrrClient.download_grib_ranges_to_file: parallelize with
Task.async_stream (max_concurrency 8). The file-backed variant was
sequential, dominating native-duct fetch time on the remaining f00
path. ~20s → ~3s per call.
Wires PromEx with its built-in Application / BEAM / Phoenix / Ecto /
Oban plugins plus a custom `Microwaveprop.PromEx.InstrumentPlugin`
that registers Prometheus histograms for every
`Microwaveprop.Instrument` span (HRRR/NEXRAD/IEM/GEFS/NARR/UWYO
/Elevation fetches, DB batch upserts, terrain analysis, radar
aggregation, propagation score band). Queue-depth gauges from the
10-second poller land at `microwaveprop_oban_queue_count{queue,state}`.
Router forwards `/metrics` to MicrowavepropWeb.MetricsPlug which
optionally requires a bearer token (`PROMETHEUS_AUTH_TOKEN` env var in
runtime.exs); leave it unset to expose metrics publicly and restrict
at the ingress / Cloudflare Access layer instead.
Example external Prometheus scrape config:
scrape_configs:
- job_name: microwaveprop
scrape_interval: 30s
metrics_path: /metrics
scheme: https
authorization:
type: Bearer
credentials: <token>
static_configs:
- targets: ['prop.w5isp.com:443']
Wires the GEFS fetch pipeline into the existing scoring machinery:
- GefsFetchWorker runs Propagation.score_grid_point over every
fetched grid cell and persists the result via replace_scores/2, so
Day 2-7 valid_times show up on the map through the same
Propagation.scores_at/3 path HRRR uses
- Empty-args perform/1 seeds the chain by enqueueing f024-f168 (25
jobs at 6-hour cadence) for the most recent GEFS run that should
be published given NOMADS' ~3-4h publication lag
- Cron: seeds at 05:30, 11:30, 17:30, 23:30 UTC — 5 hours after each
00/06/12/18Z run
- GefsClient.build_profile now emits :wind_u / :wind_v atom keys to
match the scorer's input shape; DB columns keep the *_mps suffix
for unit clarity, remapped at write time
GEFS pgrb2a lacks HPBL, native-level duct data, NEXRAD, and
commercial-link degradation — the extended-horizon score is a
rougher signal than the HRRR-driven one but covers the window HRRR
can't reach.
- GefsClient.build_profile/1: converts raw wgrib2 output into the
scorer-shaped profile, deriving Td from RH via Magnus at both the
surface and each pressure level
- GefsClient.fetch_grid_profiles/3: downloads the idx sidecar from
NOMADS, byte-ranges only the surface messages, hands the slim
GRIB2 binary to wgrib2 -lola for bilinear regridding from 0.5°
onto the CONUS 0.125° grid
- GefsFetchWorker: one Oban job per (run_time, forecast_hour),
upserts into gefs_profiles and runs SoundingParams.derive so the
same refractivity/ducting metrics flow through
Oban gets a new :gefs queue (2 slots dev/config, 1 slot per prod
pod). Test env gets a gefs_req_options Req.Test stub slot so future
integration tests can drive the full fetch path. No cron wiring or
scorer integration yet — that's Step 3.
Classifies every contact's likely non-LOS propagation mechanism and
persists the result on contacts.propagation_mechanism. Mechanism is
determined in priority order:
1. user_declared_prop_mode (ADIF PROP_MODE from the operator log)
2. EME — moon-ephemeris check, ≥2m band, >1800 km path
3. aurora — Kp≥5 + high-lat path, 50-432 MHz
4. sporadic-E — foEs × 5 ≥ band_mhz, 400-2500 km path
5. meteor_scatter — ±3 days of a shower peak, VHF/UHF
6. rain_scatter — common-volume radar heavy rain, 5-11 GHz ≤800 km
7. tropo_duct — HRRR native_best_duct ≥ band or ducting_detected
8. line_of_sight — ≤50 km path
9. troposcatter — default
Persisted via MechanismClassifyWorker (queue: :mechanism, unique on
contact_id). Submit-time enqueue path includes :mechanism by default;
BackfillEnqueueWorker cron now handles :mechanism alongside existing
types so prod continuously backfills any contact with
propagation_mechanism_status in (:pending, :queued, :failed). Also
added :radar to the cron's type list so common-volume radar backfill
runs automatically rather than only via `mix radar_backfill`.
New modules:
- Microwaveprop.Propagation.MoonEphemeris — Meeus low-precision moon
position, accuracy ±1° — enough for the mutual-visibility EME test
- Microwaveprop.Propagation.MechanismClassifier — plug-in priority
chain over the evidence map
- Microwaveprop.Workers.MechanismClassifyWorker — assembles inputs
from HRRR / native profiles / common-volume radar / solar_indices /
ionosonde + calls the classifier
ADIF importer now reads PROP_MODE into user_declared_prop_mode so
operator-tagged mechanisms (EME/ES/MS/RS/AS/AUR) become ground truth.
Rename the local dev database from microwaveprop_dev to prop_dev so
the name matches the production database. Add scripts/restore_prod_to_local.sh,
which pulls the latest pg_dump off the backup server, drops the local
prop_dev, recreates it, and pg_restores into it.
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.
- New import_runs table: stores preview rows + running counters (total,
processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
its slice via CsvImport.commit_rows/1, atomically increments counters
in one update_all, flips status, and broadcasts progress on
'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.
Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.
LiveView UI for /imports/:id is a separate commit.
