CommonVolumeRadarWorker was enqueuing one job per contact — every
job fetched + decoded the ~5 MB n0q PNG for its own 5-min frame.
At current backlog depth that's 17,588 contacts across just 1,747
distinct 5-min frames, i.e. ~10 contacts per frame paying for the
same PNG decode over and over.
New RadarFrameWorker takes a batch of contact_ids sharing one frame,
fetches + decodes the frame ONCE, then walks the in-memory pixel
buffer per contact. build_radar_jobs/1 in ContactWeatherEnqueueWorker
now groups the input contacts by their rounded 5-min timestamp and
emits one RadarFrameWorker job per frame instead of one
CommonVolumeRadarWorker per contact.
process_frame/4 is public so the same code path is used both by
perform/1 (production fetch) and tests (pre-decoded pixel buffer);
keeps the happy-path test hermetic without hand-crafting a PNG.
CommonVolumeRadarWorker stays in the tree — still used from the
single-contact submit path where batching is pointless and the
aggregate_stats/5 pure helper is a dependency of the new batched
worker.
Expected impact on the backfill: ~10x fewer fetch + decode cycles,
so the 17k queued-contact backlog drains in minutes instead of ~50.
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
The hourly cron now only seeds grid_tasks. The chain step, native-duct
merge, NEXRAD merge, commercial-link merge, scoring, ProfilesFile
write, and band-score writes all moved to rust/prop_grid_rs.
Elixir changes:
- GridTaskEnqueuer.seed_with_analysis/1: inserts 1 kind='analysis' row
(f00) + 18 kind='forecast' rows (f01..f18).
- PropagationGridWorker: stripped from 423 LOC to a thin seeder.
perform(%{}) → GridTaskEnqueuer.seed_with_analysis.
Deleted: process_forecast_hour, merge_native_duct_data,
merge_nexrad_data, merge_commercial_link_data, compute_scores_*,
persist_profiles, record_run_timing (Rust emits spans to Prometheus
instead), apply_nexrad_observations, apply_duct_grid, timed helpers.
Test rewritten for the new shape: 0 Oban fan-out jobs, 19 grid_tasks
rows with the expected kind distribution.
HrrrNativeClient and NexradClient remain — they have other callers
(HrrrNativeGridWorker for per-QSO duct batch; NexradWorker and
CommonVolumeRadarWorker for per-contact radar). Only f00's direct
use moved.
MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an
MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer,
AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime).
Net effect: 30 GRIB2 decodes/hour of dead work on hot pods.
Removed:
- MrmsFetchWorker, MrmsClient, MrmsCache and their tests
- cron entries in config.exs, dev.exs, runtime.exs
- mrms_req_options stub in test.exs
- MrmsCache from supervision tree in application.ex
- stale MRMS references in PropagationGridWorker docstring and test
Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks
the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port)
follow-on work.
Rust `prop-grid-rs` has been validated end-to-end on talos5 on the
streamed-bands image (main-1776635096-6d91461): real chain steps
complete in ~7 s each and the peak-heap refactor landed.
Changes:
- `PropagationGridWorker.seed_chain` now enqueues a single f00 Oban
job instead of f00–f18. The f00 analysis-hour keeps its enrichment
(native-level duct, NEXRAD composite, commercial-link boost,
ProfilesFile write) and the GridCache broadcast.
- `GridTaskEnqueuer.seed/1` is called again so grid_tasks gets the
18 forecast-hour rows the Rust worker claims.
- Test asserts only one Oban job is enqueued and the matching 18
grid_tasks rows land in the DB.
- `deployment-grid-rs.yaml` flips `PROP_SCORES_DIR` from
`/data/scores_shadow` to `/data/scores` so Rust writes to the live
score tree. No file collision — Elixir only writes the f00 files.
The hot pod memory shrink (6 Gi → 1 Gi) is deferred until one full
hourly cycle runs clean under the new split.
Option A from the shadow-mode duplication discussion — Elixir still
owns f00-f18 fan-out via Oban and writes to /data/scores. The Rust
worker sits idle polling an empty grid_tasks table instead of
re-fetching the same HRRR GRIB2 and writing a parallel tree to
/data/scores_shadow.
Brings the seed call back when Phase 2 cutover is approved (at which
point the Elixir fan-out shrinks to fh=0 and PROP_SCORES_DIR flips
from /data/scores_shadow to /data/scores).
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
Telemetry showed the application-master process holding ~830 MiB of
terms from warm_grid_cache_from_latest_profile — the data lives in
the app master's heap and never GCs because the process is idle.
Running it in a Task.start lets the terms die with the task.
Mark GridCache, MrmsCache, NexradCache, and ScoreCache ETS tables
:compressed. The scored-band-map and HRRR grid data are map-heavy;
compression trims hundreds of MiB at a few percent CPU cost.
Memoise HRRR .idx responses in Microwaveprop.Cache. Published idx
files are immutable for a model run, but the hourly chain re-fetches
the same URL dozens of times across forecast hours. Cuts ~10s per
repeat out of hrrr_fetch_idx.
Force a garbage collect at the end of HrrrFetchWorker.perform to
reclaim the refc binary heap held from GRIB2 ranges before the Oban
producer hands the process its next job.
Four changes sized by measured prod telemetry (83m of spans):
1. propagation queue: 2 → 1 slot per pod. Two concurrent forecast-hour
steps per pod stacked HRRR grid + native duct grid + scored band
map into ~5-6 GiB RSS, OOM-killing every ~15 min. 3-way parallelism
cluster-wide still finishes the chain inside the hourly interval.
2. weather queue: 3 → 1 slot per pod. ASOS backfill was 429-thrashing
IEM (1,296 retryable jobs; logs were nothing but 429 backoffs).
3. PropagationGridWorker: skip native-level duct fetch on f01..f18.
At ~7-11 min/fh and 18 forecast hours, this was the largest single
cost per chain. Forecast hours fall back to
derived[:min_refractivity_gradient] from the pressure-level
profile. f00 still gets full native-level duct analysis.
4. HrrrClient.download_grib_ranges_to_file: parallelize with
Task.async_stream (max_concurrency 8). The file-backed variant was
sequential, dominating native-duct fetch time on the remaining f00
path. ~20s → ~3s per call.
The :score_band span was reading point_count via length(grid_data),
but grid_data is a %{{lat, lon} => profile} map. Every forecast-hour
job was crashing with ArgumentError before any scores landed, leaving
the map without current-hour forecasts. Swap to map_size/1.
Two wins on the hourly propagation pipeline:
1. Parallelize the chain. seed_chain was enqueuing only f00 and
each step self-enqueued f00+1, serializing ~2.5 min × 19
forecast hours into a ~48 min chain. Fan out all 19 jobs at
once — they're genuinely independent (different HRRR URLs,
different output files) — and let queue concurrency (2 slots
× 3 pods = 6 parallel workers) drop wall time to ~10 min.
Side effect: one permanently-failing step no longer takes out
the rest of the chain, so the rescue-chain logic I added last
commit becomes unnecessary. Removed.
The :final cleanup (retain_window + purge) used to run on the
fh=18 transition; with parallel execution we don't know which
step finishes last. Cleanup now relies on the existing
PropagationPruneWorker 15-min cron for time-based pruning.
Stale chain leftovers are already a minor concern with the
15-min prune; if file bloat becomes real, PruneWorker can
gain retain_window later.
2. Hoist band-invariant factors. score_time_of_day, score_sky,
score_wind, score_pressure depend on conditions only, not the
band — but composite_score was recomputing them 17 times per
point. Added Scorer.precompute_band_invariants/1, called once
per point before the bands loop; composite_score uses the
cached values when present and falls back to computing for
any caller that doesn't pre-warm (test harness, path
integrator). Saves ~30% of the scoring inner loop.
Telemetry showed ~66 PropagationGridWorker exceptions per 6h with
55 ArgumentErrors and 11 TimeoutErrors, producing ~13 discarded
chain steps. Each discard broke the chain: subsequent forecast
hours were never enqueued, leaving the score store with huge gaps
(e.g. at 14:11 UTC the earliest available forecast was 18:00,
because f00-f05 all failed somewhere upstream and nothing ran
after them).
Three changes:
1. PropagationGridWorker: on the final attempt, still enqueue
fh+1 even when this step failed. Oban discards the current
job normally — but the rest of the chain keeps running, so
one bad hour doesn't take out the remaining 12-18. The
rescue is factored into a tested public helper.
2. HrrrClient.parse_idx: skip malformed idx lines instead of
raising. NOAA S3 occasionally serves an HTML error page as
the idx body, and the old strict String.to_integer path
raised ArgumentError on the first non-numeric line and took
down the chain step. This is the root cause of the 55
ArgumentErrors.
3. JS renderTimeline: when no forecast hour is at-or-before
wall-clock (all times are future — the gap scenario the
fixes above are designed to prevent), stop labeling the
earliest future slot "Now". Lets the user see honest
"+Nh" offsets instead of a lie on the pill.
The :propagation queue (2 slots) is shared by PropagationGridWorker,
MrmsFetchWorker (every 2 min), and PropagationPruneWorker (every
15 min). Oban dispatches priority-then-FIFO, so a post-deploy
MRMS/prune backlog could delay the hourly chain step until the
siblings drained.
Set PropagationGridWorker priority: 0 explicitly and drop the two
siblings to priority: 5 so new chain steps always jump ahead.
Backfill workers live on separate queues (:hrrr, :weather, :terrain,
:iemre, :narr, :radar, :mechanism) and already don't interact with
the predictor.
Adds spans to 15 previously-unmeasured hot paths so every question we
might ask while tuning has a histogram to answer it:
External I/O:
- iem.fetch_iemre (gridded weather reanalysis)
- mrms.list_latest / mrms.download (precip radar)
- rtma.fetch_observation
- ncei.fetch_metar (historical 5-min METAR backfill)
- solar.fetch_indices (GFZ solar indices)
- swpc.fetch (SWPC Kp/F10.7/X-ray)
- giro.fetch (ionosonde)
- qrz.request, geocoder.geocode (callsign enrichment)
- srtm.download_tile (terrain tile download + gunzip)
- hrrr.download_grib_ranges (parallel byte-range fetch phase)
Subprocess:
- wgrib2.extract_grid / extract_grid_from_file / extract_grid_from_file_mapped
LiveView hot paths:
- propagation.scores_at (map score fetch + cache hit/miss counter)
- propagation.point_forecast (sparkline)
- propagation.point_detail (click-to-inspect)
- propagation.daily_outlook_at (/map outlook strip)
Worker-level end-to-end:
- worker.terrain_profile
- worker.mechanism_classify
- worker.mrms_fetch
Each event is registered in Microwaveprop.PromEx.InstrumentPlugin as
a Prometheus histogram (default / long buckets as appropriate) plus
a counter for the scores_at cache hit/miss ratio. Prometheus at
10.0.15.25 will start seeing the new series on the next scrape after
deploy.
Previously the CommonVolumeRadarWorker ran silently — no URL logged,
no sign in the logs that it was doing any work. Added info-level
fetch/ok/error logs in NexradClient (same shape as WeatherFetchWorker
RAOB logs) plus start-of-work + ingestion lines in the worker itself.
Unchanged: the existing "no frame" warning and all control flow.
Contact detail now searches for soundings in progressively wider radii
(150 → 300 → 600 → 1000 km) before declaring "No soundings found".
When even 1000 km turns up nothing, enqueue WeatherFetchWorker RAOB
jobs for every station at the widest non-empty radius. The worker
passes the contact_id through so it can PubSub back to
contact_enrichment:<id> as each sounding lands, and the LiveView
rehydrates without a refresh.
Also adds Weather.soundings_with_widening_radius/1 and
ContactWeatherEnqueueWorker.enqueue_raob_fetch_with_widening/1 so the
same logic can be reused from other callers.
Wires the GEFS fetch pipeline into the existing scoring machinery:
- GefsFetchWorker runs Propagation.score_grid_point over every
fetched grid cell and persists the result via replace_scores/2, so
Day 2-7 valid_times show up on the map through the same
Propagation.scores_at/3 path HRRR uses
- Empty-args perform/1 seeds the chain by enqueueing f024-f168 (25
jobs at 6-hour cadence) for the most recent GEFS run that should
be published given NOMADS' ~3-4h publication lag
- Cron: seeds at 05:30, 11:30, 17:30, 23:30 UTC — 5 hours after each
00/06/12/18Z run
- GefsClient.build_profile now emits :wind_u / :wind_v atom keys to
match the scorer's input shape; DB columns keep the *_mps suffix
for unit clarity, remapped at write time
GEFS pgrb2a lacks HPBL, native-level duct data, NEXRAD, and
commercial-link degradation — the extended-horizon score is a
rougher signal than the HRRR-driven one but covers the window HRRR
can't reach.
- GefsClient.build_profile/1: converts raw wgrib2 output into the
scorer-shaped profile, deriving Td from RH via Magnus at both the
surface and each pressure level
- GefsClient.fetch_grid_profiles/3: downloads the idx sidecar from
NOMADS, byte-ranges only the surface messages, hands the slim
GRIB2 binary to wgrib2 -lola for bilinear regridding from 0.5°
onto the CONUS 0.125° grid
- GefsFetchWorker: one Oban job per (run_time, forecast_hour),
upserts into gefs_profiles and runs SoundingParams.derive so the
same refractivity/ducting metrics flow through
Oban gets a new :gefs queue (2 slots dev/config, 1 slot per prod
pod). Test env gets a gefs_req_options Req.Test stub slot so future
integration tests can drive the full fetch path. No cron wiring or
scorer integration yet — that's Step 3.
Audit pass found two real issues (most agent-flagged items were false
positives — the hrrr_native_profiles composite index exists,
pwat_mm/bl_depth_m are already in the path conditions map, and the
/1.0 in MechanismClassifier is the standard Elixir int→float coerce).
1. Path calculator's 9-point HRRR sampling was running sequentially,
~9× the latency of a single query. Task.async_stream with
max_concurrency: 9 makes them concurrent so path calc pays roughly
one query's worth of time instead of nine.
2. MechanismClassifyWorker's meteor-shower window used hardcoded 2024
peak dates with `%{date | year: peak.year}` projection. That broke
across year boundaries — a Dec 30 contact couldn't match a Jan 4
Quadrantids peak even in-range of the ±3 day window. Switched to
{month, day} tuples and check the peak projected onto year N-1, N,
N+1 so the window works correctly near Dec 31 / Jan 1.
Classifies every contact's likely non-LOS propagation mechanism and
persists the result on contacts.propagation_mechanism. Mechanism is
determined in priority order:
1. user_declared_prop_mode (ADIF PROP_MODE from the operator log)
2. EME — moon-ephemeris check, ≥2m band, >1800 km path
3. aurora — Kp≥5 + high-lat path, 50-432 MHz
4. sporadic-E — foEs × 5 ≥ band_mhz, 400-2500 km path
5. meteor_scatter — ±3 days of a shower peak, VHF/UHF
6. rain_scatter — common-volume radar heavy rain, 5-11 GHz ≤800 km
7. tropo_duct — HRRR native_best_duct ≥ band or ducting_detected
8. line_of_sight — ≤50 km path
9. troposcatter — default
Persisted via MechanismClassifyWorker (queue: :mechanism, unique on
contact_id). Submit-time enqueue path includes :mechanism by default;
BackfillEnqueueWorker cron now handles :mechanism alongside existing
types so prod continuously backfills any contact with
propagation_mechanism_status in (:pending, :queued, :failed). Also
added :radar to the cron's type list so common-volume radar backfill
runs automatically rather than only via `mix radar_backfill`.
New modules:
- Microwaveprop.Propagation.MoonEphemeris — Meeus low-precision moon
position, accuracy ±1° — enough for the mutual-visibility EME test
- Microwaveprop.Propagation.MechanismClassifier — plug-in priority
chain over the evidence map
- Microwaveprop.Workers.MechanismClassifyWorker — assembles inputs
from HRRR / native profiles / common-volume radar / solar_indices /
ionosonde + calls the classifier
ADIF importer now reads PROP_MODE into user_declared_prop_mode so
operator-tagged mechanisms (EME/ES/MS/RS/AS/AUR) become ground truth.
Add a propagation_run_timings table so the wall-clock duration of each
(run_time, forecast_hour) step is queryable long after the run is over.
Keyed by (run_time, forecast_hour) with a status column that captures
whether the step succeeded or bailed out, and an error string on
failure.
PropagationGridWorker stamps every step (ok and failed) via
Propagation.record_run_timing/1. Timing inserts are wrapped in rescue +
changeset-error handling so the instrumentation can never brick the
chain.
Both workers had unique: [period: 300] which only dedupes within a
5-minute window. Re-running the enrichment-enqueue script hours or
days later stacked identical jobs in the available queue — we found
~60k redundant weather + 3k redundant hrrr jobs in prod from the
March and April enqueue passes.
:infinity means Oban dedupes against every live job in the states
list regardless of age, so a second enqueue pass is a no-op as long
as the first pass's jobs haven't completed yet.
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.
- New import_runs table: stores preview rows + running counters (total,
processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
its slice via CsvImport.commit_rows/1, atomically increments counters
in one update_all, flips status, and broadcasts progress on
'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.
Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.
LiveView UI for /imports/:id is a separate commit.
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).
- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
(was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
Backtest branch's unknown-feature guard was dead code: the if block's
{:error, _} return was discarded and execution fell through, running the
backtest with a just-created atom and writing a report for nonsense
features. Two compounding issues:
- function_exported?/3 returns false for modules not yet loaded in the
BEAM, so even valid feature names triggered the warning path on a
cold VM.
- String.to_atom/1 on caller-supplied strings created unbounded atoms.
Fix: Code.ensure_loaded?(Features) before the check, String.to_existing_atom/1
with rescue (Features exports __info__(:functions) so every valid function
atom already exists), and an explicit early return via a with pipeline so
the error tuple actually leaves the worker.
FreshnessMonitor's comment claimed 'Oban unique constraint on
PropagationGridWorker prevents duplicates' — but the worker declared
no unique: option. During a long outage the monitor's 5-min stale-check
tick would pile up identical jobs (a 2h outage = 24 stacked jobs, each
doing the full f00-f18 HRRR chain).
Put the dedup on the worker (vs. the monitor's insert call) so anything
enqueuing it benefits — FreshnessMonitor, the hourly cron, or a manual
mix propagation_grid run.
unique: [period: 3600, states: [:available, :scheduled, :executing,
:retryable]] aligns with the hourly cron cadence. The seed job args
(%{}) collapse with themselves while chain-step jobs keep distinct
run_time/forecast_hour and remain enqueuable.
Status page NARR progress counted `hrrr_status = :unavailable` as the
candidate set, which swept in ~13.6K post-2014 HRRR retry rows and
showed "0 / 13,652". Candidate eligibility is qso_timestamp-based, not
hrrr_status-based — switch both the numerator and denominator to
`qso_timestamp < NarrClient.coverage_end()` so pre-2014 rows are
counted whether their hrrr_status is :queued, :unavailable, or
:complete.
ContactWeatherEnqueueWorker: short-circuit hrrr_points_for_contact/1
for pre-coverage timestamps and mark the contact :unavailable when no
HRRR jobs are built. Was producing a ping-pong where reconcile flipped
pre-2014 :queued → :unavailable and the same cron run flipped it back
to :queued from a rebuilt HRRR job that could never land data.
Makes the 'always recompute' intent explicit by removing the dead-code
short-circuit in add_distance_change — the candidate query already
requires at least one nil pos, so any previously stored distance is
tied to a stale grid/pos pair and must be overwritten.
10K+ HRRR-pending contacts were starving the 200-ish NARR candidates
out of the 500 per-run slot each 30 min because NARR candidates sort
last by the :hrrr_status priority. Drop the cap so every eligible
contact gets enqueued on each cron run; queue concurrency still paces
the actual fetches. Worker keeps supporting an explicit "limit" arg
for ad-hoc runs and tests.
Safety net for rows that land via direct DB writes (manual fixes, bulk
imports) and bypass Radio.create_contact's grid-resolution requirement.
Runs every hour on the :admin queue with a 500-row per-invocation cap.
Fills whichever side resolves — if one grid is invalid, the other still
gets populated; distance_km is only written when both positions end up
set.
Post-2014 contacts with hrrr_status=:unavailable were being dispatched to
NARR, which 404s because NCEI's archive ends 2014-10-02. 14K of the 14.3K
candidates in prod are post-2014 — these are HRRR's responsibility.
- NarrClient.in_coverage?/1 returns true only inside 1979-01-01 → 2014-10-02
- narr_jobs_for_contact and maybe_enqueue_narr (contact_live) bail out of
coverage returning []
- BackfillEnqueueWorker.type_filter scopes :narr to qso_timestamp <
coverage_end so the cron doesn't keep picking post-2014 candidates
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.
Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs
Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
(the symbol still means "historical backfill", just routed to NARR now)
Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).
Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
Swap the body of era5_jobs_for_contact/1 to enqueue NarrFetchWorker
instead of Era5FetchWorker, and snap qso_timestamp via
NarrClient.snap_to_analysis_hour/1 so it lands on a 3-hourly NARR
analysis slot (00/03/06/09/12/15/18/21 UTC) — NarrFetchWorker raises
on anything else.
The :era5 type key, the era5_jobs_for_contact function name, and
the era5_status field are kept as historical artifacts — they
still refer to the era5_profiles table, which is reused 1:1 by
NARR. A schema rename is a follow-up PR.
The CDS Era5* workers (Era5FetchWorker, Era5SubmitWorker,
Era5PollWorker, Era5MonthBatchWorker, Era5BatchClient) are now
unreachable from the live submission flow. They stay in tree until
the deletion follow-up PR.
Per-contact Oban worker that fetches one NARR profile via
NarrClient.fetch_profile_at/2 and inserts it into era5_profiles.
Replaces Era5FetchWorker for pre-2014 contacts. Unlike the ERA5
router, NARR fetches are per-point so this worker does the fetch
+ insert directly — no downstream batch worker, no routing.
Args shape matches Era5FetchWorker: %{"lat", "lon", "valid_time"}.
Lat/lon get snapped to 0.25° (same dedup granularity as ERA5).
valid_time MUST already be on a 3-hourly NARR analysis slot
(00/03/06/09/12/15/18/21 UTC on the hour) — the worker raises
ArgumentError via NarrClient.url_for/1 otherwise. Caller-side
snapping (in ContactWeatherEnqueueWorker) lands in the next task.
Existence check tightened to ±15 minutes (vs ERA5's ±30) since NARR
is exact 3-hourly. On error, logs a warning and returns {:error,
reason} so Oban retries with backoff (max_attempts: 3).
The :narr Oban queue config and the ContactWeatherEnqueueWorker
swap are deliberately NOT touched in this commit — those are the
next two tasks of the plan.
Prod has 0 era5_profiles rows after ~500 submit cycles: every CDS job
either gets marked "rejected" (likely cap mismatch) or runs for 13+h
without transitioning to successful, so the handle_both_done branch
has literally never fired and the ERA5Batch: stored log line has
never been emitted once. Tighten Era5PollWorker's @stuck_after_seconds
from 18h to 8h so dead rows recycle within a work shift instead of a
full day, and pause the era5 / era5_submit / era5_poll / era5_batch
queues in runtime.exs so existing rows stay put for inspection while
we figure out why CDS is rejecting everything. Flip paused: true back
once the root cause is identified.
Adds the second ionospheric data source layer. Polls NOAA SWPC's free
public JSON services every 5 minutes for three products:
* Planetary K-index (1-min cadence) → geomagnetic_observations
* 10.7 cm solar flux (hourly) → solar_flux_observations
* GOES X-ray flux (1-min, 0.1-0.8nm long-wave band only) →
solar_xray_observations
All three products feed HF / Es scoring inputs that the propagation
engine will start using in a follow-up commit:
* Kp drives HF absorption and Es damping (memory notes Kp is inversely
correlated with tropo/Es stability).
* F10.7 is the SSN proxy for ITU-R P.533 HF MUF prediction.
* GOES X-ray long-wave flux is the short-wave fade (SWF) / D-layer
absorption proxy — a C-class flare starts attenuating HF within
seconds of the X-ray peak.
New context `Microwaveprop.SpaceWeather` exposes `upsert_*/1` and
`latest_*/0` per product; `SpaceWeatherFetchWorker` pulls all three
independently so one endpoint's outage doesn't block the others.
Fixtures captured from live SWPC endpoints on 2026-04-15 for the
parser tests.
Not yet wired into scoring — that's a separate commit.
First step toward physics-based HF / sporadic-E scoring. Polls GIRO's
DIDBase tabular endpoint every 10 minutes for two US ionosondes
(Millstone Hill MHJ45, Alpena AL945), parses the plain-text response
into observation rows, and upserts into a new ionosonde_observations
table keyed on (station_code, valid_time).
This gets foEs (E-layer sporadic critical frequency) into the database
as a direct measurement — the key input for predicting 144 MHz Es
openings via ITU-R P.534-6. foF2 + MUFD are the F2-layer inputs for
HF MUF predictions.
Not yet wired into scoring. Boulder/Wallops/Austin/Idaho/Point Arguello
are in the GIRO catalog but were silent when probed — add them back
if/when they come online.
Next steps: SWPC JSON (Kp, F10.7, sunspot), GOES X-ray flux, D-RAP text,
and the P.534-6 Es scoring factor that uses foEs at midpoint for the
144/440 band configs.
A brief CDS network blip on 2026-04-15 returned HTTP 5xx from check_status
for multiple concurrently-polling workers. The old `{:error, _}` branches
consumed attempts, max_attempts=10 burned out in seconds, and 13 rows
were discarded with no cleanup — orphaned with no active poll worker,
including two rows that had both CDS legs done and were ready to decode.
Result: zero era5_profiles ever inserted despite 59 tile-months in flight.
Transient HTTP errors are now a `{:snooze, 60}` so a CDS outage pauses
polling instead of killing it. Terminal states (:failed, :rejected,
:not_found, :stuck) still drive the fail/resubmit paths.
The grid worker stopped writing HRRR profiles to the DB on April 14
when the forecast grid moved to /data/scores binary files, but the
24/72h-windowed prune left every is_grid_point=true row older than
72 hours sitting in the table (billions of rows across every
partition back to 2016, ~90% of the 146 GB table size). Nothing
reads them anymore.
Replace prune_old_grid_profiles/0 with purge_grid_point_profiles/0,
which walks every hrrr_profiles partition directly and deletes
is_grid_point=true rows regardless of age. Per-partition DELETEs
avoid a parent-table full scan. QSO-linked rows (is_grid_point=false)
are untouched — those still back contact enrichment and /path.
Call the new purge from the grid worker's end-of-chain hook so any
grid-point row that somehow slips back in gets cleaned up within the
next hour. Add `mix hrrr.purge_grid_points` for a one-shot historical
cleanup.
The "Updating propagation +Nh" chip was double-misleading in prod:
the label's +Nh was relative to the HRRR run time (which lags wall
clock by ~2h), and the progress broadcast fired before the hour was
fetched + persisted — so the chip could read "+8h" while the map
timeline only extended to +4h.
Reframe the label as "through now" / "through +Nh" / "through Nh ago"
by computing the offset from valid_time → now in PipelineStatus.
running_detail, so it matches the semantic the user reads off the
timeline. Move the PubSub broadcast in PropagationGridWorker to fire
after Propagation.replace_scores/2 succeeds, so the chip only
advances once that forecast hour is readable from ScoresFile.
Two related optimizations in the propagation chain hot path. Both
land on the f01..f18 step that was previously OOM-killing prod
pods after the HRRR pressure-level footprint halved wasn't enough
to fit inside 4 Gi.
1. Skip the GridCache broadcast on forecast hours.
/weather only ever renders the analysis hour (latest_grid_valid_time
feeds the map). Building 92k rows, serializing them through PubSub,
and rebuilding the {lat,lon}→row map on all three replicas was
pure waste for f01..f18 — no consumer was reading that data. Only
f00 now calls build_grid_cache_rows + broadcast_put. Point lookups
for non-analysis hours still work through ProfilesFile on disk
(weather_point_detail_from_profiles/3) exactly as before.
2. Fold replace_scores into a single streaming pass.
The old path did `Enum.to_list/1` on the ~460k-entry score stream
followed by `Enum.group_by/2`, holding two full copies of the grid
before any file was written. A single `Enum.reduce/3` that folds
each score into a per-band accumulator keeps only one copy and
eliminates the group_by intermediate entirely. The public
signature — an Enumerable in, {:ok, count} out — is unchanged.
Replace the timer-based Oban.Plugins.Lifeline (45 min rescue_after)
with Oban.Pro.Plugins.DynamicLifeline, which watches producer
records and rescues orphaned jobs within one rescue_interval (30s)
after the owning pod disappears. This cuts PropagationGridWorker
chain recovery from up to 55 min (wait for next hourly cron) down
to ~30s after a rolling deploy kills a mid-flight step. Bump the
worker's max_attempts 3 → 5 so a couple of rescues during a deploy
don't exhaust the chain's retry budget.
Extend the f00 profile persistence to cover every forecast hour
(f00-f18) so /weather keeps working across pod restarts and shows
forecast-hour atmospheric data without a fresh database. Route the
Weather cold path (latest_grid_valid_time, load_weather_grid,
weather_point_detail) through ProfilesFile first, with the legacy
hrrr_profiles table as a last-resort historical fallback. Warm
GridCache from the latest profile on app start so /weather is hot
the moment a pod boots.
The propagation_scores → binary files cutover dropped the factors
JSONB column, which left point_detail returning factors: %{} and
broke the analysis breakdown popup on map clicks. Persist the
enriched f00 grid_data to /data/scores/profiles/{iso}.etf.gz and
rescore on demand at click time so the factor block renders again.
Flip the cron from every-3-hours to every-hour now that a full
f00-f18 chain runs in ~45-60 min instead of ~170 min. With the
:propagation queue's concurrency-of-2, a slow chain won't block
the next one — they interleave, and the file store's last-writer
-wins semantics give the newer run_time's analysis data naturally.
Add ScoresFile.retain_window/2 that deletes any file whose
valid_time falls outside [run_time, run_time + 18h]. Call it at
fh=18 of the chain so leftover files from the previous chain
(specifically the old f00 whose valid_time the new chain doesn't
overwrite because it advanced by one hour) are discarded
immediately instead of waiting for the 2h prune cron.
The step transition path is extracted to handle_step_transition/2
to keep the nesting under credo's limit and make the final-step
logic easier to read.
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:
1. New migration drops the propagation_scores table and its indexes
(the earlier tuning migrations for it were already applied and
are now no-ops against a missing table, which is fine — Ecto
just runs them on fresh environments).
2. Propagation context is gutted of every GridScore reference.
replace_scores/2 writes files only. upsert_scores/2 is deleted.
load_scores_from_db, available_valid_times_from_db,
point_detail_from_db, point_forecast_from_db, fetch_factors,
coalesce_factors, the Postgres side of prune_old_scores, and
the Postgres fallbacks in latest/earliest_valid_time are all
removed. point_detail always returns an empty factors map now
since factor breakdowns were retired with the table.
3. Deleted modules:
- Propagation.GridScore (the schema)
- Propagation.ScorerDiff (read factors from the table)
- Propagation.AsosNudge (helper for AsosAdjustmentWorker)
- Workers.AsosAdjustmentWorker (its cron was already disabled)
- Mix.Tasks.ScorerDiff (wrapper around the deleted module)
And their tests. AdminTaskWorker's scorer_diff task is a
logging no-op so any queued Oban rows drain cleanly.
Release.scorer_diff stays as a stub that tells the operator.
propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
PropagationGridWorker.merge_commercial_link_data/2 called
Commercial.link_degradation_at twice per grid cell (once to count,
once to merge), and each call re-ran enabled_links plus two sample
queries per in-range link. That was ~500k SQL queries per forecast
hour — fine for the DB but catastrophic for log volume in dev
iex sessions trying to watch a chain step run.
Split into two new functions:
* build_link_lookup/2 does all the DB work up front — one
enabled_links query + one link_degradation per enabled link
(~10 queries total).
* link_degradation_from_lookup/3 is pure: takes a (lat, lon)
and the precomputed lookup, returns the aggregate or nil.
The worker now calls build_link_lookup once per forecast hour and
the per-cell path is haversine-only. Net: 500k queries → ~10.
PipelineStatus freshness detection moves off oban_jobs.completed_at
onto ScoresFile.latest_valid_time(). The on-disk files ARE the data
the map renders from, so the chip's "Up to date · Nm ago" and the
2h stale threshold now track actual data presence. Running-state
detection still comes from oban_jobs since the chain step is still
an Oban row. Tests updated to seed a ScoresFile instead of a
completed Oban row for the idle/stale cases.
