PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
Replaces per-QSO-endpoint asos_batch granularity with per-station-day
granularity. Each contact in the same UTC day near the same station
now produces identical unique-args and collapses to ONE Oban job, so
the backlog shrinks as cross-contact duplicates dedup.
Each fetch also covers a full 24h window (vs the previous 4h) so one
IEM request returns ~24 hourly observations instead of ~1-2. The
effective bytes-per-request is 10x higher, amortizing the
IemRateLimiter gap + 429 retry tail across far more data.
Changes:
- Weather.station_day_covered?/2 + station_day_pairs_covered/1 — cheap
UTC-day coverage checks for the enqueuer + worker skip paths.
- WeatherFetchWorker gains an "asos_day" clause that fetches the full
UTC day for one station and upserts. Skips if the day is already
covered in the DB.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one asos_day
job per (station_id, UTC date) pair. ±2h windows crossing midnight
enumerate both dates.
- Microwaveprop.Weather.RebatchAsos.to_day_jobs/1 migrates any
already-queued asos/asos_batch jobs into the new shape. Idempotent;
supports dry_run.
- asos_batch worker clause retained for jobs already in flight at
deploy time.
2835 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Pure-function invariants that example tests can't express at scale:
- Weather.round_to_iemre_grid/2, HrrrClient.nearest_hrrr_hour/1,
NexradClient.round_to_5min/1, NarrClient.snap_to_analysis_hour/1:
each is idempotent, its output satisfies a grid/slot-membership
predicate, and lies within a bounded distance of the input.
- NarrClient.in_coverage?/1: coverage_end is an exclusive upper
bound; any timestamp after it is out of coverage.
- IemreFetchWorker.backoff/1: result is in [120, 21_600], monotonic
non-decreasing in attempt, equals 120*2^(attempt-1) pre-cap.
- Contact submission_changeset notes field: every non-blank string
up to 2000 chars validates; any longer string is rejected.
22 new properties (139 → 161 including existing); all 2,300 tests
green, credo clean.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
Three bugs were letting mechanism/terrain/radar contacts ping-pong
between :complete and :queued on every backfill cron:
1. enqueue_for_contact/2 now filters out types that are already
:complete on the contact before building jobs. Previously
mark_status!/3 demoted every non-empty jobs_by_type entry back to
:queued, overwriting the :complete that workers had just set.
2. reconcile_stale_queued/1 now also flips mechanism_status and
radar_status back to :complete when the companion data
(propagation_mechanism / contact_common_volume_radar) already
exists. Drains the existing ~14k row backlog on the first post-
deploy tick.
3. IemClient.parse_iemre_json/1 accepts "" and nil so a 200-with-
empty-body IEMRE response doesn't FunctionClauseError the worker
through four retry attempts.
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.
Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.
Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)
Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
Wraps the hourly grid_tasks seed with
[:microwaveprop, :propagation, :grid_worker, :perform] so Prometheus
can see duration, success, and failure for the chain-seed cron.
Emits a :seeded event with the queued_tasks count and an :exception
event on GridTaskEnqueuer failure so silent seed errors show up in
the existing PromEx dashboards instead of dying as a lone Logger.info.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.
- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
errors using the same pattern as RadarFrameWorker. Permanent errors
still mark :unavailable + :ok so the backfill stops re-queueing them.
Transient errors return {:error, reason} and leave radar_status alone
so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
operators can see which contacts blew up, but return
{:error, inspect(e)} from the rescue instead of swallowing it as :ok.
- MapLiveTest timestamp indicator: earlier tests in the suite leave
score files in the shared `:propagation_scores_dir` and warm the 5 s
`list_valid_times` cache; clear both in setup so the "No data"
assertion sees the state the test documents.
- PropagationPruneWorkerTest: `old = now - 3h` collides with the
worker's cutoff (also `now - 3h`) once both are truncated to the
second, so they land equal when test + worker run in the same wall
clock second. Shift `old` by an extra -1s for unconditional ordering.
- ContactWeatherEnqueueWorkerTest: flip to async: false — perform/1
runs inline Oban child jobs that upsert rows with a shared conflict
target, and parallel DataCase siblings hit "ShareLock deadlock_detected"
intermittently.
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):
- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
Previously it called ScoresFile.read_bounds/3 which silently returns
[] on missing/corrupt files, poisoning ScoreCache with an empty grid
that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
pattern-match {:error, reason} and skip (log, return :error) instead
of caching empty. rescue clauses kept as defense-in-depth for
unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
need to distinguish missing file from empty grid can feed read/2
payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
helper (Map.fetch) so a legitimate persisted ducting_detected: false
is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
guard so callers with partial contacts get a clean false rather
than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
and wrapped in try/rescue with per-row fallback on Postgrex errors
so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
explicit {:error, _} pattern match). FM5 clarifies the surviving
PropagationGridWorker is a cron-fired seed worker, not a fallback
compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
:unmatched_returns, :extra_return, :missing_return). Baseline
emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
returns; a follow-up will tighten those and backfill @specs.
New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
error string contract so permanent_error?/1 classification doesn't
silently regress if the client's error format changes.
1. GridCache: auto-release fill lock when the claimer process crashes.
claim_fill/1 + release_fill/1 go through the GenServer so the
server can Process.monitor the caller and clean up the ETS entry
on :DOWN. Clear/0 now resets both the data table and the lock
table. Fixes a latent bug where a crashed fill leaked the lock
indefinitely, preventing every subsequent /weather mount for that
valid_time from claiming and leaving cache cold.
2. RadarFrameWorker: distinguish permanent vs transient fetch errors.
404 from the IEM n0q archive is permanent (file will never exist)
and marks contacts :unavailable as before. Any other error shape
(5xx, timeout, transport failure) now returns {:error, reason}
so Oban retries — previously those also pinned contacts at
:unavailable after a transient outage.
3. AdminTaskWorker.native_derive: replace per-row Repo.update_all
(N round-trips + N fsyncs) with one UPDATE ... FROM unnest(...)
per 2000-row batch. For the 10k-profile budget this is one
network round trip per chunk instead of 10k, and one fsync per
chunk instead of 10k. Restructured the clause to separate
derivation (pure) from persistence (I/O).
All three changes are test-covered (grid_cache_test auto-release
test, radar_frame_worker_test 5xx + transport tests, existing
admin_task_worker_test native_derive coverage exercises the new
bulk path). Also drops the scorer_diff no-op test that was
verifying the clause removed in 61da51c.
Column name now matches the type key used everywhere else:
:hrrr -> hrrr_status, :weather -> weather_status,
:terrain -> terrain_status, :mechanism -> mechanism_status
Drops the @status_column_overrides remap in BackfillEnqueueWorker
(status_column/1 is now a straight :"#{type}_status" derivation) and
the long-form references in Contact, ContactWeatherEnqueueWorker,
and MechanismClassifyWorker. Migration is a metadata-only RENAME
COLUMN + RENAME INDEX — safe to land in prod (no table rewrite,
brief catalog lock only).
hrrr_placeholder_jobs previously returned [contact.id] for any
post-2014 contact with pos1, regardless of whether the HRRR
profiles were already in the DB. That meant BackfillEnqueueWorker
re-flagged the same ~22k contacts as :queued on every 30 min
cron tick forever — HrrrPointEnqueuer saw 0 missing points and
inserted nothing, but the status field never advanced.
Now the placeholder also returns [] when every path point has
an hrrr_profile row, so mark_hrrr_status!/3's empty-list branch
flips the contact to :complete directly. Genuinely-missing
contacts still flow through HrrrPointEnqueuer unchanged.
The Contact list's 'Partial' badge was sticky: once a worker gave up on
a contact (upstream 404, no nearby station, IEM throttling) or the
stale-queued reconciler flipped it to :unavailable after 3 days, the
backfill cron never looked at it again. Upstream archives heal over
time — IEM backfills gaps, new ASOS stations come online, Rust worker
gains capability — so rows stayed 'Partial' that shouldn't have.
Changes:
* BackfillEnqueueWorker.type_filter now also matches <status>_status =
:unavailable when contacts.updated_at is older than a 24 h cooldown.
enqueue_for_contact is idempotent: if there's still nothing to
fetch it'll flip the status to :complete on the empty-jobs branch;
if new data shows up it builds fresh jobs.
* reconcile_stale_queued_to_unavailable now bumps updated_at when
flipping to :unavailable so the cooldown starts fresh — otherwise
the main loop would pick it right back up on the same cron tick
(stale updated_at < any reasonable cooldown) and immediately
transition :unavailable → :complete, defeating the :unavailable
signal entirely.
Also:
* Drop ScoresFile.point_score/3, dead since read_point moved to the
pread fast path in commit bd3b114.
* Clear Microwaveprop.Cache in WeatherGridTest setup. The new
ProfilesFile read cache is keyed by (base_dir, valid_time); tests
in this file reuse DateTime.utc_now() truncated to seconds and
share the default base_dir, so a prior test's cached read leaks
into the next one's expectations.
Contacts lacking pos2 had terrain/radar/mechanism stuck on :pending, so
BackfillEnqueueWorker re-scanned them every cron cycle. Mark these
statuses :unavailable so the enqueue filter excludes them; a later pos2
edit resets them back to :pending via Radio.reset_enrichment_statuses/2.
BackfillEnqueueWorker re-enqueued every :queued contact each cycle, and
RadarFrameWorker had no unique clause, so the same 5-min NEXRAD frame
got scheduled repeatedly with overlapping contact lists. Prod had 9,134
duplicate available jobs across 1,061 distinct frames — 89% waste.
Two coupled changes so contacts aren't stranded by the dedup:
1. Worker now queries eligible contacts by the frame's 5-min bucket
instead of the args' contact_ids list. A contact that flips to
:queued between the enqueue and the job running gets picked up.
2. unique: [frame_ts] on states [available, scheduled, retryable].
:executing intentionally excluded — a contact that flips to :queued
mid-execution can still enqueue a follow-up instead of getting
stuck behind a running job that missed it.
CommonVolumeRadarWorker was enqueuing one job per contact — every
job fetched + decoded the ~5 MB n0q PNG for its own 5-min frame.
At current backlog depth that's 17,588 contacts across just 1,747
distinct 5-min frames, i.e. ~10 contacts per frame paying for the
same PNG decode over and over.
New RadarFrameWorker takes a batch of contact_ids sharing one frame,
fetches + decodes the frame ONCE, then walks the in-memory pixel
buffer per contact. build_radar_jobs/1 in ContactWeatherEnqueueWorker
now groups the input contacts by their rounded 5-min timestamp and
emits one RadarFrameWorker job per frame instead of one
CommonVolumeRadarWorker per contact.
process_frame/4 is public so the same code path is used both by
perform/1 (production fetch) and tests (pre-decoded pixel buffer);
keeps the happy-path test hermetic without hand-crafting a PNG.
CommonVolumeRadarWorker stays in the tree — still used from the
single-contact submit path where batching is pointless and the
aggregate_stats/5 pure helper is a dependency of the new batched
worker.
Expected impact on the backfill: ~10x fewer fetch + decode cycles,
so the 17k queued-contact backlog drains in minutes instead of ~50.
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
The hourly cron now only seeds grid_tasks. The chain step, native-duct
merge, NEXRAD merge, commercial-link merge, scoring, ProfilesFile
write, and band-score writes all moved to rust/prop_grid_rs.
Elixir changes:
- GridTaskEnqueuer.seed_with_analysis/1: inserts 1 kind='analysis' row
(f00) + 18 kind='forecast' rows (f01..f18).
- PropagationGridWorker: stripped from 423 LOC to a thin seeder.
perform(%{}) → GridTaskEnqueuer.seed_with_analysis.
Deleted: process_forecast_hour, merge_native_duct_data,
merge_nexrad_data, merge_commercial_link_data, compute_scores_*,
persist_profiles, record_run_timing (Rust emits spans to Prometheus
instead), apply_nexrad_observations, apply_duct_grid, timed helpers.
Test rewritten for the new shape: 0 Oban fan-out jobs, 19 grid_tasks
rows with the expected kind distribution.
HrrrNativeClient and NexradClient remain — they have other callers
(HrrrNativeGridWorker for per-QSO duct batch; NexradWorker and
CommonVolumeRadarWorker for per-contact radar). Only f00's direct
use moved.
MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an
MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer,
AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime).
Net effect: 30 GRIB2 decodes/hour of dead work on hot pods.
Removed:
- MrmsFetchWorker, MrmsClient, MrmsCache and their tests
- cron entries in config.exs, dev.exs, runtime.exs
- mrms_req_options stub in test.exs
- MrmsCache from supervision tree in application.ex
- stale MRMS references in PropagationGridWorker docstring and test
Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks
the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port)
follow-on work.
Rust `prop-grid-rs` has been validated end-to-end on talos5 on the
streamed-bands image (main-1776635096-6d91461): real chain steps
complete in ~7 s each and the peak-heap refactor landed.
Changes:
- `PropagationGridWorker.seed_chain` now enqueues a single f00 Oban
job instead of f00–f18. The f00 analysis-hour keeps its enrichment
(native-level duct, NEXRAD composite, commercial-link boost,
ProfilesFile write) and the GridCache broadcast.
- `GridTaskEnqueuer.seed/1` is called again so grid_tasks gets the
18 forecast-hour rows the Rust worker claims.
- Test asserts only one Oban job is enqueued and the matching 18
grid_tasks rows land in the DB.
- `deployment-grid-rs.yaml` flips `PROP_SCORES_DIR` from
`/data/scores_shadow` to `/data/scores` so Rust writes to the live
score tree. No file collision — Elixir only writes the f00 files.
The hot pod memory shrink (6 Gi → 1 Gi) is deferred until one full
hourly cycle runs clean under the new split.
The :score_band span was reading point_count via length(grid_data),
but grid_data is a %{{lat, lon} => profile} map. Every forecast-hour
job was crashing with ArgumentError before any scores landed, leaving
the map without current-hour forecasts. Swap to map_size/1.
Two wins on the hourly propagation pipeline:
1. Parallelize the chain. seed_chain was enqueuing only f00 and
each step self-enqueued f00+1, serializing ~2.5 min × 19
forecast hours into a ~48 min chain. Fan out all 19 jobs at
once — they're genuinely independent (different HRRR URLs,
different output files) — and let queue concurrency (2 slots
× 3 pods = 6 parallel workers) drop wall time to ~10 min.
Side effect: one permanently-failing step no longer takes out
the rest of the chain, so the rescue-chain logic I added last
commit becomes unnecessary. Removed.
The :final cleanup (retain_window + purge) used to run on the
fh=18 transition; with parallel execution we don't know which
step finishes last. Cleanup now relies on the existing
PropagationPruneWorker 15-min cron for time-based pruning.
Stale chain leftovers are already a minor concern with the
15-min prune; if file bloat becomes real, PruneWorker can
gain retain_window later.
2. Hoist band-invariant factors. score_time_of_day, score_sky,
score_wind, score_pressure depend on conditions only, not the
band — but composite_score was recomputing them 17 times per
point. Added Scorer.precompute_band_invariants/1, called once
per point before the bands loop; composite_score uses the
cached values when present and falls back to computing for
any caller that doesn't pre-warm (test harness, path
integrator). Saves ~30% of the scoring inner loop.
Telemetry showed ~66 PropagationGridWorker exceptions per 6h with
55 ArgumentErrors and 11 TimeoutErrors, producing ~13 discarded
chain steps. Each discard broke the chain: subsequent forecast
hours were never enqueued, leaving the score store with huge gaps
(e.g. at 14:11 UTC the earliest available forecast was 18:00,
because f00-f05 all failed somewhere upstream and nothing ran
after them).
Three changes:
1. PropagationGridWorker: on the final attempt, still enqueue
fh+1 even when this step failed. Oban discards the current
job normally — but the rest of the chain keeps running, so
one bad hour doesn't take out the remaining 12-18. The
rescue is factored into a tested public helper.
2. HrrrClient.parse_idx: skip malformed idx lines instead of
raising. NOAA S3 occasionally serves an HTML error page as
the idx body, and the old strict String.to_integer path
raised ArgumentError on the first non-numeric line and took
down the chain step. This is the root cause of the 55
ArgumentErrors.
3. JS renderTimeline: when no forecast hour is at-or-before
wall-clock (all times are future — the gap scenario the
fixes above are designed to prevent), stop labeling the
earliest future slot "Now". Lets the user see honest
"+Nh" offsets instead of a lie on the pill.
The :propagation queue (2 slots) is shared by PropagationGridWorker,
MrmsFetchWorker (every 2 min), and PropagationPruneWorker (every
15 min). Oban dispatches priority-then-FIFO, so a post-deploy
MRMS/prune backlog could delay the hourly chain step until the
siblings drained.
Set PropagationGridWorker priority: 0 explicitly and drop the two
siblings to priority: 5 so new chain steps always jump ahead.
Backfill workers live on separate queues (:hrrr, :weather, :terrain,
:iemre, :narr, :radar, :mechanism) and already don't interact with
the predictor.
Wires the GEFS fetch pipeline into the existing scoring machinery:
- GefsFetchWorker runs Propagation.score_grid_point over every
fetched grid cell and persists the result via replace_scores/2, so
Day 2-7 valid_times show up on the map through the same
Propagation.scores_at/3 path HRRR uses
- Empty-args perform/1 seeds the chain by enqueueing f024-f168 (25
jobs at 6-hour cadence) for the most recent GEFS run that should
be published given NOMADS' ~3-4h publication lag
- Cron: seeds at 05:30, 11:30, 17:30, 23:30 UTC — 5 hours after each
00/06/12/18Z run
- GefsClient.build_profile now emits :wind_u / :wind_v atom keys to
match the scorer's input shape; DB columns keep the *_mps suffix
for unit clarity, remapped at write time
GEFS pgrb2a lacks HPBL, native-level duct data, NEXRAD, and
commercial-link degradation — the extended-horizon score is a
rougher signal than the HRRR-driven one but covers the window HRRR
can't reach.
- GefsClient.build_profile/1: converts raw wgrib2 output into the
scorer-shaped profile, deriving Td from RH via Magnus at both the
surface and each pressure level
- GefsClient.fetch_grid_profiles/3: downloads the idx sidecar from
NOMADS, byte-ranges only the surface messages, hands the slim
GRIB2 binary to wgrib2 -lola for bilinear regridding from 0.5°
onto the CONUS 0.125° grid
- GefsFetchWorker: one Oban job per (run_time, forecast_hour),
upserts into gefs_profiles and runs SoundingParams.derive so the
same refractivity/ducting metrics flow through
Oban gets a new :gefs queue (2 slots dev/config, 1 slot per prod
pod). Test env gets a gefs_req_options Req.Test stub slot so future
integration tests can drive the full fetch path. No cron wiring or
scorer integration yet — that's Step 3.
Audit pass found two real issues (most agent-flagged items were false
positives — the hrrr_native_profiles composite index exists,
pwat_mm/bl_depth_m are already in the path conditions map, and the
/1.0 in MechanismClassifier is the standard Elixir int→float coerce).
1. Path calculator's 9-point HRRR sampling was running sequentially,
~9× the latency of a single query. Task.async_stream with
max_concurrency: 9 makes them concurrent so path calc pays roughly
one query's worth of time instead of nine.
2. MechanismClassifyWorker's meteor-shower window used hardcoded 2024
peak dates with `%{date | year: peak.year}` projection. That broke
across year boundaries — a Dec 30 contact couldn't match a Jan 4
Quadrantids peak even in-range of the ±3 day window. Switched to
{month, day} tuples and check the peak projected onto year N-1, N,
N+1 so the window works correctly near Dec 31 / Jan 1.
Classifies every contact's likely non-LOS propagation mechanism and
persists the result on contacts.propagation_mechanism. Mechanism is
determined in priority order:
1. user_declared_prop_mode (ADIF PROP_MODE from the operator log)
2. EME — moon-ephemeris check, ≥2m band, >1800 km path
3. aurora — Kp≥5 + high-lat path, 50-432 MHz
4. sporadic-E — foEs × 5 ≥ band_mhz, 400-2500 km path
5. meteor_scatter — ±3 days of a shower peak, VHF/UHF
6. rain_scatter — common-volume radar heavy rain, 5-11 GHz ≤800 km
7. tropo_duct — HRRR native_best_duct ≥ band or ducting_detected
8. line_of_sight — ≤50 km path
9. troposcatter — default
Persisted via MechanismClassifyWorker (queue: :mechanism, unique on
contact_id). Submit-time enqueue path includes :mechanism by default;
BackfillEnqueueWorker cron now handles :mechanism alongside existing
types so prod continuously backfills any contact with
propagation_mechanism_status in (:pending, :queued, :failed). Also
added :radar to the cron's type list so common-volume radar backfill
runs automatically rather than only via `mix radar_backfill`.
New modules:
- Microwaveprop.Propagation.MoonEphemeris — Meeus low-precision moon
position, accuracy ±1° — enough for the mutual-visibility EME test
- Microwaveprop.Propagation.MechanismClassifier — plug-in priority
chain over the evidence map
- Microwaveprop.Workers.MechanismClassifyWorker — assembles inputs
from HRRR / native profiles / common-volume radar / solar_indices /
ionosonde + calls the classifier
ADIF importer now reads PROP_MODE into user_declared_prop_mode so
operator-tagged mechanisms (EME/ES/MS/RS/AS/AUR) become ground truth.
apply_valid_rows only matched {:ok, _} and {:error, %Changeset{}}, so when
Radio.create_contact returned {:error, :duplicate, existing} — the race
window where preview flagged a row as new but another writer inserts the
same contact before the worker picks up the chunk — the reduce callback
crashed with FunctionClauseError. Oban retried the chunk, rolling back
the Postgrex transaction and surfacing the DBConnection errors the
operator was seeing.
Treat :duplicate as a silent skip: the row is already in the DB, there
is nothing to insert, and the counter should not double-count the existing
contact as an error either.
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.
- New import_runs table: stores preview rows + running counters (total,
processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
its slice via CsvImport.commit_rows/1, atomically increments counters
in one update_all, flips status, and broadcasts progress on
'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.
Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.
LiveView UI for /imports/:id is a separate commit.
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).
- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
(was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
Backtest branch's unknown-feature guard was dead code: the if block's
{:error, _} return was discarded and execution fell through, running the
backtest with a just-created atom and writing a report for nonsense
features. Two compounding issues:
- function_exported?/3 returns false for modules not yet loaded in the
BEAM, so even valid feature names triggered the warning path on a
cold VM.
- String.to_atom/1 on caller-supplied strings created unbounded atoms.
Fix: Code.ensure_loaded?(Features) before the check, String.to_existing_atom/1
with rescue (Features exports __info__(:functions) so every valid function
atom already exists), and an explicit early return via a with pipeline so
the error tuple actually leaves the worker.
Same fix as rtma_client: MrmsClient had a hardcoded req_options/0 —
switched to a Keyword.merge with Application.get_env so tests can stub
HTTP. 13 new characterization tests: cache put/get/clear/broadcast and
worker listing/download/up-to-date paths. Full GRIB2 parse happy path
deferred to the wgrib2 coverage task.
RtmaClient was the only weather client without a Req.Test plug seam —
added the standard req_options() helper matching NarrClient/HrrrClient/
etc. so its HTTP calls can be stubbed in test.
17 characterization tests: URL construction, idx fetch 404/503, hour
truncation, malformed idx, range-GET error propagation, worker's
existing-row short-circuit, client-error propagation. GRIB2 byte
parsing paths deferred to the wgrib2 coverage task.
