Four changes sized by measured prod telemetry (83m of spans):
1. propagation queue: 2 → 1 slot per pod. Two concurrent forecast-hour
steps per pod stacked HRRR grid + native duct grid + scored band
map into ~5-6 GiB RSS, OOM-killing every ~15 min. 3-way parallelism
cluster-wide still finishes the chain inside the hourly interval.
2. weather queue: 3 → 1 slot per pod. ASOS backfill was 429-thrashing
IEM (1,296 retryable jobs; logs were nothing but 429 backoffs).
3. PropagationGridWorker: skip native-level duct fetch on f01..f18.
At ~7-11 min/fh and 18 forecast hours, this was the largest single
cost per chain. Forecast hours fall back to
derived[:min_refractivity_gradient] from the pressure-level
profile. f00 still gets full native-level duct analysis.
4. HrrrClient.download_grib_ranges_to_file: parallelize with
Task.async_stream (max_concurrency 8). The file-backed variant was
sequential, dominating native-duct fetch time on the remaining f00
path. ~20s → ~3s per call.
Telemetry showed ~66 PropagationGridWorker exceptions per 6h with
55 ArgumentErrors and 11 TimeoutErrors, producing ~13 discarded
chain steps. Each discard broke the chain: subsequent forecast
hours were never enqueued, leaving the score store with huge gaps
(e.g. at 14:11 UTC the earliest available forecast was 18:00,
because f00-f05 all failed somewhere upstream and nothing ran
after them).
Three changes:
1. PropagationGridWorker: on the final attempt, still enqueue
fh+1 even when this step failed. Oban discards the current
job normally — but the rest of the chain keeps running, so
one bad hour doesn't take out the remaining 12-18. The
rescue is factored into a tested public helper.
2. HrrrClient.parse_idx: skip malformed idx lines instead of
raising. NOAA S3 occasionally serves an HTML error page as
the idx body, and the old strict String.to_integer path
raised ArgumentError on the first non-numeric line and took
down the chain step. This is the root cause of the 55
ArgumentErrors.
3. JS renderTimeline: when no forecast hour is at-or-before
wall-clock (all times are future — the gap scenario the
fixes above are designed to prevent), stop labeling the
earliest future slot "Now". Lets the user see honest
"+Nh" offsets instead of a lie on the pill.
Telemetry puts nexrad_decode_png at 1.76 s/call × 37,847 calls/h
— ~18.5 h of CPU across the cluster every real hour, the single
biggest CPU consumer in the system. The map-click path
(fetch_rain_cells) already cached decoded frames by 5-min
bucket, but the per-contact CommonVolumeRadarWorker path
(fetch_decoded_frame) went straight to network + decode on every
call. Backfill means many contacts share a 5-min window, so the
same 66 MB frame was being decoded dozens of times.
Wire fetch_decoded_frame through NexradCache keyed on the
rounded timestamp. Add a 20-entry size cap in NexradCache so
backfill processing contacts in random timestamp order can't
grow the ETS table to hundreds of GB. Each frame is ~66 MB, so
20 = ~1.3 GB worst case, well under typical pod memory.
Expected impact: cuts sustained decode load by an order of
magnitude depending on backfill temporal locality; map-click
path is unchanged.
Adds spans to 15 previously-unmeasured hot paths so every question we
might ask while tuning has a histogram to answer it:
External I/O:
- iem.fetch_iemre (gridded weather reanalysis)
- mrms.list_latest / mrms.download (precip radar)
- rtma.fetch_observation
- ncei.fetch_metar (historical 5-min METAR backfill)
- solar.fetch_indices (GFZ solar indices)
- swpc.fetch (SWPC Kp/F10.7/X-ray)
- giro.fetch (ionosonde)
- qrz.request, geocoder.geocode (callsign enrichment)
- srtm.download_tile (terrain tile download + gunzip)
- hrrr.download_grib_ranges (parallel byte-range fetch phase)
Subprocess:
- wgrib2.extract_grid / extract_grid_from_file / extract_grid_from_file_mapped
LiveView hot paths:
- propagation.scores_at (map score fetch + cache hit/miss counter)
- propagation.point_forecast (sparkline)
- propagation.point_detail (click-to-inspect)
- propagation.daily_outlook_at (/map outlook strip)
Worker-level end-to-end:
- worker.terrain_profile
- worker.mechanism_classify
- worker.mrms_fetch
Each event is registered in Microwaveprop.PromEx.InstrumentPlugin as
a Prometheus histogram (default / long buckets as appropriate) plus
a counter for the scores_at cache hit/miss ratio. Prometheus at
10.0.15.25 will start seeing the new series on the next scrape after
deploy.
Previously the CommonVolumeRadarWorker ran silently — no URL logged,
no sign in the logs that it was doing any work. Added info-level
fetch/ok/error logs in NexradClient (same shape as WeatherFetchWorker
RAOB logs) plus start-of-work + ingestion lines in the worker itself.
Unchanged: the existing "no frame" warning and all control flow.
Wires the GEFS fetch pipeline into the existing scoring machinery:
- GefsFetchWorker runs Propagation.score_grid_point over every
fetched grid cell and persists the result via replace_scores/2, so
Day 2-7 valid_times show up on the map through the same
Propagation.scores_at/3 path HRRR uses
- Empty-args perform/1 seeds the chain by enqueueing f024-f168 (25
jobs at 6-hour cadence) for the most recent GEFS run that should
be published given NOMADS' ~3-4h publication lag
- Cron: seeds at 05:30, 11:30, 17:30, 23:30 UTC — 5 hours after each
00/06/12/18Z run
- GefsClient.build_profile now emits :wind_u / :wind_v atom keys to
match the scorer's input shape; DB columns keep the *_mps suffix
for unit clarity, remapped at write time
GEFS pgrb2a lacks HPBL, native-level duct data, NEXRAD, and
commercial-link degradation — the extended-horizon score is a
rougher signal than the HRRR-driven one but covers the window HRRR
can't reach.
- GefsClient.build_profile/1: converts raw wgrib2 output into the
scorer-shaped profile, deriving Td from RH via Magnus at both the
surface and each pressure level
- GefsClient.fetch_grid_profiles/3: downloads the idx sidecar from
NOMADS, byte-ranges only the surface messages, hands the slim
GRIB2 binary to wgrib2 -lola for bilinear regridding from 0.5°
onto the CONUS 0.125° grid
- GefsFetchWorker: one Oban job per (run_time, forecast_hour),
upserts into gefs_profiles and runs SoundingParams.derive so the
same refractivity/ducting metrics flow through
Oban gets a new :gefs queue (2 slots dev/config, 1 slot per prod
pod). Test env gets a gefs_req_options Req.Test stub slot so future
integration tests can drive the full fetch path. No cron wiring or
scorer integration yet — that's Step 3.
Lays the groundwork for a Day 2-7 propagation outlook driven by NCEP
GEFS ensemble-mean output, picking up where HRRR's 18-hour horizon
leaves off.
- GefsClient: NOMADS URL builder, forecast-hour list (3h to +240,
6h to +384), ensemble-mean message inventory, and a Magnus
dewpoint derivation (pgrb2a publishes RH rather than Td)
- gefs_profiles table + GefsProfile schema mirroring hrrr_profiles
so the scorer can run over rows from either source
- Weather.upsert_gefs_profile/1 and upsert_gefs_profiles_batch/1
No worker or UI yet — those land in follow-up commits.
parse_cdo_row/1 assumed cdo -outputtab,code,lev,value would always emit
numeric codes in the first column, but Debian's cdo 2.5.1 (the version
in the prod container) emits mixed output:
- 'codeNN' with a literal 'code' prefix for most GRIB1 parameters
- Short GRIB1 names for a few: 2tag2 (PRES sfc), saip (DPT 2m),
tpag10 (HGT)
Result: every row fell through to :error, yielding a 'no parseable
values' error and 986 discarded NarrFetchWorker jobs.
Fix: normalize the code token by stripping 'code' prefix when present
and mapping known 2.5.1 short names. UGRD/VGRD intentionally omitted
(no discard evidence yet) — comment documents the add-when-they-surface
plan.
Also exposed parse_cdo_outputtab/1 as @doc false for testability.
build_messages_per_message/3 was slicing the wgrib2 -lola bin output as
a packed float32 stream, ignoring the 4-byte record-length header + 4-byte
trailer that Fortran unformatted records prepend/append around each
message's data block. First value of message 0 came back as ~2.24e-44
(little-endian float32 of the int-16 record length), and every subsequent
message was shifted by 8 bytes so values spilled into neighbouring cells.
Fix: propagate the correct offset math already used by parse_lola_binary/3
(record_overhead = 8, stride = bytes_per_message + 8, data_offset =
msg_idx * stride + 4).
Flipped the characterization test from metadata-only to per-cell physical
asserts (200 K < TMP, DPT < 340 K; DPT <= TMP) and a 0.01 K cross-check
against extract_grid/3 output on the same fixture + bbox. Latent bug —
no production callers of extract_grid_messages*.
Same fix as rtma_client: MrmsClient had a hardcoded req_options/0 —
switched to a Keyword.merge with Application.get_env so tests can stub
HTTP. 13 new characterization tests: cache put/get/clear/broadcast and
worker listing/download/up-to-date paths. Full GRIB2 parse happy path
deferred to the wgrib2 coverage task.
RtmaClient was the only weather client without a Req.Test plug seam —
added the standard req_options() helper matching NarrClient/HrrrClient/
etc. so its HTTP calls can be stubbed in test.
17 characterization tests: URL construction, idx fetch 404/503, hour
truncation, malformed idx, range-GET error propagation, worker's
existing-row short-circuit, client-error propagation. GRIB2 byte
parsing paths deferred to the wgrib2 coverage task.
Prod cdo 2.5.1 (Debian) emits named shortnames (tpag10, 2tag2, saip, code11…)
where cdo 2.6.0 (Mac) emits var11/var33/etc. The parser only recognized
varNN, so every prod NARR fetch failed with "no parseable values in cdo
output" and all 350 in-flight jobs discarded.
Switch cdo args from `-outputtab,name,lev,value` to `-outputtab,code,...`.
cdo then outputs the raw numeric NCEP GRIB1 parameter code regardless of
version, and @cdo_code_to_name maps that integer to the semantic name we
already use downstream (TMP/HGT/SPFH/…).
Verified by reproducing on both a 2010 NARR file (fixture) and a 2007 file
fetched live — both now emit identical numeric code rows.
Prod hit "no parseable values in cdo output" for a 2007 record.
Without the actual cdo output it's guesswork whether it's a different
parameter-table format, an empty result, or something else. Capture
stderr (stderr_to_stdout: true) and include first 500 chars of output
so the error message has enough signal to diagnose.
Post-2014 contacts with hrrr_status=:unavailable were being dispatched to
NARR, which 404s because NCEI's archive ends 2014-10-02. 14K of the 14.3K
candidates in prod are post-2014 — these are HRRR's responsibility.
- NarrClient.in_coverage?/1 returns true only inside 1979-01-01 → 2014-10-02
- narr_jobs_for_contact and maybe_enqueue_narr (contact_live) bail out of
coverage returning []
- BackfillEnqueueWorker.type_filter scopes :narr to qso_timestamp <
coverage_end so the cron doesn't keep picking post-2014 candidates
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.
Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs
Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
(the symbol still means "historical backfill", just routed to NARR now)
Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).
Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
extract_profile_from_file shells out to cdo:
cdo -outputtab,name,lev,value -remapnn,lon=X_lat=Y <merged.grb>
parses the text output (varNNN ↔ NCEP GRIB1 param number lookup),
maps surface records to era5_profiles fields, builds the pressure
profile from each (TMP, HGT, SPFH) triple, and merges in the derived
fields from SoundingParams.derive/1. Returns the same attrs shape as
Era5BatchClient.build_profile_attrs/5 so the existing
bulk_insert_profiles path can ingest NARR rows unchanged.
fetch_profile_at picks records from a fetched inventory, byte-range
fetches each one into a per-record temp file, runs cdo -merge to build
a composite, calls extract_profile_from_file, and cleans up via
try/after.
Drive-by fix: ThetaE.dewpoint_from_spfh/2 returns Kelvin (its existing
test asserts that explicitly). Era5BatchClient was passing that
Kelvin value straight into "dwpc" (Celsius), which would then crash
SoundingParams.compute_refractivity_profile when it called Buck's
saturation-vapor-pressure equation with t=294 instead of t=21. The
bug never surfaced because era5_profiles is empty in production —
no ERA5 backfill has ever succeeded. Fixed in both era5_batch_client
and the new narr_client by subtracting 273.15 in the wrapper. Both
wrappers now have a comment pointing at the K→C conversion.
Test fixture test/fixtures/narr/narr_dfw_2010-06-15_12z.grb is the
spike-captured composite (1.1 MB — Lambert conformal projection
isn't sellonlatbox-subsettable so we can't shrink it further).
parse_inventory walks the wgrib1-format .inv text and returns a
{var, level} -> {byte_offset, length} index, computing each record's
length from the next record's offset (or file_size for the last one).
fetch_inventory does a GET on the .inv URL plus a HEAD on the .grb URL
to learn its size, then delegates to parse_inventory.
Both stubbable via Req.Test (config/test.exs adds narr_req_options
matching the existing era5_req_options / giro_req_options pattern).
Spike fixture at test/fixtures/narr/narr-a_221_20100615_1200_000.inv
backs the parser test — verifies the ("TMP", "2 m above gnd") and
("WVVFLX", "atmos col") records line up with the real bytes.
First slice of the NARR backfill client: url_for/1,
url_for_inventory/1, and snap_to_analysis_hour/1. Pure functions,
no network. Validates that NARR analyses only exist at 3-hourly
slots (00/03/06/09/12/15/18/21 UTC). The byte-range fetch and
cdo-driven point extraction land in follow-up commits per
docs/plans/2026-04-15-merra2-historical-backfill.md.
The skew-T commit (30c1018) doubled @pressure_levels from 13 to 25 so
new contact fetches would cover the full troposphere. That list is
also what PropagationGridWorker pulls per forecast hour, which
doubled the GRIB footprint (~57 MB compressed + 92k points × 25
levels × 3 vars decoded through wgrib2) and pushed prod pods over
their 4 Gi OOMKill threshold. Every chain died during f00 and the
map timeline never got beyond now and now+1h because the .ntms files
for f02-f18 were never written.
Split the constant:
* @profile_pressure_levels (25 levels, 1000-100 mb) drives the
per-contact HrrrClient.fetch_profile path so the skew-T plot
keeps its full-atmosphere trace.
* @grid_pressure_levels (13 levels, 1000-700 mb) drives the grid
hot path. That's the band SoundingParams.derive reads for
min_refractivity_gradient, and native hybrid-sigma data
(native_min_gradient) takes priority over the pressure-level
fallback anyway, so upper-air levels contribute nothing to
scoring — pure memory waste on this path.
build_profile/1 still iterates the full 25-level list; grid fetches
simply populate the 13 near-surface slots and skip the rest.
Makes HrrrClient.pressure_messages public with a :grid | :profile
variant so the split is testable from outside the module.
Historical contacts showed a skew-T log-P diagram that stopped at 700 mb
because HrrrClient and Era5Client only fetched pressure-level data down
to the top of the boundary layer. The chart canvas already ran up to
100 mb, so the trace clipped mid-atmosphere.
Two complementary fixes:
1. Extend @pressure_levels in HrrrClient and Era5Client with
650/600/550/500/450/400/350/300/250/200/150/100 mb so new fetches
cover the full troposphere + lower stratosphere.
2. Prefer the native hybrid-sigma profile for the contact-detail
skew-T when one has been backfilled for the contact's hour. The
native profile already stores all 50 hybrid levels up to ~19 km,
so historical contacts covered by the native backfill get a full
trace without re-hitting S3. A new HrrrNativeProfile.to_skew_t_profile/1
converts the parallel arrays into the %{"pres","tmpc","dwpc","hght"}
list shape the renderer expects, deriving dewpoint from SPFH via the
Magnus inverse. Weather.find_nearest_native_profile/3 mirrors
find_nearest_hrrr/3 for the lookup.
CDS returns status="rejected" (distinct from "failed") when a submit
is rate-limited past the 150-job per-user cap. The poll worker's
interpret_status_body treated this as an unknown status → generic
retryable error → 10 retries → discarded. 102 jobs hit this path
overnight.
Fix:
* Era5Client.check_status/1 now returns {:rejected, reason} (and
treats "dismissed" the same way — that's the status a manually
deleted job transitions to).
* Era5PollWorker handles :rejected exactly like :not_found: drop the
DB row, clean up the sibling CDS job, re-enqueue the submit, and
discard the current Oban job so it doesn't burn attempts retrying
a terminal state.
* Widened Era5SubmitWorker cap headroom from 10 → 30 (effective
ceiling 120 not 140). The race between concurrent workers all
passing the count check simultaneously meant we were reaching 141
in-flight against the 150 hard cap; 30 slots of headroom tolerates
a ~15-worker race before clipping CDS's ceiling.
Refactored handle_row/1 into handle_row + handle_non_both_done with a
leg_outcome/1 tag helper so the cross-product of leg statuses collapses
to a single pair match (credo complexity limit).
When CDS returns 404 for a poll check, the job is gone (manually
deleted, reaped, or never existed). Retrying will always 404, so treat
it as terminal: drop the era5_cds_jobs row, delete the sibling CDS job,
re-enqueue Era5SubmitWorker for the tile-month, and discard the Oban
poll job. Fixes 30 retryable poll jobs left behind by the interrupted
manual-cleanup script.
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Four independent wins from a perf audit of the era5_batch worker path:
1. Parallel CDS submits per job. fetch_single_level_month and
fetch_pressure_level_month have no ordering dependency — each blocks
30+ min on CDS queue/download — so running them in two linked Tasks
halves wall time for every job in the backlog.
2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
A month-tile GRIB is 50–200 MB and was previously slurped into the
BEAM heap and then written back out to a temp file before wgrib2 ran.
New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
full parallelism. Partial files are removed on HTTP error.
3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
streamed temp file directly. The binary variant still exists for the
single-point path and now delegates through the same helper.
4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
three-range scan used by Weather.find_nearest_era5 (contact detail +
path profiles) and StatusLive.count_era5_done (status page EXISTS
subquery). The unique index on (lat, lon, valid_time) couldn't serve
it efficiently.
Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
CDS rejects `reanalysis-era5-pressure-levels` requests that include
dewpoint_temperature because it's only a single-level 2m variable; no
pressure-level dewpoint exists in ERA5. The failure mode is a 2–3 second
`{"status": "failed"}` response with no message field, which is why
prod was burning through Era5MonthBatchWorker attempts without anything
landing in era5_profiles.
Switch the pressure-level moisture variable to specific_humidity (SPFH),
update the wgrib2 extraction regex, and convert SPFH → dewpoint °C via
Weather.ThetaE.dewpoint_from_spfh/2 in the profile builder. Surface
dewpoint keeps using the single-level 2m dewpoint_temperature, which is
valid and already working.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
IEM's RAOB endpoint stopped keeping Canadian stations current — most
stalled around Sep 2024 — which leaves a gap in refractivity/ducting
calibration over all the CW*/CY* stations already in weather_stations.
MSC's own tephi CSV is fixed-width and rendering-focused; the WMO TEMP
bulletins would need a full decoder. University of Wyoming serves the
same stations in a clean space-delimited format, is current, and
accepts the 3-letter station code (drop the leading C from Canadian
ICAO). Its output shape matches IemClient.parse_raob_json/1 exactly,
so it drops straight into Weather.upsert_sounding/2.
- New UwyoSoundingClient with URL builder, Req-based fetch_sounding,
and a fixed-width parse_sounding_html that extracts PRES/HGHT/TEMP/
DWPT/DRCT/SKNT from the <PRE> block and DateTime from the <H2> header.
- New CanadianSoundingFetchWorker: Oban batch worker that finds all
sounding stations whose code starts with C, picks the most recently
publishable 00Z or 12Z slot (90 min publish delay), calls UWYO per
station, and upserts with SoundingParams-derived refractivity/
ducting fields.
- Oban cron at 01:30Z and 13:30Z (dev + prod) to drive the worker.
- Req.Test plug wiring in config/test.exs.
- Captured Goose Bay fixture as test/support/fixtures/uwyo_sounding_yyr.html.
Also drops two implementation plans under docs/plans/:
- 2026-04-13-rdps-vertical-profiles.md: ~2 day plan for RDPS 10 km
Canadian ducting outside HRRR's Lambert footprint.
- 2026-04-13-hrdps-canadian-prop-grid.md: ~5 day plan for the full
HRDPS 2.5 km Canadian prop grid. Explicitly recommends doing RDPS
first.
TDD throughout, 19 new tests, 1318/1318 passing, credo strict clean.
MRMS
----
Layer the NOAA MRMS PrecipRate product onto the score grid so rain fade
updates every 2 minutes instead of every hour alongside HRRR. New modules:
- Microwaveprop.Weather.MrmsClient: fetches the latest .grib2.gz off the
NCEP mirror (Req auto-decompresses so no gunzip step), writes the raw
GRIB2 to a temp file, and calls the existing wgrib2 wrapper with the
0.125 propagation grid spec to get interpolated cells. Returns a
%{{lat, lon} => mm_per_hour} map with missing-value sentinels dropped.
- Microwaveprop.Weather.MrmsCache: ETS-backed GenServer mirroring
ScoreCache/GridCache. Caches a single "current" entry keyed by
valid_time with PubSub broadcast so peer nodes stay in sync and only
the Oban leader pays the fetch + regrid cost.
- Microwaveprop.Workers.MrmsFetchWorker: cron every 2 minutes, short-
circuits when the cached valid_time already matches the newest file.
Microwaveprop.Propagation.AsosNudge.compute/4 now takes an optional
rain_grid. When a cell has MRMS rain >= 0.1 mm/hr it gets patched onto
the HRRR profile's `precip_mm` field (the scorer already reads it there)
and the cell is re-scored even with no ASOS station nearby. Cells with
MRMS rain below the threshold aren't touched so dry cells keep their
raw HRRR scores (which have the wind/sky/native-gradient signal that
isn't persisted on HrrrProfile rows and would otherwise be lost).
AsosAdjustmentWorker pulls MrmsCache on every tick and passes the grid
through to AsosNudge.compute/4. Also skips the IemClient error branch
that can never happen and handles the ASOS-empty + MRMS-empty case
explicitly. MrmsCache wired into the supervision tree; MrmsFetchWorker
cron entry added to config.exs and dev.exs.
Four new AsosNudge cases cover MRMS-only re-scoring, threshold gating,
and the wet/dry score delta.
Beacons 500
-----------
Beacon.format_freq/1 and format_mw/1 crashed on whole-number floats
(e.g. 24192.0) because `frac == 0.0` could become false under float
rounding while `trim_trailing_zeros/1` stripped the decimal point,
leaving a 1-element list that couldn't be destructured as [_, frac].
Shared format_number/1 helper handles integer input directly and
pattern-matches both the "int-only" and "int + frac" shapes.
Added stream_data property tests covering the whole microwave range for
both integers and floats to catch this class of bug before prod.
UTC clock flash
---------------
The /weather and /map UTC clocks were empty until the JS hook mounted
post-WebSocket, producing a several-second blank spot on initial load
and a clobber risk on sidebar re-renders. Mount now computes a
server-rendered `initial_utc_clock` string and the template seeds the
element with that plus `phx-update="ignore"` so LiveView morphdom won't
overwrite what the hook writes.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
The /weather LiveView was timing out on pod restart because
latest_weather_grid/1 ran load_weather_grid_from_db/1 synchronously on
cache miss — that query reads 176k hrrr_profiles rows with two huge
JSONB columns (profile array + duct_characteristics), and JSONB
decoding blocked the LiveView process past the 15-second pool
ownership timeout.
On cache miss latest_weather_grid/1 now returns empty immediately and
kicks off a deduped background fill through GridCache.claim_fill/1 (a
per-valid_time ETS lock so N concurrent mounts after restart don't each
fire the slow query). When the fill completes it broadcasts
weather:updated, and the handle_info in WeatherMapLive hits the warm
cache and pushes rows over the socket.
Added load_weather_grid/1 as a synchronous sibling for tests and any
caller that genuinely needs inline data. Tests updated to use it and
to clear GridCache in setup so ETS state doesn't leak between cases.
Also untrack k8s/secret.yaml and add both it and k8s/*-secret.yaml to
.gitignore — those manifests hold plaintext SMTP and database
credentials that should not live in the repo. Secrets already in git
history should be rotated separately.
- Add Weather.GridCache: ETS cache of derived HRRR grid rows keyed by
valid_time, cluster-synced via PubSub. Eagerly warmed from
PropagationGridWorker after each upsert so /weather map pan/zoom and
weather_point_detail hit zero DB on warm cache.
- Replace latest_weather_grid DB query path with cache-first lookup +
DB fallback. hrrr_profiles is 42M rows partitioned; pulling 3-10k
rows per viewport on every pan was the main cost.
- ContactLive.Show: defer the heavy enrichment loads (weather, solar,
HRRR, terrain, IEMRE, elevation profile, ITU-R propagation analysis,
data_sources) into a handle_info(:hydrate) that runs after the shell
renders. Initial mount now returns nil placeholders; template already
had :if guards for all of them. Shell-to-first-paint goes from
~500ms-2s down to ~20ms.
- Cache fetch_queue_counts for 5s in ContactLive.Show — oban_jobs group
by query was running on every contact page view.
- Backfill stats: wrap count_unprocessed, fetch_stats, fetch_db_stats
in Microwaveprop.Cache with 2-5s TTLs; bump refresh debounce from 1s
to 2s so bulk enrichment events don't thrash the DB.
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
immediately with a pending marker; push rain_scatter_update when
NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
viewport after mount/band change/propagation_updated; client caches
them and renders timeline scrubs instantly without a server roundtrip.
Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
timeline click uses preloaded cache when available
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
Registered users can suggest edits to any contact's core fields
(callsigns, grids, band, mode, timestamp). Edits enter an admin
approval queue with field-by-field diff view. On approve, changes
are applied and enrichment re-enqueued if grids/band changed.
Users receive email notification on approve or reject.
Also updates dependabot.yml for mix ecosystem.
When clicking a grid point, fetches the latest NEXRAD composite
reflectivity and identifies rain cells within 300 km that could
enable rain scatter contacts. Shows:
- Scatter classification (excellent/good/marginal/none)
- Top 3 cells with dBZ, distance, bearing, and relative signal
- Colored circle markers on the map at rain cell locations
- Markers sized by reflectivity, colored by intensity
Uses simplified bistatic radar equation accounting for reflectivity,
frequency-dependent scattering (Rayleigh/Mie), and R^4 path loss.
NEXRAD cells sampled every ~5 km within bounding box for efficiency.
When clicking a grid point with ducting, the panel now shows each
duct layer with base-top height in feet, thickness in meters, and
minimum trapped frequency. Data flows from Duct.analyze through
the scoring factors as a ducts array.
The PropagationGridWorker now fetches native hybrid-sigma levels
(TMP, SPFH, HGT, PRES × 50 levels) alongside the standard surface
and pressure products. Native data provides 10-50m vertical spacing
vs 250m from pressure levels, detecting thin surface ducts invisible
to the standard product.
Key design: cell-by-cell reducer in Wgrib2.extract_grid_from_file_mapped
processes each of the 95k CONUS cells through a duct analysis function
inline, keeping only scalar metrics per cell. Peak memory ~86 MB
instead of ~1.8 GB for the full grid map.
Per-cell output: native_min_gradient, best_duct_freq_ghz,
max_duct_thickness_m, duct_count. The scorer prefers the native
gradient over the pressure-level gradient when available.
Native fetch is optional — if it fails, scoring continues with
pressure-level data only.
Without -s, wgrib2 -lon only outputs msg:offset:lon=X,lat=Y,val=Z
with no variable name or level. The -s flag adds the short inventory
(d=DATE:VAR:LEVEL:...) so the parser can identify which variable
each value belongs to.
Points spread coast-to-coast created a ~476k cell bounding grid
(350 messages × 476k cells × 4 bytes ≈ 665 MB), causing OOM.
Switch to -lon which extracts values at specific lat/lon points
with text output. One wgrib2 call, one file scan, negligible
BEAM memory regardless of point geographic spread.
- Stream profile storage and score upsert instead of materializing
full 20k+ item lists (propagation_grid_worker, propagation.ex)
- GC between forecast hours and store/compute phases to reclaim
~400 MB of grid data between steps
- Single-pass field extraction in scorer.ex path_integrated_conditions
instead of 6 separate Enum traversals
- Eliminate intermediate merged map in fetch_grid by combining
merge + profile build into one pipe
- Fix UUID bug: bingenerate → generate in native grid worker
(same issue previously fixed in nexrad_worker)
The parallel download was holding all ~530MB of range responses in
memory before writing to disk. Now each range is fetched and written
one at a time, so only one chunk is in memory at a time.
Instead of holding ~530MB GRIB binary in BEAM memory, download
ranges directly to a temp file and run wgrib2 on it. Peak memory
drops from ~530MB to just HTTP chunk buffers.
