The three BackfillEnqueueWorkerTest cases were timing out because the
default types list includes :era5, which cascades through inline Oban
into Era5Client.poll_and_download where Process.sleep blocks for the
full 60s test timeout. Era5Client uses bare Req.get with no plug hook,
so it can't be stubbed via Req.Test the way the other clients are. Pass
explicit non-ERA5 types in the three affected cases — ERA5 has its own
coverage and these tests don't assert anything era5-specific.
Also replace two `length(results) > 0` checks in asos_nudge_test with
`results != []` to silence credo warnings.
MRMS
----
Layer the NOAA MRMS PrecipRate product onto the score grid so rain fade
updates every 2 minutes instead of every hour alongside HRRR. New modules:
- Microwaveprop.Weather.MrmsClient: fetches the latest .grib2.gz off the
NCEP mirror (Req auto-decompresses so no gunzip step), writes the raw
GRIB2 to a temp file, and calls the existing wgrib2 wrapper with the
0.125 propagation grid spec to get interpolated cells. Returns a
%{{lat, lon} => mm_per_hour} map with missing-value sentinels dropped.
- Microwaveprop.Weather.MrmsCache: ETS-backed GenServer mirroring
ScoreCache/GridCache. Caches a single "current" entry keyed by
valid_time with PubSub broadcast so peer nodes stay in sync and only
the Oban leader pays the fetch + regrid cost.
- Microwaveprop.Workers.MrmsFetchWorker: cron every 2 minutes, short-
circuits when the cached valid_time already matches the newest file.
Microwaveprop.Propagation.AsosNudge.compute/4 now takes an optional
rain_grid. When a cell has MRMS rain >= 0.1 mm/hr it gets patched onto
the HRRR profile's `precip_mm` field (the scorer already reads it there)
and the cell is re-scored even with no ASOS station nearby. Cells with
MRMS rain below the threshold aren't touched so dry cells keep their
raw HRRR scores (which have the wind/sky/native-gradient signal that
isn't persisted on HrrrProfile rows and would otherwise be lost).
AsosAdjustmentWorker pulls MrmsCache on every tick and passes the grid
through to AsosNudge.compute/4. Also skips the IemClient error branch
that can never happen and handles the ASOS-empty + MRMS-empty case
explicitly. MrmsCache wired into the supervision tree; MrmsFetchWorker
cron entry added to config.exs and dev.exs.
Four new AsosNudge cases cover MRMS-only re-scoring, threshold gating,
and the wet/dry score delta.
Beacons 500
-----------
Beacon.format_freq/1 and format_mw/1 crashed on whole-number floats
(e.g. 24192.0) because `frac == 0.0` could become false under float
rounding while `trim_trailing_zeros/1` stripped the decimal point,
leaving a 1-element list that couldn't be destructured as [_, frac].
Shared format_number/1 helper handles integer input directly and
pattern-matches both the "int-only" and "int + frac" shapes.
Added stream_data property tests covering the whole microwave range for
both integers and floats to catch this class of bug before prod.
UTC clock flash
---------------
The /weather and /map UTC clocks were empty until the JS hook mounted
post-WebSocket, producing a several-second blank spot on initial load
and a clobber risk on sidebar re-renders. Mount now computes a
server-rendered `initial_utc_clock` string and the template seeds the
element with that plus `phx-update="ignore"` so LiveView morphdom won't
overwrite what the hook writes.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
immediately with a pending marker; push rain_scatter_update when
NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
viewport after mount/band change/propagation_updated; client caches
them and renders timeline scrubs instantly without a server roundtrip.
Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
timeline click uses preloaded cache when available
- Add ScoreCache GenServer with node-local ETS table keyed by
{band, valid_time}, subscribed to "propagation:cache" PubSub topic so
every pod stays in sync with a single hourly compute
- scores_at/3 checks cache first, falls back to DB and populates on miss
- PropagationGridWorker warms and broadcasts the cache for each band
after every forecast hour upsert; prunes >2h old entries
- Replace per-pixel string-keyed Map with flat Int8Array over the CONUS
grid in propagation_map_hook.ts to eliminate allocations in the tile
rasterization hot loop (interpolateScore / propagationReach)
Add Propagation.Region module with 8 CONUS climate zones (gulf_coast,
southeast, southern_plains, corn_belt, northeast, desert_southwest,
pacific_northwest, mountain_west) and per-region monthly seasonal
adjustment multipliers.
The scorer's score_season now takes lat/lon and applies a regional
multiplier from Region.seasonal_adjustment on top of the band's
seasonal_base + seasonal_adj. Gulf coast August gets a 1.15x boost
(drier, better for ducting) while Corn Belt August gets a 0.80x
penalty (corn evapotranspiration = miserable dewpoints).
Adjustments are hand-tuned starting points from the meteorologist's
qualitative guidance. Phase 9 recalibration will refine them from
backtest data.
Duct module (Propagation.Duct):
- refractivity_profile/1: ITU-R P.453 N at each native level
- m_profile/1: modified refractivity M = N + 157*h(km)
- detect_ducts/1: find contiguous regions where dM/dh < 0, returning
base/top height, thickness, and M-deficit per duct
- min_trapped_frequency_ghz/1: waveguide approximation (Bean & Dutton)
for the minimum frequency a duct of given geometry can trap
- analyze/1: full pipeline from native profile to duct list + best
trapped frequency across all ducts
Derive task updated to also compute ducts JSONB and best_duct_band_ghz
alongside the Phase 2 turbulence fields.
Backtest features: duct_thickness, best_duct_freq, duct_usable_10ghz,
duct_usable_24ghz, duct_usable_47ghz.
Real-data validation: 2022-08-20 12Z TX profile shows 0 ducts (M
increases monotonically) — correct for a well-mixed boundary layer
on a turbulent August afternoon (Ri=0.16).
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286
Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric
mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.
First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
New BandConfig entries for 902, 1296, 2304, 3456, 5760 MHz:
- All beneficial humidity effect (like 10 GHz)
- Near-zero gaseous absorption and rain attenuation
- Ranges: 902 MHz typical 400 km, 5760 MHz typical 220 km
- Same seasonal curves as 10 GHz (ducting-driven)
BandConfig.band_options/0 generates dropdown options from configs.
All pages (path, rover, submit) use centralized band_options instead
of hardcoded lists. Map page already used BandConfig.all_bands().
10 GHz remains the default on all pages.
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
normalized distance — reduces noise from operator/equipment variation
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
Previous thresholds (-500 to -60) were calibrated for radiosonde data.
HRRR profiles have coarser vertical resolution, with gradients clustering
between -40 and -130 N/km (median -70). Nearly all grid points were
falling through to the default score of 42, wasting the refractivity
factor. New thresholds (-200 to -40) spread across HRRR percentiles.
Replace circle markers with a canvas tile layer that renders smooth,
flowing colored regions using bilinear interpolation between grid
points. Colors interpolate between tiers for gradients. ~95k grid
points at 0.125 degree resolution with wgrib2 extraction.
95k points at 0.125 degree resolution caused the GRIB2 extraction to
take too long. 0.5 degree (~55 km) resolution gives 6k points which
completes in under a minute. Can increase resolution later once the
extraction is optimized.
Define 0.125-degree CONUS grid (25-50N, 125-66W) for propagation
scoring and create propagation_scores table with composite unique
index on lat/lon/valid_time/band_mhz for upsert support.
Single source of truth for all scoring parameters: weights, thresholds,
seasonal tables, and per-band coefficients for 8 microwave bands
(10G through 241G). Includes ITU-R P.838-3 rain attenuation
coefficients, humidity effects, refractivity scoring thresholds,
and sunrise/tier definitions.
