MRMS
----
Layer the NOAA MRMS PrecipRate product onto the score grid so rain fade
updates every 2 minutes instead of every hour alongside HRRR. New modules:
- Microwaveprop.Weather.MrmsClient: fetches the latest .grib2.gz off the
NCEP mirror (Req auto-decompresses so no gunzip step), writes the raw
GRIB2 to a temp file, and calls the existing wgrib2 wrapper with the
0.125 propagation grid spec to get interpolated cells. Returns a
%{{lat, lon} => mm_per_hour} map with missing-value sentinels dropped.
- Microwaveprop.Weather.MrmsCache: ETS-backed GenServer mirroring
ScoreCache/GridCache. Caches a single "current" entry keyed by
valid_time with PubSub broadcast so peer nodes stay in sync and only
the Oban leader pays the fetch + regrid cost.
- Microwaveprop.Workers.MrmsFetchWorker: cron every 2 minutes, short-
circuits when the cached valid_time already matches the newest file.
Microwaveprop.Propagation.AsosNudge.compute/4 now takes an optional
rain_grid. When a cell has MRMS rain >= 0.1 mm/hr it gets patched onto
the HRRR profile's `precip_mm` field (the scorer already reads it there)
and the cell is re-scored even with no ASOS station nearby. Cells with
MRMS rain below the threshold aren't touched so dry cells keep their
raw HRRR scores (which have the wind/sky/native-gradient signal that
isn't persisted on HrrrProfile rows and would otherwise be lost).
AsosAdjustmentWorker pulls MrmsCache on every tick and passes the grid
through to AsosNudge.compute/4. Also skips the IemClient error branch
that can never happen and handles the ASOS-empty + MRMS-empty case
explicitly. MrmsCache wired into the supervision tree; MrmsFetchWorker
cron entry added to config.exs and dev.exs.
Four new AsosNudge cases cover MRMS-only re-scoring, threshold gating,
and the wet/dry score delta.
Beacons 500
-----------
Beacon.format_freq/1 and format_mw/1 crashed on whole-number floats
(e.g. 24192.0) because `frac == 0.0` could become false under float
rounding while `trim_trailing_zeros/1` stripped the decimal point,
leaving a 1-element list that couldn't be destructured as [_, frac].
Shared format_number/1 helper handles integer input directly and
pattern-matches both the "int-only" and "int + frac" shapes.
Added stream_data property tests covering the whole microwave range for
both integers and floats to catch this class of bug before prod.
UTC clock flash
---------------
The /weather and /map UTC clocks were empty until the JS hook mounted
post-WebSocket, producing a several-second blank spot on initial load
and a clobber risk on sidebar re-renders. Mount now computes a
server-rendered `initial_utc_clock` string and the template seeds the
element with that plus `phx-update="ignore"` so LiveView morphdom won't
overwrite what the hook writes.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
The /weather LiveView was timing out on pod restart because
latest_weather_grid/1 ran load_weather_grid_from_db/1 synchronously on
cache miss — that query reads 176k hrrr_profiles rows with two huge
JSONB columns (profile array + duct_characteristics), and JSONB
decoding blocked the LiveView process past the 15-second pool
ownership timeout.
On cache miss latest_weather_grid/1 now returns empty immediately and
kicks off a deduped background fill through GridCache.claim_fill/1 (a
per-valid_time ETS lock so N concurrent mounts after restart don't each
fire the slow query). When the fill completes it broadcasts
weather:updated, and the handle_info in WeatherMapLive hits the warm
cache and pushes rows over the socket.
Added load_weather_grid/1 as a synchronous sibling for tests and any
caller that genuinely needs inline data. Tests updated to use it and
to clear GridCache in setup so ETS state doesn't leak between cases.
Also untrack k8s/secret.yaml and add both it and k8s/*-secret.yaml to
.gitignore — those manifests hold plaintext SMTP and database
credentials that should not live in the repo. Secrets already in git
history should be rotated separately.
- Add Weather.GridCache: ETS cache of derived HRRR grid rows keyed by
valid_time, cluster-synced via PubSub. Eagerly warmed from
PropagationGridWorker after each upsert so /weather map pan/zoom and
weather_point_detail hit zero DB on warm cache.
- Replace latest_weather_grid DB query path with cache-first lookup +
DB fallback. hrrr_profiles is 42M rows partitioned; pulling 3-10k
rows per viewport on every pan was the main cost.
- ContactLive.Show: defer the heavy enrichment loads (weather, solar,
HRRR, terrain, IEMRE, elevation profile, ITU-R propagation analysis,
data_sources) into a handle_info(:hydrate) that runs after the shell
renders. Initial mount now returns nil placeholders; template already
had :if guards for all of them. Shell-to-first-paint goes from
~500ms-2s down to ~20ms.
- Cache fetch_queue_counts for 5s in ContactLive.Show — oban_jobs group
by query was running on every contact page view.
- Backfill stats: wrap count_unprocessed, fetch_stats, fetch_db_stats
in Microwaveprop.Cache with 2-5s TTLs; bump refresh debounce from 1s
to 2s so bulk enrichment events don't thrash the DB.
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
immediately with a pending marker; push rain_scatter_update when
NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
viewport after mount/band change/propagation_updated; client caches
them and renders timeline scrubs instantly without a server roundtrip.
Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
timeline click uses preloaded cache when available
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
Registered users can suggest edits to any contact's core fields
(callsigns, grids, band, mode, timestamp). Edits enter an admin
approval queue with field-by-field diff view. On approve, changes
are applied and enrichment re-enqueued if grids/band changed.
Users receive email notification on approve or reject.
Also updates dependabot.yml for mix ecosystem.
When clicking a grid point, fetches the latest NEXRAD composite
reflectivity and identifies rain cells within 300 km that could
enable rain scatter contacts. Shows:
- Scatter classification (excellent/good/marginal/none)
- Top 3 cells with dBZ, distance, bearing, and relative signal
- Colored circle markers on the map at rain cell locations
- Markers sized by reflectivity, colored by intensity
Uses simplified bistatic radar equation accounting for reflectivity,
frequency-dependent scattering (Rayleigh/Mie), and R^4 path loss.
NEXRAD cells sampled every ~5 km within bounding box for efficiency.
When clicking a grid point with ducting, the panel now shows each
duct layer with base-top height in feet, thickness in meters, and
minimum trapped frequency. Data flows from Duct.analyze through
the scoring factors as a ducts array.
The PropagationGridWorker now fetches native hybrid-sigma levels
(TMP, SPFH, HGT, PRES × 50 levels) alongside the standard surface
and pressure products. Native data provides 10-50m vertical spacing
vs 250m from pressure levels, detecting thin surface ducts invisible
to the standard product.
Key design: cell-by-cell reducer in Wgrib2.extract_grid_from_file_mapped
processes each of the 95k CONUS cells through a duct analysis function
inline, keeping only scalar metrics per cell. Peak memory ~86 MB
instead of ~1.8 GB for the full grid map.
Per-cell output: native_min_gradient, best_duct_freq_ghz,
max_duct_thickness_m, duct_count. The scorer prefers the native
gradient over the pressure-level gradient when available.
Native fetch is optional — if it fails, scoring continues with
pressure-level data only.
Without -s, wgrib2 -lon only outputs msg:offset:lon=X,lat=Y,val=Z
with no variable name or level. The -s flag adds the short inventory
(d=DATE:VAR:LEVEL:...) so the parser can identify which variable
each value belongs to.
Points spread coast-to-coast created a ~476k cell bounding grid
(350 messages × 476k cells × 4 bytes ≈ 665 MB), causing OOM.
Switch to -lon which extracts values at specific lat/lon points
with text output. One wgrib2 call, one file scan, negligible
BEAM memory regardless of point geographic spread.
- Stream profile storage and score upsert instead of materializing
full 20k+ item lists (propagation_grid_worker, propagation.ex)
- GC between forecast hours and store/compute phases to reclaim
~400 MB of grid data between steps
- Single-pass field extraction in scorer.ex path_integrated_conditions
instead of 6 separate Enum traversals
- Eliminate intermediate merged map in fetch_grid by combining
merge + profile build into one pipe
- Fix UUID bug: bingenerate → generate in native grid worker
(same issue previously fixed in nexrad_worker)
The parallel download was holding all ~530MB of range responses in
memory before writing to disk. Now each range is fetched and written
one at a time, so only one chunk is in memory at a time.
Instead of holding ~530MB GRIB binary in BEAM memory, download
ranges directly to a temp file and run wgrib2 on it. Peak memory
drops from ~530MB to just HTTP chunk buffers.
FrontalAnalysis module (Weather.FrontalAnalysis):
- detect_fronts/3 computes the Thermal Front Parameter (TFP) from
2D grids of surface temperature and pressure using Nx vectorized
ops. TFP = -nabla|nabla(theta)| . nabla(theta)/|nabla(theta)|.
Most negative values mark cold fronts.
- central_gradient/1 for 2D finite differences with edge handling
- nearest_front/3 finds closest front point with distance and bearing
- path_front_angle/2 computes angle between a QSO path and the
front (0 = parallel = good, 90 = crosses = dead)
Backtest feature stubs for distance_to_front and parallel_to_front
(return nil until the pipeline caches per-cell frontal features from
the hourly HRRR grid run). The FrontalAnalysis module itself is
tested and ready for integration.
NEXRAD spike docs also included in this commit.
NCEI ASOS 5-minute data client (Weather.NceiMetarClient):
- fetch/3 pulls per-station monthly .dat files from NCEI C00418
- parse/1 decodes the fixed-width METAR format including precise
T-group temperatures (T02110094 → 21.1/9.4°C)
- metar_5min_observations table: schema-identical to
surface_observations, separate table to avoid mixing cadences
Weather.recent_surface_obs/3 prefers 5-min data when available,
falls back to the hourly surface_observations table.
Data URL: https://www.ncei.noaa.gov/data/automated-surface-observing-system-five-minute/access/YYYY/MM/asos-5min-KXXX-YYYYMM.dat
Available back to 1996.
Phase 3 NEXRAD spike: IEM n0q composite available at 5-min cadence
back to 2022+. Compression-ratio proxy shows afternoon images have
13-81% more texture than dawn (directionally correct), but the n0q
product thresholds out the faint clear-air returns needed for BL
stability detection. Parked until MRMS or Level III products can be
investigated. See docs/research/nexrad_spike.md.
Phase 6: hrrr_climatology table aggregating surface_temp_c by
(lat, lon, month, hour) from the 42M+ hrrr_profiles grid-point
rows. mix hrrr_climatology builds it via a single SQL GROUP BY +
upsert. Backtest.Features.temperature_anomaly computes current_temp
minus climatological mean — the meteorologist's "temperature
deviation above normal" predictor for summer afternoon enhancement.
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286
Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric
mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.
First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
wgrib2 -lola ... bin writes Fortran unformatted records (4-byte
length header + data + 4-byte length trailer per message).
parse_lola_binary was treating the binary as tightly packed,
causing every message after the first to read from the wrong
offset — values came out as garbage across all grid points.
Fix: account for the 8-byte record overhead per message when
computing the data offset for each message's grid values.
This bug affects both the existing propagation grid extraction
(which may have been producing subtly wrong scores) and the new
native-level extraction (which was producing obviously wrong
values). The fix is a one-line stride change.
Also adds Backtest.Features.native_surface_refractivity for the
Phase 1 sanity check, plus a tighter wgrib2 match pattern that
selects only hybrid-level messages from the native file.
The Elixir GRIB2 decoder didn't map level type 105 (hybrid) or
variable IDs for SPFH and TKE, so native-level messages decoded as
"unknown:105:N" keys that build_native_profile couldn't find. Add
the three missing mappings to Section.identify_level/identify_var.
Also add HrrrNativeClient.extract_native_profiles/2 which uses
wgrib2's -lola on a tight bounding-box subgrid for speed (the pure
Elixir decoder takes ~70s per point on a 395 MB file; wgrib2 handles
40 points in seconds). The worker now routes through this path.
- Spike docs at docs/research/hrrr_native_levels.md confirming files
are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
SPFH needed for Phase 2 turbulence features. Architectural finding:
per-point on-demand fetching is impractical (~530 MB/file), so
the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
scalars and placeholder columns for Phase 2/4 derived fields.
Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
unique at :infinity, pulls distinct (lat, lon) points from contacts
in the ±30 min window, downloads the native grib2, extracts per
point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
contact count.
Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.
- Era5BatchClient builds the monthly CDS requests, extracts every
(lat, lon, hour) from the GRIB2 blob with wgrib2, derives
refractivity params, and bulk-inserts in 2k-row chunks with
on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
→ enqueue the month-batch for the point's tile-month and return.
No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
- Cache raw GRIB2 byte ranges to ~/hrrr in dev (never deleted)
- mix hrrr_backfill re-fetches QSO-linked HRRR profiles with 13 levels
- Supports --all (re-fetch everything) and --limit N flags
- Groups by HRRR hour to batch requests, 500ms rate limit
- AWS archive has full HRRR history, same URL pattern as live feed
Fetch every 25mb from 1000-700mb (13 levels, up from 8). Gives ~80m
vertical spacing near the surface, enough to resolve ducting layers
that were previously invisible between the coarse 250m level spacing.
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2)
- PropagationGridWorker fetches all 19 forecast hours per run
- Propagation.scores_at/3 queries scores at specific valid_time
- Propagation.available_valid_times/1 returns all forecast times for timeline
- Pruning keeps scores with valid_time >= now - 2h (forecast-aware)
- MapLive: select_time event, timeline data pushed to JS
- JS: forecast timeline bar at bottom of map with clickable hour buttons
- PubSub broadcast sends list of valid_times instead of single time
Req default 15s receive_timeout too short for GRIB2 byte-range
downloads on prod. Also reduce HRRR queue concurrency from 20 to 5
to avoid NOAA rate limiting (was causing burst-then-stall pattern).
New AsosAdjustmentWorker runs every 10 minutes:
- Fetches latest ASOS observations from all ~2900 US stations via
IEM bulk currents API (parallel fetch across 51 state networks)
- For each grid point within 75km of a reporting station, re-scores
using fresh ASOS data (temp, dewpoint, wind, sky, pressure, precip)
with HRRR refractivity gradient from the last hourly computation
- Pushes updated scores to the map via PubSub
Also stores HRRR profiles in the database during grid computation
so the data persists for reference and ASOS blending.
Add a separate wgrib2-builder stage that compiles wgrib2 from source.
Docker layer caching means this only builds once — subsequent deploys
reuse the cached layer. The binary is copied into the final runtime
image. Also fixed wgrib2 path lookup to be runtime instead of
compile-time so it works in the Docker build pipeline.
Add Wgrib2 module that shells out to wgrib2 binary for fast GRIB2
grid extraction using -lola (nearest-neighbor to regular lat-lon grid).
Falls back to pure-Elixir decoder if wgrib2 is not installed.
Also: parallel GRIB2 range downloads, merge adjacent byte ranges,
skip corrupt messages instead of failing, pressure fetch is optional.
Both extract_points and extract_grid now skip messages with missing
data sections instead of halting. Pressure product fetch is now
optional for both grid and single-point modes — if pressure data
has corrupt messages, surface-only profiles are still stored. Fixes
historical HRRR backfill failing on old data with incomplete files.
Add fetch_grid/3 for batch HRRR data retrieval across multiple lat/lon
points in a single download pass. Expand surface messages to include
wind (UGRD/VGRD), cloud cover (TCDC), and precipitation (APCP). Add
extract_grid/2 to the GRIB2 extractor for multi-point extraction from
a single GRIB binary, and add GRIB2 variable identifiers for the new
surface fields.
Add extract_values/3 to SimplePacking and ComplexPacking for batch
index extraction from a single GRIB2 message. Add extract_grid/2 to
Extractor which takes a list of {lat, lon} points and returns all
variable values for each point, skipping points outside the grid.
This enables extracting weather data for many grid points from a
single HRRR download instead of re-parsing per point.
- Add precip_1h_in and wx_codes fields to ASOS surface observations
- Add IEMRE reanalysis schema for radar-derived hourly precipitation
- Add IemreFetchWorker with exponential backoff and idempotency
- Integrate IEMRE enqueue into cron weather backfill pipeline
- All existing QSOs marked iemre_queued=false for automatic backfill
extract_n_values_array used Enum.take(count) on a reversed list
before reversing it, which included padding values from byte
alignment and dropped actual values. When group count * bits
wasn't a multiple of 8, the extra padding bits produced a
phantom value that shifted the entire array by one position.
This caused cascading errors in spatial differencing — values
started correct but diverged exponentially (DPT decoded as
38 billion K instead of 275 K).
Fix: reverse the list first, then take count, so padding values
at the end are discarded instead of actual values at the start.
AWS S3 doesn't support multi-range HTTP requests. When given
Range: bytes=0-999, 2000-2999 it ignores the header and returns
the full file (200 instead of 206). Download each range separately.
