The grid-cached profile is trimmed to 13 levels (1000→700 mb) for
memory reasons, so the skew-T plot showed nothing above 700 mb.
HrrrClient.fetch_profile/4 now accepts a forecast_hour opt; SkewtLive
issues a per-point fetch against the same cycle the cached row came
from to get the full 1000→100 mb set, falling back to the cached
truncated cell on any failure.
Time selection no longer round-trips through the URL — `select_time`
updates state in-process and triggers a focused :load_skewt_time
async, so refreshing or sharing a /skewt?q=... link always lands on
the most recent analysis hour.
Each fix is covered by a regression test that fails on `main` and
passes on this commit.
Round 1 (initial review):
* propagation: thread `latitude` into the conditions map so
`score_season/4` actually picks up regional multipliers
* hrrr_client / fetcher.rs: `nearest_hrrr_hour` rounds DOWN, never at
a future cycle that NOAA hasn't published yet
* radio: spherical-vector great-circle midpoint replaces the
arithmetic mean — anti-meridian paths no longer fold to Greenwich
* weather: `reconcile_weather_statuses` scales the longitude band by
`1 / cos(lat)` so the bbox stays ~150 km wide at every latitude
* radio/maidenhead: clamp 90°/180° below the field-bucket overflow so
`from_latlon` never emits invalid characters like 'S'
* prop_grid_rs/pipeline: merge HRRR + NEXRAD-derived rain rates and
read `best_duct_freq_ghz` into `best_duct_band_ghz` so the Native
Duct Boost actually fires
* propagation/region (Elixir + Rust): inclusive upper bounds so points
exactly at lat_max get the regional multiplier
* weather/sounding_params (Elixir + Rust): drop the 10 m gradient
floor so HRRR's thin near-surface layers stop hiding sharp ducts
* weather/sounding_params: when the profile ends inside a duct,
finalize it with the highest sample as the top instead of throwing
it away (Rust port already correct)
Round 2 (post-fix sweep):
* radio + commercial: single canonical haversine in Radio (atan2
form); Commercial delegates instead of carrying a second copy that
could disagree at threshold distances
* prop_grid_rs/profiles_file: `snap_coords` matches Elixir's
step-aware snap (`round(coord/0.125) * 0.125`, then 3-dp round) so
Rust-keyed and Elixir-keyed profile maps land on the same cell
* weather/grib2/wgrib2: `parse_lon_val_segment` uses `Float.parse`
uniformly — wgrib2 dropping the trailing `.0` from a longitude no
longer crashes the whole chain step
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
- HrrrClient idx-cache test only invalidated the surface idx URL, but
fetch_profile also fetches a pressure idx. Previous runs' state for
the pressure key decided whether the counter landed at 1 (prior run
cached it, pass) or 2 (cold, fail). Invalidate both + assert 2 total
fetches to reflect the actual code path.
- CsvImportTest deadlocked against other async DataCase tests when
inline Oban child jobs upserted iemre_observations/terrain_profiles
with a shared conflict target. Flip to async: false — same fix as
ContactWeatherEnqueueWorkerTest earlier this session.
- PromEx.Plugins.Oban runs a 5s telemetry_poller that queries the DB,
but its poller PID has no sandbox connection in test and crashed
with DBConnection.OwnershipError on every tick, spamming the log.
Gate the plugin on a config flag and skip it in config/test.exs;
prod behaviour unchanged.
Telemetry showed the application-master process holding ~830 MiB of
terms from warm_grid_cache_from_latest_profile — the data lives in
the app master's heap and never GCs because the process is idle.
Running it in a Task.start lets the terms die with the task.
Mark GridCache, MrmsCache, NexradCache, and ScoreCache ETS tables
:compressed. The scored-band-map and HRRR grid data are map-heavy;
compression trims hundreds of MiB at a few percent CPU cost.
Memoise HRRR .idx responses in Microwaveprop.Cache. Published idx
files are immutable for a model run, but the hourly chain re-fetches
the same URL dozens of times across forecast hours. Cuts ~10s per
repeat out of hrrr_fetch_idx.
Force a garbage collect at the end of HrrrFetchWorker.perform to
reclaim the refc binary heap held from GRIB2 ranges before the Oban
producer hands the process its next job.
Four changes sized by measured prod telemetry (83m of spans):
1. propagation queue: 2 → 1 slot per pod. Two concurrent forecast-hour
steps per pod stacked HRRR grid + native duct grid + scored band
map into ~5-6 GiB RSS, OOM-killing every ~15 min. 3-way parallelism
cluster-wide still finishes the chain inside the hourly interval.
2. weather queue: 3 → 1 slot per pod. ASOS backfill was 429-thrashing
IEM (1,296 retryable jobs; logs were nothing but 429 backoffs).
3. PropagationGridWorker: skip native-level duct fetch on f01..f18.
At ~7-11 min/fh and 18 forecast hours, this was the largest single
cost per chain. Forecast hours fall back to
derived[:min_refractivity_gradient] from the pressure-level
profile. f00 still gets full native-level duct analysis.
4. HrrrClient.download_grib_ranges_to_file: parallelize with
Task.async_stream (max_concurrency 8). The file-backed variant was
sequential, dominating native-duct fetch time on the remaining f00
path. ~20s → ~3s per call.
Telemetry showed ~66 PropagationGridWorker exceptions per 6h with
55 ArgumentErrors and 11 TimeoutErrors, producing ~13 discarded
chain steps. Each discard broke the chain: subsequent forecast
hours were never enqueued, leaving the score store with huge gaps
(e.g. at 14:11 UTC the earliest available forecast was 18:00,
because f00-f05 all failed somewhere upstream and nothing ran
after them).
Three changes:
1. PropagationGridWorker: on the final attempt, still enqueue
fh+1 even when this step failed. Oban discards the current
job normally — but the rest of the chain keeps running, so
one bad hour doesn't take out the remaining 12-18. The
rescue is factored into a tested public helper.
2. HrrrClient.parse_idx: skip malformed idx lines instead of
raising. NOAA S3 occasionally serves an HTML error page as
the idx body, and the old strict String.to_integer path
raised ArgumentError on the first non-numeric line and took
down the chain step. This is the root cause of the 55
ArgumentErrors.
3. JS renderTimeline: when no forecast hour is at-or-before
wall-clock (all times are future — the gap scenario the
fixes above are designed to prevent), stop labeling the
earliest future slot "Now". Lets the user see honest
"+Nh" offsets instead of a lie on the pill.
The skew-T commit (30c1018) doubled @pressure_levels from 13 to 25 so
new contact fetches would cover the full troposphere. That list is
also what PropagationGridWorker pulls per forecast hour, which
doubled the GRIB footprint (~57 MB compressed + 92k points × 25
levels × 3 vars decoded through wgrib2) and pushed prod pods over
their 4 Gi OOMKill threshold. Every chain died during f00 and the
map timeline never got beyond now and now+1h because the .ntms files
for f02-f18 were never written.
Split the constant:
* @profile_pressure_levels (25 levels, 1000-100 mb) drives the
per-contact HrrrClient.fetch_profile path so the skew-T plot
keeps its full-atmosphere trace.
* @grid_pressure_levels (13 levels, 1000-700 mb) drives the grid
hot path. That's the band SoundingParams.derive reads for
min_refractivity_gradient, and native hybrid-sigma data
(native_min_gradient) takes priority over the pressure-level
fallback anyway, so upper-air levels contribute nothing to
scoring — pure memory waste on this path.
build_profile/1 still iterates the full 25-level list; grid fetches
simply populate the 13 near-surface slots and skip the rest.
Makes HrrrClient.pressure_messages public with a :grid | :profile
variant so the split is testable from outside the module.
Historical contacts showed a skew-T log-P diagram that stopped at 700 mb
because HrrrClient and Era5Client only fetched pressure-level data down
to the top of the boundary layer. The chart canvas already ran up to
100 mb, so the trace clipped mid-atmosphere.
Two complementary fixes:
1. Extend @pressure_levels in HrrrClient and Era5Client with
650/600/550/500/450/400/350/300/250/200/150/100 mb so new fetches
cover the full troposphere + lower stratosphere.
2. Prefer the native hybrid-sigma profile for the contact-detail
skew-T when one has been backfilled for the contact's hour. The
native profile already stores all 50 hybrid levels up to ~19 km,
so historical contacts covered by the native backfill get a full
trace without re-hitting S3. A new HrrrNativeProfile.to_skew_t_profile/1
converts the parallel arrays into the %{"pres","tmpc","dwpc","hght"}
list shape the renderer expects, deriving dewpoint from SPFH via the
Magnus inverse. Weather.find_nearest_native_profile/3 mirrors
find_nearest_hrrr/3 for the lookup.
Add fetch_grid/3 for batch HRRR data retrieval across multiple lat/lon
points in a single download pass. Expand surface messages to include
wind (UGRD/VGRD), cloud cover (TCDC), and precipitation (APCP). Add
extract_grid/2 to the GRIB2 extractor for multi-point extraction from
a single GRIB binary, and add GRIB2 variable identifiers for the new
surface fields.
AWS S3 doesn't support multi-range HTTP requests. When given
Range: bytes=0-999, 2000-2999 it ignores the header and returns
the full file (200 instead of 206). Download each range separately.
Eliminates the external wgrib2 C tool dependency that blocked HRRR
processing. Implements Lambert Conformal projection, simple packing
(Template 5.0), complex packing with spatial differencing (Template 5.3),
and GRIB2 section parsing — enough to extract point values from HRRR
grid data using only Elixir.
HRRR provides hourly 3km-resolution atmospheric profiles, filling the
temporal gaps (12-hourly soundings) and spatial gaps (only 9 sounding
stations) in our current weather data.
- Add hrrr_profiles table and hrrr_queued flag on QSOs
- HrrrClient fetches GRIB2 data via HTTP Range requests + wgrib2
- HrrrFetchWorker derives refractivity/ducting params via SoundingParams
- QsoWeatherEnqueueWorker now also enqueues HRRR jobs
- QSO show page displays HRRR section with collapsible profile
- Dockerfile builds wgrib2 from source for production
