The PropagationGridWorker now fetches native hybrid-sigma levels
(TMP, SPFH, HGT, PRES × 50 levels) alongside the standard surface
and pressure products. Native data provides 10-50m vertical spacing
vs 250m from pressure levels, detecting thin surface ducts invisible
to the standard product.
Key design: cell-by-cell reducer in Wgrib2.extract_grid_from_file_mapped
processes each of the 95k CONUS cells through a duct analysis function
inline, keeping only scalar metrics per cell. Peak memory ~86 MB
instead of ~1.8 GB for the full grid map.
Per-cell output: native_min_gradient, best_duct_freq_ghz,
max_duct_thickness_m, duct_count. The scorer prefers the native
gradient over the pressure-level gradient when available.
Native fetch is optional — if it fails, scoring continues with
pressure-level data only.
Backtest on 11,431 native profiles (2026-04-11):
- Drop duct_usable_10/24/47ghz (always 1.0, no discrimination)
- Drop bulk_richardson (near-identical QSO vs baseline means)
- Document all feature results with signal strength assessment
- theta_e_jump is strongest native discriminator (44% lift)
- best_duct_freq and duct_thickness show clear physical signal
- Round grid-derived lat/lon in maybe_fill_latlon changeset step
- Format lat/lon to 6 decimal places on index table
- Migrate existing beacon data to 6 decimal precision
- Add comma separators to EIRP mW display (e.g. 10,000 mW)
- Extract shared add_commas helper for format_freq and format_mw
Height in feet doesn't need decimal precision. Migrates the DB
column, updates schema type, and strips trailing .0 from form
input so the integer cast succeeds.
- Add CommaNumber JS hook for live comma formatting while typing
frequency MHz in the beacon form
- Strip commas server-side before changeset validation
- Order beacon list by most recently added (desc inserted_at)
- Round lat/lon to 6 decimal places in changeset and display
- Round height_ft to integer in changeset and display
- Display coords at 6 decimal places on show page
Without -s, wgrib2 -lon only outputs msg:offset:lon=X,lat=Y,val=Z
with no variable name or level. The -s flag adds the short inventory
(d=DATE:VAR:LEVEL:...) so the parser can identify which variable
each value belongs to.
setup-buildx-action downloads from GitHub Releases which is slow
and unreliable from the Forgejo runner. Plain docker build/push
works fine since Docker is already on the runner host.
Points spread coast-to-coast created a ~476k cell bounding grid
(350 messages × 476k cells × 4 bytes ≈ 665 MB), causing OOM.
Switch to -lon which extracts values at specific lat/lon points
with text output. One wgrib2 call, one file scan, negligible
BEAM memory regardless of point geographic spread.
- Stream profile storage and score upsert instead of materializing
full 20k+ item lists (propagation_grid_worker, propagation.ex)
- GC between forecast hours and store/compute phases to reclaim
~400 MB of grid data between steps
- Single-pass field extraction in scorer.ex path_integrated_conditions
instead of 6 separate Enum traversals
- Eliminate intermediate merged map in fetch_grid by combining
merge + profile build into one pipe
- Fix UUID bug: bingenerate → generate in native grid worker
(same issue previously fixed in nexrad_worker)
The parallel download was holding all ~530MB of range responses in
memory before writing to disk. Now each range is fetched and written
one at a time, so only one chunk is in memory at a time.
Instead of holding ~530MB GRIB binary in BEAM memory, download
ranges directly to a temp file and run wgrib2 on it. Peak memory
drops from ~530MB to just HTTP chunk buffers.
Release.backtest_all, climatology, native_derive now enqueue an
AdminTaskWorker job on the new :admin queue and return immediately.
Progress visible in Oban Web at /admin/oban.
