Registered users can suggest edits to any contact's core fields
(callsigns, grids, band, mode, timestamp). Edits enter an admin
approval queue with field-by-field diff view. On approve, changes
are applied and enrichment re-enqueued if grids/band changed.
Users receive email notification on approve or reject.
Also updates dependabot.yml for mix ecosystem.
When clicking a grid point, fetches the latest NEXRAD composite
reflectivity and identifies rain cells within 300 km that could
enable rain scatter contacts. Shows:
- Scatter classification (excellent/good/marginal/none)
- Top 3 cells with dBZ, distance, bearing, and relative signal
- Colored circle markers on the map at rain cell locations
- Markers sized by reflectivity, colored by intensity
Uses simplified bistatic radar equation accounting for reflectivity,
frequency-dependent scattering (Rayleigh/Mie), and R^4 path loss.
NEXRAD cells sampled every ~5 km within bounding box for efficiency.
- Stream profile storage and score upsert instead of materializing
full 20k+ item lists (propagation_grid_worker, propagation.ex)
- GC between forecast hours and store/compute phases to reclaim
~400 MB of grid data between steps
- Single-pass field extraction in scorer.ex path_integrated_conditions
instead of 6 separate Enum traversals
- Eliminate intermediate merged map in fetch_grid by combining
merge + profile build into one pipe
- Fix UUID bug: bingenerate → generate in native grid worker
(same issue previously fixed in nexrad_worker)
Add Propagation.Region module with 8 CONUS climate zones (gulf_coast,
southeast, southern_plains, corn_belt, northeast, desert_southwest,
pacific_northwest, mountain_west) and per-region monthly seasonal
adjustment multipliers.
The scorer's score_season now takes lat/lon and applies a regional
multiplier from Region.seasonal_adjustment on top of the band's
seasonal_base + seasonal_adj. Gulf coast August gets a 1.15x boost
(drier, better for ducting) while Corn Belt August gets a 0.80x
penalty (corn evapotranspiration = miserable dewpoints).
Adjustments are hand-tuned starting points from the meteorologist's
qualitative guidance. Phase 9 recalibration will refine them from
backtest data.
Duct module (Propagation.Duct):
- refractivity_profile/1: ITU-R P.453 N at each native level
- m_profile/1: modified refractivity M = N + 157*h(km)
- detect_ducts/1: find contiguous regions where dM/dh < 0, returning
base/top height, thickness, and M-deficit per duct
- min_trapped_frequency_ghz/1: waveguide approximation (Bean & Dutton)
for the minimum frequency a duct of given geometry can trap
- analyze/1: full pipeline from native profile to duct list + best
trapped frequency across all ducts
Derive task updated to also compute ducts JSONB and best_duct_band_ghz
alongside the Phase 2 turbulence fields.
Backtest features: duct_thickness, best_duct_freq, duct_usable_10ghz,
duct_usable_24ghz, duct_usable_47ghz.
Real-data validation: 2022-08-20 12Z TX profile shows 0 ducts (M
increases monotonically) — correct for a well-mixed boundary layer
on a turbulent August afternoon (Ri=0.16).
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286
Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric
mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.
First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
New BandConfig entries for 902, 1296, 2304, 3456, 5760 MHz:
- All beneficial humidity effect (like 10 GHz)
- Near-zero gaseous absorption and rain attenuation
- Ranges: 902 MHz typical 400 km, 5760 MHz typical 220 km
- Same seasonal curves as 10 GHz (ducting-driven)
BandConfig.band_options/0 generates dropdown options from configs.
All pages (path, rover, submit) use centralized band_options instead
of hardcoded lists. Map page already used BandConfig.all_bands().
10 GHz remains the default on all pages.
Contact scoring now uses all HRRR profiles along the path instead of
just pos1. Aggregation strategy:
- Best along path for beneficial factors (refractivity, pressure)
- Worst along path for harmful factors (rain, wind)
- Average for neutral factors (temp, dewpoint, PWAT, BL depth)
Scorer.path_integrated_conditions/2 merges multiple profiles into a
single conditions map. Falls back gracefully to single-profile scoring
when only one profile is available.
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
normalized distance — reduces noise from operator/equipment variation
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
Previous thresholds (-500 to -60) were calibrated for radiosonde data.
HRRR profiles have coarser vertical resolution, with gradients clustering
between -40 and -130 N/km (median -70). Nearly all grid points were
falling through to the default score of 42, wasting the refractivity
factor. New thresholds (-200 to -40) spread across HRRR percentiles.
- Add enqueue_for_qso/1 to directly enqueue weather/HRRR/terrain/IEMRE
jobs for a single user-submitted QSO (no cron, no bulk processing)
- Submit flow calls enqueue_for_qso instead of generic enqueue worker
- Add enrichment queues to prod config for on-demand processing
- Guard against HRRR fill values in store_hrrr_profiles (fixes badarith)
- Filter QSOs without pos2 in build_terrain_jobs
Replaces the one-shot startup Task with a GenServer that checks every
5 minutes whether propagation scores are older than 2 hours. If stale,
enqueues a PropagationGridWorker job (with dedup check to avoid double
queuing). Disabled in test env to avoid SQL Sandbox conflicts.
Replace circle markers with a canvas tile layer that renders smooth,
flowing colored regions using bilinear interpolation between grid
points. Colors interpolate between tiers for gradients. ~95k grid
points at 0.125 degree resolution with wgrib2 extraction.
Add Wgrib2 module that shells out to wgrib2 binary for fast GRIB2
grid extraction using -lola (nearest-neighbor to regular lat-lon grid).
Falls back to pure-Elixir decoder if wgrib2 is not installed.
Also: parallel GRIB2 range downloads, merge adjacent byte ranges,
skip corrupt messages instead of failing, pressure fetch is optional.
95k points at 0.125 degree resolution caused the GRIB2 extraction to
take too long. 0.5 degree (~55 km) resolution gives 6k points which
completes in under a minute. Can increase resolution later once the
extraction is optimized.
Define 0.125-degree CONUS grid (25-50N, 125-66W) for propagation
scoring and create propagation_scores table with composite unique
index on lat/lon/valid_time/band_mhz for upsert support.
Single source of truth for all scoring parameters: weights, thresholds,
seasonal tables, and per-band coefficients for 8 microwave bands
(10G through 241G). Includes ITU-R P.838-3 rain attenuation
coefficients, humidity effects, refractivity scoring thresholds,
and sunrise/tier definitions.
