Normalize ML features to prevent NaN gradient explosion

Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.

2026-04-01 09:08:33 -05:00

3.7 KiB

Raw Permalink Blame History

Algorithm Refinement & ML Training Design

Goal

Refine the propagation scoring algorithm using 57K+ historical QSOs matched to HRRR atmospheric conditions, then train the Axon ML model on the refined algorithm's output for fast inference from HRRR forecast data.

Data Assets

57,488 tropospheric QSOs (1992-2024) with distance as ground truth
54M HRRR grid profiles (2016-2026) at 0.125° resolution
326K surface observations
58K terrain profiles (97% BLOCKED — confirms atmospheric propagation)
3M propagation_scores (current algorithm output)
Band distribution: 53K at 10G, 3.8K at 24G, 800 at 47G, 141 at 75G, sparse above

Design Decisions

Target variable: Hybrid — refine algorithm from QSO data, then train ML on refined scores
QSO-HRRR matching: Endpoint average — HRRR conditions at both QSO endpoints, averaged
Algorithm scope: Full re-examination — weights, thresholds, add/remove factors
ML deployment: ML primary scorer with algorithm fallback for missing data

Implementation Plan

Stage 1: Analysis Mix Task (`mix propagation.analyze`)

Build QSO↔HRRR analysis dataset:

For each QSO, find nearest HRRR grid point to pos1 and pos2 (round to 0.125°)
Find HRRR profile at nearest hour to qso_timestamp
Average atmospheric conditions across both endpoints
Output: CSV or in-memory dataset with columns:
- qso_id, band, distance_km, month, utc_hour
- avg_surface_temp_c, avg_surface_dewpoint_c, avg_surface_pressure_mb
- avg_min_refractivity_gradient, avg_hpbl_m, avg_pwat_mm
- avg_surface_refractivity, ducting_at_either_endpoint
- terrain_verdict (BLOCKED/CLEAR/FRESNEL_PARTIAL)

Run correlation analysis:

Spearman rank correlation of each variable with distance_km, stratified by band
Bin each variable into deciles, compute median distance per bin (natural breakpoint curves)
Check for interaction effects (e.g., gradient × time_of_day, hpbl × month)
Look for currently-missing predictive factors
Compare factor distributions for short (<100km) vs long (>300km) contacts

Output: printed report with findings and recommended changes.

Stage 2: Update algo.md

Document all findings with data backing:

Which factors gained/lost predictive power
New threshold curves derived from data
Revised weights with justification
Any new factors added or old factors removed
Before/after comparison of scoring behavior

Stage 3: Update scorer.ex + band_config.ex

Implement revised algorithm:

Updated scoring functions with new thresholds
Updated weights in band_config
New factors if analysis warrants
All changes match algo.md documentation
Run mix test to verify no regressions (update tests for new behavior)

Stage 4: Recompute Propagation Scores

Run refined algorithm across HRRR grid:

Either mix propagation_grid or a targeted recompute task
Generates fresh propagation_scores with refined algorithm
These become ML training data

Stage 5: Training Mix Task (`mix propagation.train`)

Build training pipeline:

Query propagation_scores joined to hrrr_profiles by lat/lon/valid_time
Features: raw atmospheric inputs + cyclical temporal + log frequency
Target: refined composite score normalized to 0-1
80/10/10 train/val/test split, stratified by band and score quartile
MSE loss, Adam optimizer, early stopping on validation loss
Save trained weights to priv/models/propagation_v1.nx
Print evaluation metrics (RMSE, R², per-band performance)

Stage 6: Integrate ML into Grid Worker

model.ex: predict_score/2 for inference
PropagationGridWorker: try ML prediction first, fall back to Scorer.composite_score/2
Graceful degradation when model file missing or inputs incomplete

3.7 KiB Raw Permalink Blame History Unescape Escape