Graham McIntire
69b5caf876
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1]) causing gradient explosion. Normalize all atmospheric features to ~[0,1] using known physical bounds. Add Polaris dep for optimizer.
87 lines
3.7 KiB
Markdown
87 lines
3.7 KiB
Markdown
# Algorithm Refinement & ML Training Design
|
||
|
||
## Goal
|
||
|
||
Refine the propagation scoring algorithm using 57K+ historical QSOs matched to HRRR atmospheric conditions, then train the Axon ML model on the refined algorithm's output for fast inference from HRRR forecast data.
|
||
|
||
## Data Assets
|
||
|
||
- 57,488 tropospheric QSOs (1992-2024) with distance as ground truth
|
||
- 54M HRRR grid profiles (2016-2026) at 0.125° resolution
|
||
- 326K surface observations
|
||
- 58K terrain profiles (97% BLOCKED — confirms atmospheric propagation)
|
||
- 3M propagation_scores (current algorithm output)
|
||
- Band distribution: 53K at 10G, 3.8K at 24G, 800 at 47G, 141 at 75G, sparse above
|
||
|
||
## Design Decisions
|
||
|
||
- **Target variable**: Hybrid — refine algorithm from QSO data, then train ML on refined scores
|
||
- **QSO-HRRR matching**: Endpoint average — HRRR conditions at both QSO endpoints, averaged
|
||
- **Algorithm scope**: Full re-examination — weights, thresholds, add/remove factors
|
||
- **ML deployment**: ML primary scorer with algorithm fallback for missing data
|
||
|
||
## Implementation Plan
|
||
|
||
### Stage 1: Analysis Mix Task (`mix propagation.analyze`)
|
||
|
||
Build QSO↔HRRR analysis dataset:
|
||
- For each QSO, find nearest HRRR grid point to pos1 and pos2 (round to 0.125°)
|
||
- Find HRRR profile at nearest hour to qso_timestamp
|
||
- Average atmospheric conditions across both endpoints
|
||
- Output: CSV or in-memory dataset with columns:
|
||
- qso_id, band, distance_km, month, utc_hour
|
||
- avg_surface_temp_c, avg_surface_dewpoint_c, avg_surface_pressure_mb
|
||
- avg_min_refractivity_gradient, avg_hpbl_m, avg_pwat_mm
|
||
- avg_surface_refractivity, ducting_at_either_endpoint
|
||
- terrain_verdict (BLOCKED/CLEAR/FRESNEL_PARTIAL)
|
||
|
||
Run correlation analysis:
|
||
- Spearman rank correlation of each variable with distance_km, stratified by band
|
||
- Bin each variable into deciles, compute median distance per bin (natural breakpoint curves)
|
||
- Check for interaction effects (e.g., gradient × time_of_day, hpbl × month)
|
||
- Look for currently-missing predictive factors
|
||
- Compare factor distributions for short (<100km) vs long (>300km) contacts
|
||
|
||
Output: printed report with findings and recommended changes.
|
||
|
||
### Stage 2: Update algo.md
|
||
|
||
Document all findings with data backing:
|
||
- Which factors gained/lost predictive power
|
||
- New threshold curves derived from data
|
||
- Revised weights with justification
|
||
- Any new factors added or old factors removed
|
||
- Before/after comparison of scoring behavior
|
||
|
||
### Stage 3: Update scorer.ex + band_config.ex
|
||
|
||
Implement revised algorithm:
|
||
- Updated scoring functions with new thresholds
|
||
- Updated weights in band_config
|
||
- New factors if analysis warrants
|
||
- All changes match algo.md documentation
|
||
- Run `mix test` to verify no regressions (update tests for new behavior)
|
||
|
||
### Stage 4: Recompute Propagation Scores
|
||
|
||
Run refined algorithm across HRRR grid:
|
||
- Either `mix propagation_grid` or a targeted recompute task
|
||
- Generates fresh propagation_scores with refined algorithm
|
||
- These become ML training data
|
||
|
||
### Stage 5: Training Mix Task (`mix propagation.train`)
|
||
|
||
Build training pipeline:
|
||
- Query propagation_scores joined to hrrr_profiles by lat/lon/valid_time
|
||
- Features: raw atmospheric inputs + cyclical temporal + log frequency
|
||
- Target: refined composite score normalized to 0-1
|
||
- 80/10/10 train/val/test split, stratified by band and score quartile
|
||
- MSE loss, Adam optimizer, early stopping on validation loss
|
||
- Save trained weights to priv/models/propagation_v1.nx
|
||
- Print evaluation metrics (RMSE, R², per-band performance)
|
||
|
||
### Stage 6: Integrate ML into Grid Worker
|
||
|
||
- model.ex: `predict_score/2` for inference
|
||
- PropagationGridWorker: try ML prediction first, fall back to Scorer.composite_score/2
|
||
- Graceful degradation when model file missing or inputs incomplete
|