# Algorithm Refinement & ML Training Design

## Goal

Refine the propagation scoring algorithm using 57K+ historical QSOs matched to HRRR atmospheric conditions, then train the Axon ML model on the refined algorithm's output for fast inference from HRRR forecast data.

## Data Assets

- 57,488 tropospheric QSOs (1992-2024) with distance as ground truth
- 54M HRRR grid profiles (2016-2026) at 0.125° resolution
- 326K surface observations
- 58K terrain profiles (97% BLOCKED — confirms atmospheric propagation)
- 3M propagation_scores (current algorithm output)
- Band distribution: 53K at 10G, 3.8K at 24G, 800 at 47G, 141 at 75G, sparse above

## Design Decisions

- **Target variable**: Hybrid — refine algorithm from QSO data, then train ML on refined scores
- **QSO-HRRR matching**: Endpoint average — HRRR conditions at both QSO endpoints, averaged
- **Algorithm scope**: Full re-examination — weights, thresholds, add/remove factors
- **ML deployment**: ML primary scorer with algorithm fallback for missing data

## Implementation Plan

### Stage 1: Analysis Mix Task (`mix propagation.analyze`)

Build QSO↔HRRR analysis dataset:
- For each QSO, find nearest HRRR grid point to pos1 and pos2 (round to 0.125°)
- Find HRRR profile at nearest hour to qso_timestamp
- Average atmospheric conditions across both endpoints
- Output: CSV or in-memory dataset with columns:
  - qso_id, band, distance_km, month, utc_hour
  - avg_surface_temp_c, avg_surface_dewpoint_c, avg_surface_pressure_mb
  - avg_min_refractivity_gradient, avg_hpbl_m, avg_pwat_mm
  - avg_surface_refractivity, ducting_at_either_endpoint
  - terrain_verdict (BLOCKED/CLEAR/FRESNEL_PARTIAL)

Run correlation analysis:
- Spearman rank correlation of each variable with distance_km, stratified by band
- Bin each variable into deciles, compute median distance per bin (natural breakpoint curves)
- Check for interaction effects (e.g., gradient × time_of_day, hpbl × month)
- Look for currently-missing predictive factors
- Compare factor distributions for short (<100km) vs long (>300km) contacts

Output: printed report with findings and recommended changes.

### Stage 2: Update algo.md

Document all findings with data backing:
- Which factors gained/lost predictive power
- New threshold curves derived from data
- Revised weights with justification
- Any new factors added or old factors removed
- Before/after comparison of scoring behavior

### Stage 3: Update scorer.ex + band_config.ex

Implement revised algorithm:
- Updated scoring functions with new thresholds
- Updated weights in band_config
- New factors if analysis warrants
- All changes match algo.md documentation
- Run `mix test` to verify no regressions (update tests for new behavior)

### Stage 4: Recompute Propagation Scores

Run refined algorithm across HRRR grid:
- Either `mix propagation_grid` or a targeted recompute task
- Generates fresh propagation_scores with refined algorithm
- These become ML training data

### Stage 5: Training Mix Task (`mix propagation.train`)

Build training pipeline:
- Query propagation_scores joined to hrrr_profiles by lat/lon/valid_time
- Features: raw atmospheric inputs + cyclical temporal + log frequency
- Target: refined composite score normalized to 0-1
- 80/10/10 train/val/test split, stratified by band and score quartile
- MSE loss, Adam optimizer, early stopping on validation loss
- Save trained weights to priv/models/propagation_v1.nx
- Print evaluation metrics (RMSE, R², per-band performance)

### Stage 6: Integrate ML into Grid Worker

- model.ex: `predict_score/2` for inference
- PropagationGridWorker: try ML prediction first, fall back to Scorer.composite_score/2
- Graceful degradation when model file missing or inputs incomplete