Commit graph

3 commits

Author SHA1 Message Date
3604896726
Rename ERA5 → NARR across the codebase
- Schema: Era5Profile → NarrProfile; table renamed via migration
  rename_era5_profiles_to_narr_profiles (ALTER TABLE + rename indexes,
  atomic and instant, no row rewrite)
- Weather helpers: find_nearest_era5/era5_for_contact/era5_profiles_for_path
  → find_nearest_narr/narr_for_contact/narr_profiles_for_path
- BackfillEnqueueWorker: :era5 → :narr type (virtual, no narr_status column);
  reconcile_stale_queued_to_unavailable and status_priority_order now skip
  virtual types via @virtual_types. Fixes the prod crash where the cron tried
  to update a non-existent era5_status column.
- ContactWeatherEnqueueWorker: build_era5_jobs → build_narr_jobs,
  era5_jobs_for_contact → narr_jobs_for_contact
- ContactLive: @era5 → @narr, maybe_enqueue_era5 → maybe_enqueue_narr;
  UI label "ERA5 (0.25°)" → "NARR (32 km)"
- Cron (runtime.exs): args types list now "narr" instead of "era5"
- /status: progress row, status-by-type card, totals.narr_profiles, and
  table-count lookup all target narr_profiles
- Drop obsolete Era5CdsJob schema + era5_cds_jobs table (inspection
  artifact from the retired CDS pipeline; 34 orphan rows in prod)
- Misc docstring/comment cleanups (skew_t, about, wgrib2, propagation_train)

Includes a regression test for the virtual-type crash.
2026-04-16 09:22:23 -05:00
f85fadecec
Retrain propagation ML on month-balanced HRRR → algorithm score
Replaces the two-phase (algorithm-scores pretrain + QSO-distance
finetune) pipeline with a single unbiased phase that samples
hrrr_profiles stratified uniformly by calendar month and trains on the
physical algorithm score from Scorer.composite_score/2.

Previous Phase 2 target was 'within-band distance percentile' computed
from the contacts table. Because non-contest months have almost no
QSOs, the model learned 'these conditions → QSO happened → good' and
implicitly 'no QSO → bad', so it systematically under-predicted
propagation during quiet months (Feb, Mar, Nov). The new pipeline
decouples the target from QSO activity entirely.

Stratification uses ROW_NUMBER() OVER (PARTITION BY month ORDER BY
random()) so every month contributes the same number of profile rows
regardless of contest-driven density. Each profile is exploded into
one training row per band.

Trained model: val RMSE=1.66 pts, R²=0.9744 on 840k rows (10k
profiles/month × 7 bands, 50 epochs). Monthly sanity check at fixed
weather conditions (10 GHz, 15°C/10°C) now shows Jan=Feb=Mar=68 — the
model no longer penalizes quiet months.
2026-04-14 08:16:08 -05:00
2c16fd979d
Move ML deps to dev/test only, exclude from production build
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
  elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
  compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
2026-04-01 12:10:31 -05:00