Commit graph

50 commits

Author SHA1 Message Date
1da78a80e5
feat(hrdps): activate Canadian propagation chain end-to-end
Stages 6, 7, 10 of the HRDPS plan, plus the Elixir read-side merge
that makes Rust's `.hrdps.prop` writes show up on /map.

Read side:

- `ScoresFile.path_for_hrdps/2` + `read_hrdps/2` — companions to the
  HRRR equivalents. Same decode path; different filename.
- `ScoresFile.read_bounds/3` now reads HRRR + HRDPS files and
  concatenates the cells. Files are disjoint by construction
  (Grid.hrdps_only_points excludes CONUS) so no de-dup needed.
- `ScoresFile.read_point/4` checks `.prop`, then `.hrdps.prop`, then
  legacy `.ntms` — first hit wins. Cells outside their owning region
  read as no-data in the other file's grid, so the order doesn't
  matter for correctness.
- `Propagation.warm_cache_and_broadcast/2` reads both files and
  merges before broadcasting to the cluster ScoreCache. A single
  missing file (HRDPS pre-cycle, HRRR briefly absent) is now OK —
  the cache warms with whichever side is available.

Activation:

- runtime.exs cron: HRDPS seed worker fires at HH:35 of each cycle
  hour (05:35Z, 11:35Z, 17:35Z, 23:35Z) — 5h after cycle, 30 min
  staggered from HRRR's */15 schedule.
- HrdpsGridWorker moduledoc updated: no longer dormant; Rust
  prop-grid-rs handles source='hrdps' rows.
- map_live North America bbox extended to (24.5, 60.0, -141.0, -52.0)
  for the out-of-coverage banner check; visitors anywhere in CONUS
  or the Canadian extent now stay banner-free.

Cleanup:

- mix hrdps.probe deleted — its risks are retired and the production
  HrdpsClient covers the same surface.

What's still in motion (not blocking the Canadian /map):

- Per-valid_time profile + scalar artifacts for HRDPS (would let
  /weather show Canadian temperature/refractivity layers). Rust
  pipeline currently skips those so the HRRR-written companion files
  aren't clobbered — separate stage when /weather UX is decided.
- FreshnessMonitor HRDPS staleness tracking — HRDPS has its own cron
  and a different cadence than HRRR; HRRR-stale logic doesn't apply
  cleanly. Defer until prod tells us what alarms we actually want.
2026-04-29 17:29:37 -05:00
14e81f9251
docs(hrdps): plan + probe wedge for Canadian propagation grid
Adds mix hrdps.probe — throwaway wedge that retires URL/projection/decode
risks against MSC Datamart. Verified against five Canadian cities with
plausible 2m temperatures via wgrib2's native rotated-lat/lon support.

Findings update what we knew:
- ECCC reorganized to date-prefixed paths; the old /model_hrdps/ root
  404s now. Real URL is dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/...
- HRDPS publishes one variable per GRIB2 file (~3.5 MB each), not
  HRRR's bundled wrfsfcf format. ~14 files per cycle/forecast hour.
- wgrib2 -lon LON LAT handles the rotated grid natively — no manual
  reprojection math needed.

New plan at 2026-04-29-hrdps-canadian-prop-grid.md supersedes the
2026-04-13 one (which assumed HRRR-style idx + byte-range fetches that
don't apply). Probe deletion is a Stage 10 step in the new plan.
2026-04-29 16:47:15 -05:00
021608dc90
feat(buildings): bulk-prefetch tiles via BulkFetch.run/4 (works in prod)
Mix tasks don't run in production, so the orchestration lives in
Microwaveprop.Buildings.BulkFetch — callable from an attached iex on
the prod pod via:

    iex> Microwaveprop.Buildings.BulkFetch.run(32.8, -97.0, 500)

Default 500 mi radius around DFW yields ~440 candidate quadkeys at
zoom 9; only those present in the MS dataset index (typically a few
hundred over CONUS) get downloaded. Concurrency 4, per-task timeout
3 min, results logged with downloaded/cached/failed/missing counts.
The mix.task is kept as a dev-only thin wrapper.
2026-04-26 10:37:13 -05:00
e4668790a4
refactor(weather): make rebatch logic callable from release eval
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
2026-04-24 13:05:52 -05:00
95c9ac3dcd
perf(weather): adaptive IemRateLimiter + rebatch mix task
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
  widens the current gap (*= 1.5, capped at max_interval_ms default
  10s); signal_success/0 narrows it back toward the configured base
  (*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
  ceiling without needing the manual "safe for 4 pods" constant.

- IemClient now routes every response through a central handle_response
  helper that fires the widen/narrow feedback signals, eliminating the
  four near-identical case blocks.

- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
  station "asos" jobs into the batched "asos_batch" shape the
  enqueuer now emits, so the pending backfill queue converts to the
  new per-request-efficient path instead of draining at the old rate.
  Idempotent; supports --dry-run.

2833 tests + credo green.
2026-04-24 12:57:25 -05:00
0c98707091
feat: harden /map analysis breakdown + move Plausible to root layout
Two fixes cover the blank "analysis breakdown" panel the user reported:

1. Bound the point_detail fallback lookback to 24h. Previously
   factors_from_fallback_profile would happily use a week-old analysis
   profile when no current one existed — now anything older than 24h
   returns an empty factor map so the UI can surface "breakdown
   unavailable" instead of silently misattributing.

2. Move the Plausible analytics snippet from Layouts.app into
   root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
   /weather, home) bypass Layouts.app, so the snippet only loaded on
   the navbar-wrapped pages. root.html.heex is loaded once per HTTP
   document so coverage is now universal.

Added ~30 tests locking both down:
  • point_detail fallback: exact-time wins, 24h boundary accepted,
    30h-stale rejected, multi-band coverage, missing-cell degrades to
    empty factors, default-valid_time path
  • analytics_test.exs: Plausible script present on 12 representative
    pages, exactly once, loaded async, surviving a login round-trip

Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
2026-04-21 15:56:30 -05:00
6c6e413105
fix(build): inline dep-patch logic into mix.exs so it runs during deps.get
Previously the `mix deps.patch` task lived at `lib/mix/tasks/deps.patch.ex`
and the `deps.get` alias invoked it. This broke `mix deps.get --only prod`
in Docker: `lib/` isn't compiled yet when deps are fetched, so the task
module couldn't be resolved and the build failed with
`The task "deps.patch" could not be found`.

Move the patch-application function directly into `mix.exs` as a private
helper and wire it into the alias via `&apply_dep_patches/1`. `mix.exs` is
always loadable, so the function is available the moment `deps.get` runs.
2026-04-21 14:06:24 -05:00
1400c38f44
fix(dialyzer): actually fix LiveTable warnings instead of hiding them
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.

Changes:

- Delete .dialyzer_ignore.exs and remove ignore_warnings from
  mix.exs. No more hidden suppressions.

- New MicrowavepropWeb.LiveTableResource shim that wraps
  use LiveTable.LiveResource and overrides:
  * maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
    (previously discarded, tripping :unmatched_return).
  * handle_info/2 — explicit _ = on File.rm, Process.send_after.
  Switch all 4 LiveTable-using LiveViews (contact_live/index,
  beacon_live/index, admin/contact_edit_live, user_management_live/index)
  to use MicrowavepropWeb.LiveTableResource instead.

- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
  LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
  the dep's macro-expanded handle_params/3. The call's return was
  silently discarded, tripping the fourth warning per LiveView.

- New lib/mix/tasks/deps.patch.ex applies every
  priv/dep_patches/*.patch to its target dep idempotently after
  mix deps.get. Wired into the :setup alias + overridden
  mix deps.get so the patch survives cache regeneration in CI.

- Other modified files in this commit are the spec-coverage
  additions from the parallel agent pass (beacons.ex,
  commercial.ex, propagation.ex, weather.ex, narr_client.ex,
  run_timing.ex, 3 workers) — tighter specs for bounds, tuple
  shapes, schema-typed params. Dialyzer remains at 0 after
  the changes.

Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.

mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
2026-04-21 12:58:10 -05:00
d61fbd346e
fix(dialyzer): clear 125+ warnings under strict flags
Enabled :error_handling, :unknown, :unmatched_returns, :extra_return,
:missing_return in an earlier commit and landed a 129-warning baseline.
Four parallel agents each fixed a directory slice:

- Core contexts (29): Radio, Release, Weather, Beacons, Cache,
  Backtest.Features, Terrain.Srtm, Ionosphere.GiroClient,
  Propagation.RunTiming, Accounts.Scope, RepoListener. Fixes were
  (a) prefix side-effect calls (Task.start, Phoenix.PubSub,
  Logger, :ets.new) with _ = ; (b) tighten/widen specs that didn't
  match actual returns; (c) add missing @type t declarations;
  (d) drop dead parse_int(nil) clause.

- Propagation + weather subdirs (15): FreshnessMonitor, NotifyListener,
  ScoreCache, ScoreCacheReconciler, Weather.FrontalAnalysis,
  Weather.Grib2.Extractor, Weather.Grib2.Wgrib2, GridCache,
  HrrrPointEnqueuer, NexradCache. Same patterns — mostly _ = on
  PubSub / :ets / Repo.insert_all; widened two specs (float ->
  number) where integer returns were reachable.

- Workers (35): BackfillEnqueue, CanadianSoundingFetch,
  ContactImport, ContactWeatherEnqueue, GefsFetch, IemreFetch,
  NarrFetch, SolarIndex, TerrainProfile, WeatherFetch. Prefixed
  Repo.update_all / Radio.set_enrichment_status! / Weather.upsert_*
  side-effect calls. Fixed one :pattern_match in
  CanadianSoundingFetch.most_recent_sounding_time/1 where a
  tautological cond guard generated unreachable code.

- Web + Mix tasks + lib_ml (46 of 50): controllers, LiveViews,
  UserAuth, and 11 mix tasks. Same prefix strategy. 4 remaining
  warnings originate in LiveTable.LiveResource dep macro expansion
  and can't be fixed without forking the dep — added .dialyzer_ignore.exs
  to suppress just those specific file:line pairs.

Also wired ignore_warnings in mix.exs dialyzer config.

mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2163 tests, 3 pre-existing flakes, 0 regressions.
2026-04-21 10:30:06 -05:00
b80878056d
feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.

Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.

Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
  - grid, region, band_config, scores_file, sounding_params
  - scorer: all 10 factors + composite. Matches the Elixir scorer
    byte-for-byte across 115 golden-fixture samples (5 scenarios
    × 23 bands).
  - decoder: wgrib2 subprocess + Fortran-record lola-binary parser
  - fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
    parallel byte-range downloads with 429/5xx retry
  - pipeline: end-to-end chain step, f00 rejected at the boundary
  - db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
  - bin/worker: tokio main loop, JSON logs, SIGTERM-safe

91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.

Elixir additions:
  - `GridTaskEnqueuer` seeds fh=1..18 rows from
    `PropagationGridWorker.seed_chain/0`
  - `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
  - `ShadowComparator` diffs prod vs shadow `.ntms` bodies
  - `mix rust.golden` task writes the Rust-side golden fixture

k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.

The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
2026-04-19 15:42:49 -05:00
33f5d4edbe
feat(rainscatter): classify QSO propagation mechanism from common-volume radar
Adds a per-contact enrichment pipeline that determines whether a QSO was
most likely carried by rain scatter, tropospheric ducting, or ordinary
troposcatter — using IEM n0q composite reflectivity sampled inside the
lens-shaped intersection of 400 km-radius disks around each endpoint.

Pieces:
  * Microwaveprop.Propagation.CommonVolume — lens geometry (haversine,
    in-CV test, bbox, area).
  * contact_common_volume_radar table (1:1 per contact) storing
    aggregate dBZ stats inside the CV + radar_status column on contacts.
  * Microwaveprop.Workers.CommonVolumeRadarWorker — Oban :radar queue,
    fetches the n0q frame at QSO time, iterates pixels inside the CV
    bbox, aggregates rain/heavy/core-pixel counts, max/mean dBZ, and
    coverage percentage.
  * Microwaveprop.Propagation.RainScatterClassifier — rule-based mapper
    from (band, distance, radar stats, duct flags) to one of
    :likely_rainscatter | :rainscatter_possible | :tropo_duct |
    :troposcatter | :unknown.
  * ContactWeatherEnqueueWorker learns a :radar enrichment type and
    enqueues the CV worker on contact submission; pre-2014 contacts
    (outside IEM n0q coverage) are pinned to :unavailable.
  * `mix radar_backfill` bulk-enqueues historical contacts with
    --year / --limit / --dry-run.
  * Contact detail page renders a mechanism badge with supporting
    stats (common-volume area, max dBZ, heavy-rain pixel count,
    coverage %).
2026-04-17 15:57:59 -05:00
1f2a729d40
Drop dead Era5*/CDS code paths
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.

Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs

Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
  table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
  (the symbol still means "historical backfill", just routed to NARR now)

Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).

Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
2026-04-16 08:26:29 -05:00
02b354f5f7
Purge legacy is_grid_point=true rows from hrrr_profiles
The grid worker stopped writing HRRR profiles to the DB on April 14
when the forecast grid moved to /data/scores binary files, but the
24/72h-windowed prune left every is_grid_point=true row older than
72 hours sitting in the table (billions of rows across every
partition back to 2016, ~90% of the 146 GB table size). Nothing
reads them anymore.

Replace prune_old_grid_profiles/0 with purge_grid_point_profiles/0,
which walks every hrrr_profiles partition directly and deletes
is_grid_point=true rows regardless of age. Per-partition DELETEs
avoid a parent-table full scan. QSO-linked rows (is_grid_point=false)
are untouched — those still back contact enrichment and /path.

Call the new purge from the grid worker's end-of-chain hook so any
grid-point row that somehow slips back in gets cleaned up within the
next hour. Add `mix hrrr.purge_grid_points` for a one-shot historical
cleanup.
2026-04-15 12:14:20 -05:00
811251dd15
Drop propagation_scores table, ScoresFile is the only store
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:

1. New migration drops the propagation_scores table and its indexes
   (the earlier tuning migrations for it were already applied and
   are now no-ops against a missing table, which is fine — Ecto
   just runs them on fresh environments).

2. Propagation context is gutted of every GridScore reference.
   replace_scores/2 writes files only. upsert_scores/2 is deleted.
   load_scores_from_db, available_valid_times_from_db,
   point_detail_from_db, point_forecast_from_db, fetch_factors,
   coalesce_factors, the Postgres side of prune_old_scores, and
   the Postgres fallbacks in latest/earliest_valid_time are all
   removed. point_detail always returns an empty factors map now
   since factor breakdowns were retired with the table.

3. Deleted modules:
   - Propagation.GridScore (the schema)
   - Propagation.ScorerDiff (read factors from the table)
   - Propagation.AsosNudge (helper for AsosAdjustmentWorker)
   - Workers.AsosAdjustmentWorker (its cron was already disabled)
   - Mix.Tasks.ScorerDiff (wrapper around the deleted module)
   And their tests. AdminTaskWorker's scorer_diff task is a
   logging no-op so any queued Oban rows drain cleanly.
   Release.scorer_diff stays as a stub that tells the operator.

propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
2026-04-14 15:13:01 -05:00
7fb340bc35 Fix all remaining credo --strict issues (0 issues)
Aliases: add module aliases for 9 nested module references
Apply: replace apply/3 with direct module attribute calls
Line length: break 1 long spec line
Refactoring: extract helpers to reduce complexity and nesting
in show.ex, radio.ex, weather workers, terrain, duct detection,
backfill dashboard, contact map, and mix tasks
2026-04-12 10:26:53 -05:00
9abbb83469 Fix credo warnings: struct specs, length/1, and test patterns
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
2026-04-12 10:26:53 -05:00
1174ecd9e5 Fix all 12 dialyzer warnings
- Replace MapSet with plain list + `in` (features.ex, scorer_diff.ex)
- Remove undefined Beacon.t() type reference (range_estimate.ex)
- Remove dead else branch in find_region (inversion.ex)
- Handle Nx special values in to_float catch-all (recalibrator.ex)
- Remove unreachable catch-all clauses (hrrr_native_client.ex, ncei_metar_client.ex)
- Remove unnecessary nil guards on always-typed values (show.ex)
- Remove dead sky_note/wind_note non-nil clauses (show.ex)
- Remove dead if-guard on always-truthy derive result (hrrr_native_derive.ex)
- Add @spec to path_integrated_conditions (scorer.ex)
2026-04-11 18:08:18 -05:00
e7a7ae073d Phase 9.3, 9.4, and Phase 3 NEXRAD pipeline
Task 9.3 - Weight recalibration via gradient descent:
- Recalibrator module fits logistic regression weights using Nx
- Trains on QSO positives vs random baseline negatives
- Cross-validates by month, normalizes weights to sum to 1.0
- Mix task: mix recalibrate_scorer --sample 5000 --epochs 2000

Task 9.4 - Side-by-side scorer comparison:
- ScorerDiff.compare/3 re-scores grid with old vs new weights
- Reports mean diff, regressions, improvements, per-band breakdown
- Mix task: mix scorer_diff --new-weights '{...}'

Phase 3 - NEXRAD ingestion pipeline:
- NexradClient fetches IEM n0q composite PNGs, extracts per-point
  box statistics (mean/max dBZ, texture variance)
- NexradObservation schema with unique (lat, lon, observed_at)
- NexradWorker on :nexrad queue for background processing
- nexrad_texture backtest feature in Features module
- mix nexrad_backfill --limit 200

All tasks added to AdminTaskWorker and Release for production use.
1116 tests, 0 failures.
2026-04-10 12:48:36 -05:00
6d92973853 Phase 9.1: Consolidated backtest report and contest log import
- Add `mix backtest --all` for consolidated pass/fail table across all features
- Add Backtest.consolidated_report/2 and to_consolidated_markdown/1
- Add Features.all_features/0 to auto-discover backtestable features
- Add `mix import_contest_logs` for bulk ARRL contest CSV import with dedup
- Fix hrrr_climatology to batch by (month, hour) to avoid query timeout
- Fix Repo.query! result pattern (Postgrex.Result, not tuple)
- Backtest reports for all Phase 1-6 features
2026-04-10 11:57:15 -05:00
65e3159a85 Pause Oban queues in all mix tasks
Mix tasks that call app.start were also booting Oban's cron scheduler,
causing PropagationGridWorker and other cron jobs to fire during
backfills. Add Oban.pause_all_queues(Oban) immediately after app.start
in every mix task that only needs Repo access.
2026-04-10 09:13:30 -05:00
0f0e5e8d43 Phase 3 spike (parked) + Phase 6: temperature anomaly
Phase 3 NEXRAD spike: IEM n0q composite available at 5-min cadence
back to 2022+. Compression-ratio proxy shows afternoon images have
13-81% more texture than dawn (directionally correct), but the n0q
product thresholds out the faint clear-air returns needed for BL
stability detection. Parked until MRMS or Level III products can be
investigated. See docs/research/nexrad_spike.md.

Phase 6: hrrr_climatology table aggregating surface_temp_c by
(lat, lon, month, hour) from the 42M+ hrrr_profiles grid-point
rows. mix hrrr_climatology builds it via a single SQL GROUP BY +
upsert. Backtest.Features.temperature_anomaly computes current_temp
minus climatological mean — the meteorologist's "temperature
deviation above normal" predictor for summer afternoon enhancement.
2026-04-10 08:47:11 -05:00
82bf248ab7 Phase 4: Ray-traced duct geometry
Duct module (Propagation.Duct):
- refractivity_profile/1: ITU-R P.453 N at each native level
- m_profile/1: modified refractivity M = N + 157*h(km)
- detect_ducts/1: find contiguous regions where dM/dh < 0, returning
  base/top height, thickness, and M-deficit per duct
- min_trapped_frequency_ghz/1: waveguide approximation (Bean & Dutton)
  for the minimum frequency a duct of given geometry can trap
- analyze/1: full pipeline from native profile to duct list + best
  trapped frequency across all ducts

Derive task updated to also compute ducts JSONB and best_duct_band_ghz
alongside the Phase 2 turbulence fields.

Backtest features: duct_thickness, best_duct_freq, duct_usable_10ghz,
duct_usable_24ghz, duct_usable_47ghz.

Real-data validation: 2022-08-20 12Z TX profile shows 0 ducts (M
increases monotonically) — correct for a well-mixed boundary layer
on a turbulent August afternoon (Ri=0.16).
2026-04-10 08:31:16 -05:00
864a91fc5c Phase 2 tasks 2.1-2.4: BL turbulence feature computations
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
  temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
  inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286

Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric

mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.

First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
2026-04-10 08:21:23 -05:00
900685aa06 Phase 1 tasks 1.1-1.5: HRRR native hybrid-sigma ingestion
- Spike docs at docs/research/hrrr_native_levels.md confirming files
  are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
  SPFH needed for Phase 2 turbulence features. Architectural finding:
  per-point on-demand fetching is impractical (~530 MB/file), so
  the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
  scalars and placeholder columns for Phase 2/4 derived fields.
  Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
  that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
  reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
  unique at :infinity, pulls distinct (lat, lon) points from contacts
  in the ±30 min window, downloads the native grib2, extracts per
  point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
  contact count.

Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
2026-04-09 16:23:51 -05:00
7630934bcb Accept CamelCase feature names in mix backtest
mix backtest --feature NaiveGradient now resolves to the same function
as --feature naive_gradient. Names are normalized via Macro.underscore,
so any casing works. Also prints the available functions when an
unknown feature is requested so typos don't produce an opaque
UndefinedFunctionError deep in the Enum.map stack.
2026-04-09 16:15:01 -05:00
ab04cb9168 Phase 0: backtest harness
Add Microwaveprop.Backtest: a feature-evaluation framework that runs
a (lat, lon, valid_time) -> float function over the historical QSO
corpus and a matched random-time baseline, reporting distribution
statistics, distance-binned lift, and band-stratified lift.

Adds four baseline feature wrappers around the current scorer inputs
(naive_gradient, td_depression, time_of_day, pressure), a mix backtest
CLI, and the first set of baseline reports under priv/backtest_reports
so downstream phases have a frozen reference point to compare against.
2026-04-09 16:10:54 -05:00
8637253fda Batch ERA5 fetches by month and 2° tile
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.

- Era5BatchClient builds the monthly CDS requests, extracts every
  (lat, lon, hour) from the GRIB2 blob with wgrib2, derives
  refractivity params, and bulk-inserts in 2k-row chunks with
  on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
  the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
  generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
  → enqueue the month-batch for the point's tile-month and return.
  No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
  datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
  single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
2026-04-09 13:01:49 -05:00
6476bc8045 Dedupe ERA5 fetch jobs by (lat, lon, valid_time)
The Era5FetchWorker now declares an Oban unique constraint on
{lat, lon, valid_time} for any job in available/scheduled/executing/
retryable, so two backfill runs targeting the same contact grid point
can no longer spawn parallel CDS requests for the same hour.

Because Oban OSS insert_all doesn't honor unique, ERA5 jobs are now
routed through Oban.insert/1 from ContactWeatherEnqueueWorker and the
era5_backfill mix task. Other worker types still use insert_all.
2026-04-08 15:52:53 -05:00
ddf86ed806 Add mix era5_backfill task and gitignore .envrc
Enqueues ERA5 fetch jobs for contacts where HRRR is unavailable.
Deduplicates by rounded grid point and hour, skips existing profiles.

Usage: mix era5_backfill [limit]
2026-04-07 12:24:18 -05:00
548e277bec Add bin/notebook script to start Phoenix + Livebook together
Starts Phoenix with a named node and Livebook attached to it on
port 8081. Single command, Ctrl-C stops both.
2026-04-07 11:15:49 -05:00
44f0ff26e5 Add mix notebook task for Livebook attached to dev node
Run `mix notebook` for instructions to start Livebook at localhost:8081
connected to the running Phoenix node with full Repo/Scorer access.
2026-04-07 11:15:49 -05:00
d275e9e7c4
fix enrichment 2026-04-02 15:30:41 -05:00
55a51f69ab
Replace boolean enrichment flags with enum status fields
- New fields: hrrr_status, weather_status, terrain_status, iemre_status
  with values: pending, queued, processing, complete, failed, unavailable
- EnrichmentStatus module defines valid state transitions
- Migration backfills: queued=true → complete, false → pending
- Workers set status on transitions (queued on enqueue, complete on finish)
- Show page reads status from contact struct, not computed dynamically
- Backfill dashboard queries status fields for accurate progress counts
- Remove cron-scheduled ContactWeatherEnqueueWorker (enrichment only on
  submission, page view, or manual backfill)
- Partial indexes on status != 'complete' for fast unprocessed lookups
2026-04-02 12:07:51 -05:00
1b5cd19d6d
Skip already-backfilled HRRR points with sufficient level count 2026-04-01 15:52:35 -05:00
ff0ca89646
Add HRRR GRIB2 caching and backfill task for finer pressure levels
- Cache raw GRIB2 byte ranges to ~/hrrr in dev (never deleted)
- mix hrrr_backfill re-fetches QSO-linked HRRR profiles with 13 levels
- Supports --all (re-fetch everything) and --limit N flags
- Groups by HRRR hour to batch requests, 500ms rate limit
- AWS archive has full HRRR history, same URL pattern as live feed
2026-04-01 15:50:55 -05:00
2c16fd979d
Move ML deps to dev/test only, exclude from production build
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
  elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
  compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
2026-04-01 12:10:31 -05:00
254e64dedc
Rename qsos to contacts throughout codebase, keep DB table name
Rename all modules, functions, variables, routes, and UI text from
qso/qsos to contact/contacts. Database table stays as "qsos" to avoid
migration. Add /qsos -> /contacts redirects for old URLs.
2026-04-01 11:25:04 -05:00
489e188956
Add --no-pretrain flag for direct QSO training comparison 2026-04-01 09:36:32 -05:00
9537c97d1d
20-feature model with solar indices, sounding stability, and ducting
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
2026-04-01 09:31:54 -05:00
07558d17eb
Two-phase training: pretrain on algorithm scores, fine-tune on QSOs
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
2026-04-01 09:27:27 -05:00
08e4b9abdd
Bigger network (128→64→32) and percentile-based training target
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
  normalized distance — reduces noise from operator/equipment variation
2026-04-01 09:17:36 -05:00
10cbbf47b3
Train ML model on real QSO-HRRR data instead of algorithm scores
Features: HRRR conditions averaged at both QSO endpoints + solar time
Target: distance_km normalized per-band (distance / p99_range, capped at 1.0)
This trains on actual propagation outcomes from 57K+ QSOs, not the
hand-tuned algorithm output.
2026-04-01 09:13:05 -05:00
69b5caf876
Normalize ML features to prevent NaN gradient explosion
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.
2026-04-01 09:08:33 -05:00
5c9b43d619
Update analysis with solar time, refine algo.md time-of-day section
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
2026-04-01 09:02:57 -05:00
c12f8cf5ed
Use local solar time for time-of-day scoring, add PWAT factor and pressure refinements
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
2026-04-01 08:58:21 -05:00
4bf9d0b4b0
Add mix propagation.analyze task for QSO-HRRR correlation analysis 2026-04-01 08:34:04 -05:00
e82e631135
HRRR forecast hours (f00-f18) with timeline map UI
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2)
- PropagationGridWorker fetches all 19 forecast hours per run
- Propagation.scores_at/3 queries scores at specific valid_time
- Propagation.available_valid_times/1 returns all forecast times for timeline
- Pruning keeps scores with valid_time >= now - 2h (forecast-aware)
- MapLive: select_time event, timeline data pushed to JS
- JS: forecast timeline bar at bottom of map with clickable hour buttons
- PubSub broadcast sends list of valid_times instead of single time
2026-03-31 16:44:47 -05:00
9c504acb67
Disable Oban queues in mix propagation_grid to prevent backfill noise 2026-03-30 17:36:32 -05:00
1e205cb471
Add propagation context, grid worker, and fix merge issues
- Replace stub propagation.ex with real implementation (score_grid_point,
  upsert_scores, latest_scores, latest_valid_time)
- Add PropagationGridWorker (hourly Oban cron) for CONUS grid scoring
- Add mix propagation_grid task for manual triggering
- Fix duplicate extract_grid/2 from parallel merges
- Fix extract_grid to skip outside-grid points instead of erroring
- Add propagation queue to Oban config
2026-03-30 17:18:05 -05:00
ded9c054ac
Multi-point path enrichment for HRRR, IEMRE, and weather data
Enqueue worker now gathers atmospheric data at pos1, midpoint, and pos2
along each QSO path instead of only pos1. Existing has_* guards prevent
duplicate fetches at each grid point.

- Add Radio.qso_path_points/1 for path point extraction
- Update hrrr_job_for_qso, iemre_job_for_qso, jobs_for_qso to iterate path points
- Refactor Weather into find_nearest_hrrr/3 and find_nearest_iemre/3
- Add hrrr_profiles_for_path/1 and iemre_for_path/1 query functions
- Add mix reset_enrichment task to trigger re-processing
2026-03-30 16:09:01 -05:00