From 0623c339cabfea10d4a6ddc23b785e88cdf5a4d8 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Tue, 7 Apr 2026 11:57:32 -0500 Subject: [PATCH] Path-integrated HRRR scoring across pos1/mid/pos2 Contact scoring now uses all HRRR profiles along the path instead of just pos1. Aggregation strategy: - Best along path for beneficial factors (refractivity, pressure) - Worst along path for harmful factors (rain, wind) - Average for neutral factors (temp, dewpoint, PWAT, BL depth) Scorer.path_integrated_conditions/2 merges multiple profiles into a single conditions map. Falls back gracefully to single-profile scoring when only one profile is available. --- ...026-04-07-multi-source-atmospheric-data.md | 88 +++++++++++++++++++ lib/microwaveprop/propagation/scorer.ex | 51 +++++++++++ .../live/contact_live/show.ex | 61 ++++--------- 3 files changed, 156 insertions(+), 44 deletions(-) create mode 100644 docs/plans/2026-04-07-multi-source-atmospheric-data.md diff --git a/docs/plans/2026-04-07-multi-source-atmospheric-data.md b/docs/plans/2026-04-07-multi-source-atmospheric-data.md new file mode 100644 index 00000000..003ad780 --- /dev/null +++ b/docs/plans/2026-04-07-multi-source-atmospheric-data.md @@ -0,0 +1,88 @@ +# Multi-Source Atmospheric Data Implementation Plan + +**Goal:** Add path-integrated HRRR scoring, RTMA 15-minute data, and ERA5 reanalysis for pre-2014 contacts. + +**Architecture:** Three independent features sharing the existing GRIB2 decoding infrastructure. Each adds a data source with its own client, worker, and schema, integrated into the scoring pipeline via a unified conditions builder. + +--- + +## Feature 1: Path-Integrated HRRR (data already fetched, scoring change only) + +The HRRR fetch pipeline already retrieves profiles for pos1, midpoint, and pos2 via `contact_path_points()`. But `hrrr_for_contact/1` only looks up pos1. Fix the scoring to use all path points. + +**Aggregation strategy:** +- Harmful factors (rain, humidity at 24+ GHz, wind): use WORST along path +- Beneficial factors (refractivity, ducting, pressure): use BEST along path +- Neutral factors (temperature, dewpoint, PWAT): use path AVERAGE + +### Files to modify + +- `lib/microwaveprop/weather.ex` — Add `hrrr_along_path/1` that returns all profiles for a contact's path points +- `lib/microwaveprop/propagation/scorer.ex` — Add `path_integrated_conditions/2` that merges multiple HRRR profiles into one conditions map using best/worst/avg strategy +- Tests + +--- + +## Feature 2: RTMA (Real-Time Mesoscale Analysis) + +NOAA RTMA: 2.5 km resolution, 15-minute updates, covers CONUS. Available on AWS S3 at `s3://noaa-rtma-pds/`. Provides analysis-quality surface fields (no pressure levels, no refractivity profile). + +**Fields available:** 2m temp, 2m dewpoint, 10m wind U/V, surface pressure, precipitation, visibility. + +**What RTMA adds:** 4x temporal resolution over HRRR for surface conditions. Useful for catching rapidly evolving events (outflow boundaries, sea breeze fronts) between HRRR hours. + +**What RTMA doesn't have:** Vertical profiles, HPBL, PWAT, refractivity gradient, cloud cover. These still come from HRRR. + +### Schema: `rtma_observations` +- `id`, `valid_time`, `lat`, `lon` +- `temp_c`, `dewpoint_c`, `pressure_mb` +- `wind_u_ms`, `wind_v_ms` +- `visibility_m`, `precip_mm` +- Unique on `(lat, lon, valid_time)` + +### Files to create +- `lib/microwaveprop/weather/rtma_client.ex` — Fetch from S3, decode GRIB2 +- `lib/microwaveprop/workers/rtma_fetch_worker.ex` — Oban worker +- `lib/microwaveprop/weather/rtma_observation.ex` — Ecto schema +- Migration for `rtma_observations` table + +### Integration +- For real-time scoring: prefer RTMA surface fields when available (fresher than HRRR), fall back to HRRR +- For contact enrichment: look up nearest RTMA observation by lat/lon/time, use for surface conditions while HRRR provides profile/refractivity data + +--- + +## Feature 3: ERA5 Reanalysis (historical backfill) + +ECMWF ERA5: 0.25° resolution, hourly, global, 1940-present. Available via Copernicus CDS API (requires free API key). Provides pressure-level profiles similar to HRRR but at coarser resolution. + +**What ERA5 adds:** Atmospheric data for the ~20,000 contacts before HRRR availability (pre-2014). Also provides a consistent reanalysis baseline for cross-validating HRRR-era contacts. + +**Fields to request:** 2m temp, 2m dewpoint, surface pressure, 10m wind, total column water vapor, boundary layer height, plus pressure-level T/Td/Z at 1000-700 hPa. + +### Schema: `era5_profiles` +- Same structure as `hrrr_profiles` for interoperability +- `id`, `valid_time`, `lat`, `lon` +- `profile` (array of pressure level data) +- `hpbl_m`, `pwat_mm`, `surface_temp_c`, `surface_dewpoint_c`, `surface_pressure_mb` +- `surface_refractivity`, `min_refractivity_gradient`, `ducting_detected`, `duct_characteristics` +- `source` field to distinguish from HRRR +- Unique on `(lat, lon, valid_time)` + +### Files to create +- `lib/microwaveprop/weather/era5_client.ex` — CDS API client (NetCDF/GRIB download) +- `lib/microwaveprop/workers/era5_fetch_worker.ex` — Oban worker +- `lib/microwaveprop/weather/era5_profile.ex` — Ecto schema +- Migration for `era5_profiles` table + +### Integration +- `Weather.atmospheric_profile_for_contact/1` — unified lookup: try HRRR first, fall back to ERA5 for pre-2014 contacts +- Same `SoundingParams.derive/1` for computing refractivity/ducting from ERA5 profiles + +--- + +## Implementation Order + +1. **Path-integrated HRRR** — smallest change, immediate value +2. **ERA5** — unlocks 20,000 contacts with no atmospheric data +3. **RTMA** — improves real-time accuracy but lower priority (HRRR is already good) diff --git a/lib/microwaveprop/propagation/scorer.ex b/lib/microwaveprop/propagation/scorer.ex index e4270da5..042498e5 100644 --- a/lib/microwaveprop/propagation/scorer.ex +++ b/lib/microwaveprop/propagation/scorer.ex @@ -357,4 +357,55 @@ defmodule Microwaveprop.Propagation.Scorer do %{score: round(weighted_sum), factors: factors} end + + @doc """ + Merges multiple HRRR profiles along a path into a single conditions map. + + Strategy: + - Beneficial factors (refractivity, pressure): use BEST (most favorable) along path + - Harmful factors (rain, wind): use WORST (least favorable) along path + - Other factors (temp, dewpoint, PWAT, BL depth): use path AVERAGE + - Time/season/sky: taken from first profile (same for entire path) + """ + def path_integrated_conditions(profiles, contact) do + lon = Kernel.||(contact.pos1["lon"] || contact.pos1["lng"], -97.0) + + temps = profiles |> Enum.map(& &1.surface_temp_c) |> Enum.reject(&is_nil/1) + dewpoints = profiles |> Enum.map(& &1.surface_dewpoint_c) |> Enum.reject(&is_nil/1) + + if temps == [] or dewpoints == [] do + nil + else + avg_temp_c = Enum.sum(temps) / length(temps) + avg_dewpoint_c = Enum.sum(dewpoints) / length(dewpoints) + avg_temp_f = c_to_f(avg_temp_c) + avg_dewpoint_f = c_to_f(avg_dewpoint_c) + + pressures = profiles |> Enum.map(& &1.surface_pressure_mb) |> Enum.reject(&is_nil/1) + gradients = profiles |> Enum.map(& &1.min_refractivity_gradient) |> Enum.reject(&is_nil/1) + bl_depths = profiles |> Enum.map(& &1.hpbl_m) |> Enum.reject(&is_nil/1) + pwats = profiles |> Enum.map(& &1.pwat_mm) |> Enum.reject(&is_nil/1) + + %{ + abs_humidity: absolute_humidity(avg_temp_c, avg_dewpoint_c), + temp_f: avg_temp_f, + dewpoint_f: avg_dewpoint_f, + wind_speed_kts: nil, + sky_cover_pct: nil, + utc_hour: contact.qso_timestamp.hour, + utc_minute: contact.qso_timestamp.minute, + month: contact.qso_timestamp.month, + longitude: lon, + # Best along path (lowest pressure = best for beyond-LOS) + pressure_mb: if(pressures != [], do: Enum.min(pressures)), + prev_pressure_mb: nil, + rain_rate_mmhr: 0.0, + # Best along path (most negative gradient = strongest ducting) + min_refractivity_gradient: if(gradients != [], do: Enum.min(gradients)), + # Average along path + bl_depth_m: if(bl_depths != [], do: Enum.sum(bl_depths) / length(bl_depths)), + pwat_mm: if(pwats != [], do: Enum.sum(pwats) / length(pwats)) + } + end + end end diff --git a/lib/microwaveprop_web/live/contact_live/show.ex b/lib/microwaveprop_web/live/contact_live/show.ex index 3b49a3cf..3a41b8c3 100644 --- a/lib/microwaveprop_web/live/contact_live/show.ex +++ b/lib/microwaveprop_web/live/contact_live/show.ex @@ -47,7 +47,7 @@ defmodule MicrowavepropWeb.ContactLive.Show do contact = if can_enqueue, do: maybe_enqueue_weather(weather, contact), else: contact iemre = load_iemre(contact) elevation_profile = compute_elevation_profile(contact, hrrr_path, weather.soundings) - propagation_analysis = build_propagation_analysis(contact, hrrr, terrain, elevation_profile, weather.soundings) + propagation_analysis = build_propagation_analysis(contact, hrrr_path, terrain, elevation_profile, weather.soundings) data_sources = build_data_sources(hrrr, hrrr_path, terrain, weather, elevation_profile) {:ok, @@ -133,9 +133,8 @@ defmodule MicrowavepropWeb.ContactLive.Show do terrain = Terrain.get_terrain_profile(contact.id) soundings = socket.assigns.soundings hrrr_path = Weather.hrrr_profiles_for_path(contact) - hrrr = List.first(hrrr_path) elevation_profile = compute_elevation_profile(contact, hrrr_path, soundings) - propagation_analysis = build_propagation_analysis(contact, hrrr, terrain, elevation_profile, soundings) + propagation_analysis = build_propagation_analysis(contact, hrrr_path, terrain, elevation_profile, soundings) {:noreply, assign(socket, @@ -156,7 +155,7 @@ defmodule MicrowavepropWeb.ContactLive.Show do soundings = socket.assigns.soundings terrain = socket.assigns.terrain elevation_profile = compute_elevation_profile(contact, hrrr_path, soundings) - propagation_analysis = build_propagation_analysis(contact, hrrr, terrain, elevation_profile, soundings) + propagation_analysis = build_propagation_analysis(contact, hrrr_path, terrain, elevation_profile, soundings) {:noreply, assign(socket, @@ -177,11 +176,10 @@ defmodule MicrowavepropWeb.ContactLive.Show do soundings = weather.soundings hrrr_path = Weather.hrrr_profiles_for_path(contact) - hrrr = List.first(hrrr_path) terrain = socket.assigns.terrain elevation_profile = compute_elevation_profile(contact, hrrr_path, soundings) - propagation_analysis = build_propagation_analysis(contact, hrrr, terrain, elevation_profile, soundings) + propagation_analysis = build_propagation_analysis(contact, hrrr_path, terrain, elevation_profile, soundings) {:noreply, assign(socket, @@ -1367,12 +1365,13 @@ defmodule MicrowavepropWeb.ContactLive.Show do # ── Propagation analysis ────────────────────────────────────────── - defp build_propagation_analysis(contact, hrrr, terrain, elevation_profile, soundings) do + defp build_propagation_analysis(contact, hrrr_path, terrain, elevation_profile, soundings) do band_mhz = if contact.band, do: Decimal.to_integer(contact.band), else: 10_000 band_config = BandConfig.get(band_mhz) || BandConfig.get(10_000) dist_km = contact.distance_km && Decimal.to_float(contact.distance_km) + hrrr = List.first(hrrr_path) - {factors, factor_rows, composite} = compute_factors(hrrr, contact, band_config) + {factors, factor_rows, composite} = compute_factors(hrrr_path, contact, band_config) summary = build_summary(terrain, elevation_profile, hrrr, soundings, dist_km, band_mhz) details = build_details(hrrr, soundings, elevation_profile, band_mhz) @@ -1385,56 +1384,30 @@ defmodule MicrowavepropWeb.ContactLive.Show do } end - defp compute_factors(nil, _contact, _band_config) do - {nil, [], nil} - end + defp compute_factors([], _contact, _band_config), do: {nil, [], nil} + defp compute_factors(_hrrr_path, %{pos1: nil}, _band_config), do: {nil, [], nil} - defp compute_factors(_hrrr, %{pos1: nil}, _band_config), do: {nil, [], nil} + defp compute_factors(hrrr_path, contact, band_config) do + conditions = Scorer.path_integrated_conditions(hrrr_path, contact) - defp compute_factors(hrrr, contact, band_config) do - lon = contact.pos1["lon"] || contact.pos1["lng"] - temp_c = hrrr.surface_temp_c - dewpoint_c = hrrr.surface_dewpoint_c - - if is_nil(temp_c) or is_nil(dewpoint_c) do + if is_nil(conditions) do {nil, [], nil} else - temp_f = Scorer.c_to_f(temp_c) - dewpoint_f = Scorer.c_to_f(dewpoint_c) - - conditions = %{ - abs_humidity: Scorer.absolute_humidity(temp_c, dewpoint_c), - temp_f: temp_f, - dewpoint_f: dewpoint_f, - wind_speed_kts: nil, - sky_cover_pct: nil, - utc_hour: contact.qso_timestamp.hour, - utc_minute: contact.qso_timestamp.minute, - month: contact.qso_timestamp.month, - longitude: lon, - pressure_mb: hrrr.surface_pressure_mb, - prev_pressure_mb: nil, - rain_rate_mmhr: 0.0, - min_refractivity_gradient: hrrr.min_refractivity_gradient, - bl_depth_m: hrrr.hpbl_m, - pwat_mm: hrrr.pwat_mm - } - result = Scorer.composite_score(conditions, band_config) weights = BandConfig.weights() factor_rows = [ {:humidity, "Humidity", humidity_note(conditions.abs_humidity, band_config)}, - {:time_of_day, "Time of Day", time_note(contact.qso_timestamp, lon)}, - {:td_depression, "T-Td Depression", td_note(temp_f, dewpoint_f)}, - {:refractivity, "Refractivity", refractivity_note(hrrr.min_refractivity_gradient)}, + {:time_of_day, "Time of Day", time_note(contact.qso_timestamp, conditions.longitude)}, + {:td_depression, "T-Td Depression", td_note(conditions.temp_f, conditions.dewpoint_f)}, + {:refractivity, "Refractivity", refractivity_note(conditions.min_refractivity_gradient)}, {:sky, "Sky Cover", sky_note(conditions.sky_cover_pct)}, {:season, "Season", season_note(contact.qso_timestamp.month, band_config)}, {:wind, "Wind", wind_note(conditions.wind_speed_kts)}, {:rain, "Rain", rain_note(conditions.rain_rate_mmhr)}, - {:pwat, "PWAT", pwat_note(hrrr.pwat_mm, band_config)}, - {:pressure, "Pressure", pressure_note(hrrr.surface_pressure_mb)} + {:pwat, "PWAT", pwat_note(conditions.pwat_mm, band_config)}, + {:pressure, "Pressure", pressure_note(conditions.pressure_mb)} ] |> Enum.map(fn {key, name, note} -> score = result.factors[key]