graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
prop/docs/plans/2026-04-07-multi-source-atmospheric-data.md
Graham McIntire 7652e41971 Path-integrated HRRR scoring across pos1/mid/pos2 
Contact scoring now uses all HRRR profiles along the path instead of
just pos1. Aggregation strategy:
- Best along path for beneficial factors (refractivity, pressure)
- Worst along path for harmful factors (rain, wind)
- Average for neutral factors (temp, dewpoint, PWAT, BL depth)

Scorer.path_integrated_conditions/2 merges multiple profiles into a
single conditions map. Falls back gracefully to single-profile scoring
when only one profile is available.
2026-04-07 12:04:16 -05:00

4.3 KiB
Raw Blame History

Multi-Source Atmospheric Data Implementation Plan

Goal: Add path-integrated HRRR scoring, RTMA 15-minute data, and ERA5 reanalysis for pre-2014 contacts.

Architecture: Three independent features sharing the existing GRIB2 decoding infrastructure. Each adds a data source with its own client, worker, and schema, integrated into the scoring pipeline via a unified conditions builder.

Feature 1: Path-Integrated HRRR (data already fetched, scoring change only)

The HRRR fetch pipeline already retrieves profiles for pos1, midpoint, and pos2 via contact_path_points(). But hrrr_for_contact/1 only looks up pos1. Fix the scoring to use all path points.

Aggregation strategy:

  • Harmful factors (rain, humidity at 24+ GHz, wind): use WORST along path
  • Beneficial factors (refractivity, ducting, pressure): use BEST along path
  • Neutral factors (temperature, dewpoint, PWAT): use path AVERAGE

Files to modify

  • lib/microwaveprop/weather.ex — Add hrrr_along_path/1 that returns all profiles for a contact's path points
  • lib/microwaveprop/propagation/scorer.ex — Add path_integrated_conditions/2 that merges multiple HRRR profiles into one conditions map using best/worst/avg strategy
  • Tests

Feature 2: RTMA (Real-Time Mesoscale Analysis)

NOAA RTMA: 2.5 km resolution, 15-minute updates, covers CONUS. Available on AWS S3 at s3://noaa-rtma-pds/. Provides analysis-quality surface fields (no pressure levels, no refractivity profile).

Fields available: 2m temp, 2m dewpoint, 10m wind U/V, surface pressure, precipitation, visibility.

What RTMA adds: 4x temporal resolution over HRRR for surface conditions. Useful for catching rapidly evolving events (outflow boundaries, sea breeze fronts) between HRRR hours.

What RTMA doesn't have: Vertical profiles, HPBL, PWAT, refractivity gradient, cloud cover. These still come from HRRR.

Schema: rtma_observations

  • id, valid_time, lat, lon
  • temp_c, dewpoint_c, pressure_mb
  • wind_u_ms, wind_v_ms
  • visibility_m, precip_mm
  • Unique on (lat, lon, valid_time)

Files to create

  • lib/microwaveprop/weather/rtma_client.ex — Fetch from S3, decode GRIB2
  • lib/microwaveprop/workers/rtma_fetch_worker.ex — Oban worker
  • lib/microwaveprop/weather/rtma_observation.ex — Ecto schema
  • Migration for rtma_observations table

Integration

  • For real-time scoring: prefer RTMA surface fields when available (fresher than HRRR), fall back to HRRR
  • For contact enrichment: look up nearest RTMA observation by lat/lon/time, use for surface conditions while HRRR provides profile/refractivity data

Feature 3: ERA5 Reanalysis (historical backfill)

ECMWF ERA5: 0.25° resolution, hourly, global, 1940-present. Available via Copernicus CDS API (requires free API key). Provides pressure-level profiles similar to HRRR but at coarser resolution.

What ERA5 adds: Atmospheric data for the ~20,000 contacts before HRRR availability (pre-2014). Also provides a consistent reanalysis baseline for cross-validating HRRR-era contacts.

Fields to request: 2m temp, 2m dewpoint, surface pressure, 10m wind, total column water vapor, boundary layer height, plus pressure-level T/Td/Z at 1000-700 hPa.

Schema: era5_profiles

  • Same structure as hrrr_profiles for interoperability
  • id, valid_time, lat, lon
  • profile (array of pressure level data)
  • hpbl_m, pwat_mm, surface_temp_c, surface_dewpoint_c, surface_pressure_mb
  • surface_refractivity, min_refractivity_gradient, ducting_detected, duct_characteristics
  • source field to distinguish from HRRR
  • Unique on (lat, lon, valid_time)

Files to create

  • lib/microwaveprop/weather/era5_client.ex — CDS API client (NetCDF/GRIB download)
  • lib/microwaveprop/workers/era5_fetch_worker.ex — Oban worker
  • lib/microwaveprop/weather/era5_profile.ex — Ecto schema
  • Migration for era5_profiles table

Integration

  • Weather.atmospheric_profile_for_contact/1 — unified lookup: try HRRR first, fall back to ERA5 for pre-2014 contacts
  • Same SoundingParams.derive/1 for computing refractivity/ducting from ERA5 profiles

Implementation Order

  1. Path-integrated HRRR — smallest change, immediate value
  2. ERA5 — unlocks 20,000 contacts with no atmospheric data
  3. RTMA — improves real-time accuracy but lower priority (HRRR is already good)