Graham McIntire
7652e41971
Contact scoring now uses all HRRR profiles along the path instead of just pos1. Aggregation strategy: - Best along path for beneficial factors (refractivity, pressure) - Worst along path for harmful factors (rain, wind) - Average for neutral factors (temp, dewpoint, PWAT, BL depth) Scorer.path_integrated_conditions/2 merges multiple profiles into a single conditions map. Falls back gracefully to single-profile scoring when only one profile is available.
88 lines
4.3 KiB
Markdown
88 lines
4.3 KiB
Markdown
# Multi-Source Atmospheric Data Implementation Plan
|
|
|
|
**Goal:** Add path-integrated HRRR scoring, RTMA 15-minute data, and ERA5 reanalysis for pre-2014 contacts.
|
|
|
|
**Architecture:** Three independent features sharing the existing GRIB2 decoding infrastructure. Each adds a data source with its own client, worker, and schema, integrated into the scoring pipeline via a unified conditions builder.
|
|
|
|
---
|
|
|
|
## Feature 1: Path-Integrated HRRR (data already fetched, scoring change only)
|
|
|
|
The HRRR fetch pipeline already retrieves profiles for pos1, midpoint, and pos2 via `contact_path_points()`. But `hrrr_for_contact/1` only looks up pos1. Fix the scoring to use all path points.
|
|
|
|
**Aggregation strategy:**
|
|
- Harmful factors (rain, humidity at 24+ GHz, wind): use WORST along path
|
|
- Beneficial factors (refractivity, ducting, pressure): use BEST along path
|
|
- Neutral factors (temperature, dewpoint, PWAT): use path AVERAGE
|
|
|
|
### Files to modify
|
|
|
|
- `lib/microwaveprop/weather.ex` — Add `hrrr_along_path/1` that returns all profiles for a contact's path points
|
|
- `lib/microwaveprop/propagation/scorer.ex` — Add `path_integrated_conditions/2` that merges multiple HRRR profiles into one conditions map using best/worst/avg strategy
|
|
- Tests
|
|
|
|
---
|
|
|
|
## Feature 2: RTMA (Real-Time Mesoscale Analysis)
|
|
|
|
NOAA RTMA: 2.5 km resolution, 15-minute updates, covers CONUS. Available on AWS S3 at `s3://noaa-rtma-pds/`. Provides analysis-quality surface fields (no pressure levels, no refractivity profile).
|
|
|
|
**Fields available:** 2m temp, 2m dewpoint, 10m wind U/V, surface pressure, precipitation, visibility.
|
|
|
|
**What RTMA adds:** 4x temporal resolution over HRRR for surface conditions. Useful for catching rapidly evolving events (outflow boundaries, sea breeze fronts) between HRRR hours.
|
|
|
|
**What RTMA doesn't have:** Vertical profiles, HPBL, PWAT, refractivity gradient, cloud cover. These still come from HRRR.
|
|
|
|
### Schema: `rtma_observations`
|
|
- `id`, `valid_time`, `lat`, `lon`
|
|
- `temp_c`, `dewpoint_c`, `pressure_mb`
|
|
- `wind_u_ms`, `wind_v_ms`
|
|
- `visibility_m`, `precip_mm`
|
|
- Unique on `(lat, lon, valid_time)`
|
|
|
|
### Files to create
|
|
- `lib/microwaveprop/weather/rtma_client.ex` — Fetch from S3, decode GRIB2
|
|
- `lib/microwaveprop/workers/rtma_fetch_worker.ex` — Oban worker
|
|
- `lib/microwaveprop/weather/rtma_observation.ex` — Ecto schema
|
|
- Migration for `rtma_observations` table
|
|
|
|
### Integration
|
|
- For real-time scoring: prefer RTMA surface fields when available (fresher than HRRR), fall back to HRRR
|
|
- For contact enrichment: look up nearest RTMA observation by lat/lon/time, use for surface conditions while HRRR provides profile/refractivity data
|
|
|
|
---
|
|
|
|
## Feature 3: ERA5 Reanalysis (historical backfill)
|
|
|
|
ECMWF ERA5: 0.25° resolution, hourly, global, 1940-present. Available via Copernicus CDS API (requires free API key). Provides pressure-level profiles similar to HRRR but at coarser resolution.
|
|
|
|
**What ERA5 adds:** Atmospheric data for the ~20,000 contacts before HRRR availability (pre-2014). Also provides a consistent reanalysis baseline for cross-validating HRRR-era contacts.
|
|
|
|
**Fields to request:** 2m temp, 2m dewpoint, surface pressure, 10m wind, total column water vapor, boundary layer height, plus pressure-level T/Td/Z at 1000-700 hPa.
|
|
|
|
### Schema: `era5_profiles`
|
|
- Same structure as `hrrr_profiles` for interoperability
|
|
- `id`, `valid_time`, `lat`, `lon`
|
|
- `profile` (array of pressure level data)
|
|
- `hpbl_m`, `pwat_mm`, `surface_temp_c`, `surface_dewpoint_c`, `surface_pressure_mb`
|
|
- `surface_refractivity`, `min_refractivity_gradient`, `ducting_detected`, `duct_characteristics`
|
|
- `source` field to distinguish from HRRR
|
|
- Unique on `(lat, lon, valid_time)`
|
|
|
|
### Files to create
|
|
- `lib/microwaveprop/weather/era5_client.ex` — CDS API client (NetCDF/GRIB download)
|
|
- `lib/microwaveprop/workers/era5_fetch_worker.ex` — Oban worker
|
|
- `lib/microwaveprop/weather/era5_profile.ex` — Ecto schema
|
|
- Migration for `era5_profiles` table
|
|
|
|
### Integration
|
|
- `Weather.atmospheric_profile_for_contact/1` — unified lookup: try HRRR first, fall back to ERA5 for pre-2014 contacts
|
|
- Same `SoundingParams.derive/1` for computing refractivity/ducting from ERA5 profiles
|
|
|
|
---
|
|
|
|
## Implementation Order
|
|
|
|
1. **Path-integrated HRRR** — smallest change, immediate value
|
|
2. **ERA5** — unlocks 20,000 contacts with no atmospheric data
|
|
3. **RTMA** — improves real-time accuracy but lower priority (HRRR is already good)
|