prop/lib/microwaveprop/workers
Graham McIntire 900685aa06 Phase 1 tasks 1.1-1.5: HRRR native hybrid-sigma ingestion
- Spike docs at docs/research/hrrr_native_levels.md confirming files
  are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
  SPFH needed for Phase 2 turbulence features. Architectural finding:
  per-point on-demand fetching is impractical (~530 MB/file), so
  the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
  scalars and placeholder columns for Phase 2/4 derived fields.
  Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
  that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
  reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
  unique at :infinity, pulls distinct (lat, lon) points from contacts
  in the ±30 min window, downloads the native grib2, extracts per
  point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
  contact count.

Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
2026-04-09 16:23:51 -05:00
..
asos_adjustment_worker.ex Database performance fixes and async backfill enqueue 2026-04-04 19:19:18 -05:00
backfill_enqueue_worker.ex Add ERA5 to backfill dashboard and enrichment pipeline 2026-04-07 12:24:18 -05:00
contact_weather_enqueue_worker.ex Dedupe ERA5 fetch jobs by (lat, lon, valid_time) 2026-04-08 15:52:53 -05:00
era5_fetch_worker.ex Batch ERA5 fetches by month and 2° tile 2026-04-09 13:01:49 -05:00
era5_month_batch_worker.ex Move Era5MonthBatchWorker to its own queue 2026-04-09 14:12:31 -05:00
hrrr_fetch_worker.ex Show dead tuple counts on backfill dashboard, fix table visibility during vacuum 2026-04-04 10:45:06 -05:00
hrrr_native_grid_worker.ex Phase 1 tasks 1.1-1.5: HRRR native hybrid-sigma ingestion 2026-04-09 16:23:51 -05:00
iemre_fetch_worker.ex Store stub records on empty weather fetches to prevent infinite backfill retries 2026-04-06 16:10:39 -05:00
propagation_grid_worker.ex Decouple propagation_scores pruning from the compute worker 2026-04-09 12:30:10 -05:00
propagation_prune_worker.ex Decouple propagation_scores pruning from the compute worker 2026-04-09 12:30:10 -05:00
rtma_fetch_worker.ex Add ERA5 reanalysis and RTMA data sources 2026-04-07 12:04:16 -05:00
solar_index_worker.ex Database performance fixes and async backfill enqueue 2026-04-04 19:19:18 -05:00
terrain_profile_worker.ex Mark terrain status complete when profile already exists 2026-04-06 16:54:29 -05:00
weather_fetch_worker.ex Store stub records on empty weather fetches to prevent infinite backfill retries 2026-04-06 16:10:39 -05:00