prop/lib/microwaveprop/workers
Graham McIntire 32fdb2583d
Shrink PropagationGridWorker per-forecast-hour memory footprint
Two related optimizations in the propagation chain hot path. Both
land on the f01..f18 step that was previously OOM-killing prod
pods after the HRRR pressure-level footprint halved wasn't enough
to fit inside 4 Gi.

1. Skip the GridCache broadcast on forecast hours.

   /weather only ever renders the analysis hour (latest_grid_valid_time
   feeds the map). Building 92k rows, serializing them through PubSub,
   and rebuilding the {lat,lon}→row map on all three replicas was
   pure waste for f01..f18 — no consumer was reading that data. Only
   f00 now calls build_grid_cache_rows + broadcast_put. Point lookups
   for non-analysis hours still work through ProfilesFile on disk
   (weather_point_detail_from_profiles/3) exactly as before.

2. Fold replace_scores into a single streaming pass.

   The old path did `Enum.to_list/1` on the ~460k-entry score stream
   followed by `Enum.group_by/2`, holding two full copies of the grid
   before any file was written. A single `Enum.reduce/3` that folds
   each score into a per-band accumulator keeps only one copy and
   eliminates the group_by intermediate entirely. The public
   signature — an Enumerable in, {:ok, count} out — is unchanged.
2026-04-15 08:36:53 -05:00
..
admin_task_worker.ex Drop propagation_scores table, ScoresFile is the only store 2026-04-14 15:13:01 -05:00
backfill_enqueue_worker.ex Give up on stuck :queued backfill contacts after 3 days 2026-04-13 17:20:18 -05:00
canadian_sounding_fetch_worker.ex Ingest Canadian radiosondes via UWYO + plans for RDPS/HRDPS 2026-04-13 09:14:34 -05:00
contact_weather_enqueue_worker.ex Fix credo warnings: struct specs, length/1, and test patterns 2026-04-12 10:26:53 -05:00
era5_fetch_worker.ex Batch ERA5 fetches by month and 2° tile 2026-04-09 13:01:49 -05:00
era5_month_batch_worker.ex Split ERA5 backfill into submit/poll workers with persistent CDS state 2026-04-13 16:26:26 -05:00
era5_poll_worker.ex Fix ERA5 cap math + stuck threshold, split chip label onto two lines 2026-04-14 12:06:39 -05:00
era5_submit_worker.ex Fix ERA5 cap math + stuck threshold, split chip label onto two lines 2026-04-14 12:06:39 -05:00
hrrr_fetch_worker.ex Fix credo warnings: struct specs, length/1, and test patterns 2026-04-12 10:26:53 -05:00
hrrr_native_grid_worker.ex Fix all remaining credo --strict issues (0 issues) 2026-04-12 10:26:53 -05:00
iemre_fetch_worker.ex Fix credo warnings: struct specs, length/1, and test patterns 2026-04-12 10:26:53 -05:00
mrms_fetch_worker.ex Add MRMS rain mosaic, fix beacons crash, fix UTC clock flash 2026-04-12 14:49:20 -05:00
nexrad_worker.ex Fix NexradWorker UUID type: use generate() not bingenerate() for insert_all 2026-04-10 14:13:38 -05:00
propagation_grid_worker.ex Shrink PropagationGridWorker per-forecast-hour memory footprint 2026-04-15 08:36:53 -05:00
propagation_prune_worker.ex Decouple propagation_scores pruning from the compute worker 2026-04-09 12:30:10 -05:00
rtma_fetch_worker.ex Add ERA5 reanalysis and RTMA data sources 2026-04-07 12:04:16 -05:00
solar_index_worker.ex Database performance fixes and async backfill enqueue 2026-04-04 19:19:18 -05:00
terrain_profile_worker.ex Fix all remaining credo --strict issues (0 issues) 2026-04-12 10:26:53 -05:00
weather_fetch_worker.ex Fix all remaining credo --strict issues (0 issues) 2026-04-12 10:26:53 -05:00