Commit graph

6 commits

Author SHA1 Message Date
e79e65db43 Add ERA5 to backfill dashboard and enrichment pipeline
- ERA5 checkbox on /backfill page alongside HRRR/weather/terrain/IEMRE
- BackfillEnqueueWorker handles era5 type (targets hrrr_status=unavailable)
- ContactWeatherEnqueueWorker.build_era5_jobs/1 creates ERA5 fetch jobs
  for path points without existing ERA5 profiles
- Progress bar shows ERA5 candidate count (HRRR-unavailable contacts)
- Status panel shows ERA5 candidates and fetched profile count
2026-04-07 12:24:18 -05:00
e03b9fbc0e Add periodic enrichment backfill cron and terrain reconciliation
Runs BackfillEnqueueWorker every 30 minutes to pick up contacts with
pending/queued/failed enrichment status and enqueue missing jobs.
Before enqueueing, reconciles terrain contacts stuck in "queued" by
checking the terrain_profiles FK directly.
2026-04-07 07:36:57 -05:00
4ba7269e57 Prioritize pending contacts over queued in backfill
Queued contacts just need reconciliation (check if data exists),
while pending contacts need actual HRRR fetches. Process pending
first so real work happens before reconciliation.
2026-04-06 11:48:55 -05:00
e99e585b66 Add type filter to backfill enqueue
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
2026-04-06 11:42:26 -05:00
eeb92bcc19 Fix HRRR enrichment status stuck at queued/pending
enqueue_for_contact unconditionally marked contacts as :queued even
when no HRRR jobs were needed (profiles already exist). Workers never
updated contact status to :complete after storing profiles. Contacts
got stuck and were never re-processed.

- Mark :complete when no jobs generated (data already exists)
- Include :queued in backfill enrichable states for reconciliation
2026-04-06 10:24:56 -05:00
16883591b4
Database performance fixes and async backfill enqueue
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
  non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
2026-04-04 19:19:18 -05:00