Commit graph

8 commits

Author SHA1 Message Date
a283ad9c66
Refactor HRRR fetch to batch points per hour
QSO enrichment now groups all path points by HRRR hour and creates
one batch job per hour instead of one job per point. The batch job
downloads the GRIB2 data once and extracts all needed points from
the same binary. Legacy single-point jobs are still supported for
backward compatibility.
2026-03-30 17:21:47 -05:00
bb590a3b64
Invert HRRR failure logic: only retry transient network/server errors
Instead of listing every permanent failure string, only retry on known
transient errors (5xx, 429, network exceptions). Everything else —
GRIB decode errors, 404s, range request failures, index out of range —
is cancelled immediately.
2026-03-30 09:28:20 -05:00
8e1fd702f9
Reject HRRR downloads when server ignores Range header
S3 returns 200 with the full ~116MB file instead of 206 for some older
HRRR data. Treat this as a permanent failure instead of trying to parse
the entire file.
2026-03-30 09:26:58 -05:00
38bfd7caed
Add logging to HRRR fetch and parse pipeline 2026-03-30 09:17:37 -05:00
ead34990b0
Fix HRRR pipeline: single concurrency, cancel bad GRIB, rescue stuck jobs
- Reduce HRRR queue to 1 worker to avoid overwhelming upstream
- Add Oban Lifeline plugin to rescue orphaned executing jobs after 30m
- Treat GRIB2 decode errors (malformed section, missing sections, etc.)
  as permanent failures instead of retrying indefinitely
2026-03-30 09:16:07 -05:00
e2f5376355
Cancel HRRR jobs on permanent failures (404, outside_grid)
HTTP 404 means the HRRR data doesn't exist on NOAA S3 (pre-2014
dates, etc.) and will never succeed. Return {:cancel, reason}
instead of {:error, reason} so Oban stops retrying immediately.
2026-03-30 07:53:49 -05:00
4468fb3b61
Increase Oban retry attempts and backoff for external API workers
20 attempts with exponential backoff capped at 6 hours ensures
data is eventually fetched even during extended rate limiting or
API outages (~3 days of retrying before giving up).
2026-03-29 17:03:56 -05:00
8bd0989fbe
Add HRRR model profile fetching for QSOs
HRRR provides hourly 3km-resolution atmospheric profiles, filling the
temporal gaps (12-hourly soundings) and spatial gaps (only 9 sounding
stations) in our current weather data.

- Add hrrr_profiles table and hrrr_queued flag on QSOs
- HrrrClient fetches GRIB2 data via HTTP Range requests + wgrib2
- HrrrFetchWorker derives refractivity/ducting params via SoundingParams
- QsoWeatherEnqueueWorker now also enqueues HRRR jobs
- QSO show page displays HRRR section with collapsible profile
- Dockerfile builds wgrib2 from source for production
2026-03-29 15:54:22 -05:00