Four independent wins from a perf audit of the era5_batch worker path:
1. Parallel CDS submits per job. fetch_single_level_month and
fetch_pressure_level_month have no ordering dependency — each blocks
30+ min on CDS queue/download — so running them in two linked Tasks
halves wall time for every job in the backlog.
2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
A month-tile GRIB is 50–200 MB and was previously slurped into the
BEAM heap and then written back out to a temp file before wgrib2 ran.
New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
full parallelism. Partial files are removed on HTTP error.
3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
streamed temp file directly. The binary variant still exists for the
single-point path and now delegates through the same helper.
4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
three-range scan used by Weather.find_nearest_era5 (contact detail +
path profiles) and StatusLive.count_era5_done (status page EXISTS
subquery). The unique index on (lat, lon, valid_time) couldn't serve
it efficiently.
Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
Graham McIntire
156b170a5c