Graham McIntire
156b170a5c
Four independent wins from a perf audit of the era5_batch worker path: 1. Parallel CDS submits per job. fetch_single_level_month and fetch_pressure_level_month have no ordering dependency — each blocks 30+ min on CDS queue/download — so running them in two linked Tasks halves wall time for every job in the backlog. 2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`. A month-tile GRIB is 50–200 MB and was previously slurped into the BEAM heap and then written back out to a temp file before wgrib2 ran. New Era5Client.submit_and_download_to_file/3 skips both copies. Saves 100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under full parallelism. Partial files are removed on HTTP error. 3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the streamed temp file directly. The binary variant still exists for the single-point path and now delegates through the same helper. 4. Composite index (valid_time, lat, lon) on era5_profiles, matching the three-range scan used by Weather.find_nearest_era5 (contact detail + path profiles) and StatusLive.count_era5_done (status page EXISTS subquery). The unique index on (lat, lon, valid_time) couldn't serve it efficiently. Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips per month-tile (still well under Postgres' 65k-parameter cap). |
||
|---|---|---|
| .. | ||
| microwaveprop | ||
| microwaveprop_web | ||
| mix/tasks | ||
| microwaveprop.ex | ||
| microwaveprop_web.ex | ||