Graham McIntire
8637253fda
Per-point ERA5 fetches were tragically slow because every point-hour triggered its own asynchronous CDS job (submit → poll → assemble → download). For backfill this meant thousands of independent jobs queued against Copernicus. The new path groups requests by calendar month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k profiles at once, and Oban uniqueness on (year, month, tile_lat, tile_lon) collapses every duplicate enqueue. - Era5BatchClient builds the monthly CDS requests, extracts every (lat, lon, hour) from the GRIB2 blob with wgrib2, derives refractivity params, and bulk-inserts in 2k-row chunks with on_conflict: :nothing. fetch_month_into_db/1 short-circuits when the month-tile already has any cached profile. - Era5MonthBatchWorker runs the batch on the :era5 queue with a generous backoff (10m → 1d) and the uniqueness key above. - Era5FetchWorker is now a thin router: cache hit → :ok, cache miss → enqueue the month-batch for the point's tile-month and return. No more per-point CDS calls. - Wgrib2 grows extract_grid_messages/3 which preserves per-message datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a single GRIB2 file carrying a whole month decodes correctly. - The era5_backfill mix task enqueues month-tile batches directly.
64 lines
2 KiB
Elixir
64 lines
2 KiB
Elixir
defmodule Microwaveprop.Workers.Era5MonthBatchWorker do
|
||
@moduledoc """
|
||
Fetches one month of ERA5 data for a 2° × 2° tile in a single CDS request
|
||
and bulk-inserts all resulting hourly profiles into `era5_profiles`.
|
||
|
||
This exists because fetching ERA5 per point-hour is tragically slow: each
|
||
CDS request goes through a full submit → poll → assemble → download cycle
|
||
whose overhead dwarfs the data transfer. Grouping by month+tile turns
|
||
thousands of CDS jobs into a handful, and once a tile-month is cached in
|
||
the DB every downstream query for a point inside that window is a local
|
||
lookup.
|
||
|
||
Uniqueness on the full arg set means Oban collapses duplicate enqueues
|
||
for the same tile-month — multiple `Era5FetchWorker` requests for the
|
||
same region collapse into a single batch.
|
||
"""
|
||
use Oban.Worker,
|
||
queue: :era5,
|
||
max_attempts: 3,
|
||
unique: [
|
||
period: :infinity,
|
||
states: [:available, :scheduled, :executing, :retryable],
|
||
keys: [:year, :month, :tile_lat, :tile_lon]
|
||
]
|
||
|
||
alias Microwaveprop.Weather.Era5BatchClient
|
||
|
||
require Logger
|
||
|
||
@impl Oban.Worker
|
||
def backoff(%Oban.Job{attempt: attempt}) do
|
||
# A full month fetch can sit in the CDS queue for ages; back off generously.
|
||
min(600 * Integer.pow(2, attempt - 1), _one_day = 86_400)
|
||
end
|
||
|
||
@impl Oban.Worker
|
||
def perform(%Oban.Job{args: args}) do
|
||
params = %{
|
||
year: fetch_int!(args, "year"),
|
||
month: fetch_int!(args, "month"),
|
||
tile_lat: fetch_int!(args, "tile_lat"),
|
||
tile_lon: fetch_int!(args, "tile_lon")
|
||
}
|
||
|
||
case Era5BatchClient.fetch_month_into_db(params) do
|
||
{:ok, _count} ->
|
||
:ok
|
||
|
||
{:error, reason} ->
|
||
Logger.warning(
|
||
"ERA5Batch: #{params.year}-#{params.month} tile #{params.tile_lat},#{params.tile_lon} failed: #{inspect(reason)}"
|
||
)
|
||
|
||
{:error, reason}
|
||
end
|
||
end
|
||
|
||
defp fetch_int!(args, key) do
|
||
case Map.fetch!(args, key) do
|
||
n when is_integer(n) -> n
|
||
n when is_binary(n) -> String.to_integer(n)
|
||
end
|
||
end
|
||
end
|