Graham McIntire
fc3eb58910
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k surface obs, new hrrr_native_profiles table) drive these changes: * Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus removed" conclusion was an artifact of the prior matching strategy; with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins. * Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in those bands stop being silently dropped. Coefficients extrapolate ITU-R P.676/P.838 trends from the existing 134/241 GHz entries. * Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump the full failure body when CDS omits the message field — the prior "ERA5 job failed: nil" was masking real error reasons in oban_jobs. * Add scripts/recalibrate_algo.py so this analysis can be re-run any time new data lands without manual SQL. Reads PROP_PROD_DB_URL from .envrc, drops a Markdown report into docs/algo-reports/. * Append a dated Part 2c to algo.md documenting the corpus expansion, the honest accounting of historical HRRR coverage (only ~1,020 of 58k contacts are precision-matchable), and the empty era5/rtma/climatology tables. Update the gaseous-absorption and rain-attenuation tables to include the new bands. Test suite: 1,335 tests, 0 failures.
72 lines
2.4 KiB
Elixir
72 lines
2.4 KiB
Elixir
defmodule Microwaveprop.Workers.Era5MonthBatchWorker do
|
||
@moduledoc """
|
||
Fetches one month of ERA5 data for a 2° × 2° tile in a single CDS request
|
||
and bulk-inserts all resulting hourly profiles into `era5_profiles`.
|
||
|
||
This exists because fetching ERA5 per point-hour is tragically slow: each
|
||
CDS request goes through a full submit → poll → assemble → download cycle
|
||
whose overhead dwarfs the data transfer. Grouping by month+tile turns
|
||
thousands of CDS jobs into a handful, and once a tile-month is cached in
|
||
the DB every downstream query for a point inside that window is a local
|
||
lookup.
|
||
|
||
Uniqueness on the full arg set means Oban collapses duplicate enqueues
|
||
for the same tile-month — multiple `Era5FetchWorker` requests for the
|
||
same region collapse into a single batch.
|
||
|
||
Runs on its own `:era5_batch` queue so its slow CDS poll cycles don't
|
||
starve the fast `Era5FetchWorker` router that also lives in the ERA5
|
||
namespace.
|
||
"""
|
||
# The :era5_batch queue is rate-limited to 10/hour so a single failed run
|
||
# eats a meaningful chunk of throughput. Retrying 5×, combined with the
|
||
# generous backoff below, gives CDS up to a day to come back without
|
||
# discarding the tile and silently dropping the historical contact.
|
||
use Oban.Worker,
|
||
queue: :era5_batch,
|
||
max_attempts: 5,
|
||
unique: [
|
||
period: :infinity,
|
||
states: [:available, :scheduled, :executing, :retryable],
|
||
keys: [:year, :month, :tile_lat, :tile_lon]
|
||
]
|
||
|
||
alias Microwaveprop.Weather.Era5BatchClient
|
||
|
||
require Logger
|
||
|
||
@impl Oban.Worker
|
||
def backoff(%Oban.Job{attempt: attempt}) do
|
||
# A full month fetch can sit in the CDS queue for ages; back off generously.
|
||
min(600 * Integer.pow(2, attempt - 1), _one_day = 86_400)
|
||
end
|
||
|
||
@impl Oban.Worker
|
||
def perform(%Oban.Job{args: args}) do
|
||
params = %{
|
||
year: fetch_int!(args, "year"),
|
||
month: fetch_int!(args, "month"),
|
||
tile_lat: fetch_int!(args, "tile_lat"),
|
||
tile_lon: fetch_int!(args, "tile_lon")
|
||
}
|
||
|
||
case Era5BatchClient.fetch_month_into_db(params) do
|
||
{:ok, _count} ->
|
||
:ok
|
||
|
||
{:error, reason} ->
|
||
Logger.warning(
|
||
"ERA5Batch: #{params.year}-#{params.month} tile #{params.tile_lat},#{params.tile_lon} failed: #{inspect(reason)}"
|
||
)
|
||
|
||
{:error, reason}
|
||
end
|
||
end
|
||
|
||
defp fetch_int!(args, key) do
|
||
case Map.fetch!(args, key) do
|
||
n when is_integer(n) -> n
|
||
n when is_binary(n) -> String.to_integer(n)
|
||
end
|
||
end
|
||
end
|