Observed in prod: after the CDS cap guard shipped, jobs started
landing but one leg per tile-month would routinely stay in 'accepted'
state for 16+ hours while the other completed in ~90 minutes. CDS's
per-user queue fairness apparently serializes competing leg requests
and the single-level ones got stranded behind higher-priority work.
Add an age-based stuck detector in Era5PollWorker: if a row has been
submitted for more than 4 hours and neither leg is fully done, treat
both legs as terminal, clean them up from CDS, and re-enqueue the
submit. Re-uses the resubmit_vanished path. 4h is ~8× the normal
completion time so transient queue variance doesn't trip it, but
tight enough that stuck jobs self-heal in a working session.
CDS returns status="rejected" (distinct from "failed") when a submit
is rate-limited past the 150-job per-user cap. The poll worker's
interpret_status_body treated this as an unknown status → generic
retryable error → 10 retries → discarded. 102 jobs hit this path
overnight.
Fix:
* Era5Client.check_status/1 now returns {:rejected, reason} (and
treats "dismissed" the same way — that's the status a manually
deleted job transitions to).
* Era5PollWorker handles :rejected exactly like :not_found: drop the
DB row, clean up the sibling CDS job, re-enqueue the submit, and
discard the current Oban job so it doesn't burn attempts retrying
a terminal state.
* Widened Era5SubmitWorker cap headroom from 10 → 30 (effective
ceiling 120 not 140). The race between concurrent workers all
passing the count check simultaneously meant we were reaching 141
in-flight against the 150 hard cap; 30 slots of headroom tolerates
a ~15-worker race before clipping CDS's ceiling.
Refactored handle_row/1 into handle_row + handle_non_both_done with a
leg_outcome/1 tag helper so the cross-product of leg statuses collapses
to a single pair match (credo complexity limit).
Replaces the two-phase (algorithm-scores pretrain + QSO-distance
finetune) pipeline with a single unbiased phase that samples
hrrr_profiles stratified uniformly by calendar month and trains on the
physical algorithm score from Scorer.composite_score/2.
Previous Phase 2 target was 'within-band distance percentile' computed
from the contacts table. Because non-contest months have almost no
QSOs, the model learned 'these conditions → QSO happened → good' and
implicitly 'no QSO → bad', so it systematically under-predicted
propagation during quiet months (Feb, Mar, Nov). The new pipeline
decouples the target from QSO activity entirely.
Stratification uses ROW_NUMBER() OVER (PARTITION BY month ORDER BY
random()) so every month contributes the same number of profile rows
regardless of contest-driven density. Each profile is exploded into
one training row per band.
Trained model: val RMSE=1.66 pts, R²=0.9744 on 840k rows (10k
profiles/month × 7 bands, 50 epochs). Monthly sanity check at fixed
weather conditions (10 GHz, 15°C/10°C) now shows Jan=Feb=Mar=68 — the
model no longer penalizes quiet months.
CDS enforces a per-user cap of 150 queued jobs. Each successful submit
adds 2 rows to era5_cds_jobs (single-level + pressure-level), so when
the current in-flight count + 2 would exceed (150 - 10 headroom), the
worker snoozes for 5 minutes instead of POSTing to CDS. Stops 'Number
of queued requests is limited to 150' rejects from burning Oban
attempts.
Contacts whose weather/hrrr/terrain/iemre status has been :queued for
more than 3 days (144 cron cycles) are flipped to :unavailable. The
set_enrichment_status!/3 helper is a no-op when the status is unchanged,
so a stuck row retains a stale updated_at — which is exactly the signal
we need for 'the cron has already tried and the data doesnt exist
upstream' (dead ASOS station, pre-2014 HRRR, out-of-grid IEMRE). Clears
~1,492 pre-existing stuck contacts and stops them from churning the
backfill queue.
When CDS returns 404 for a poll check, the job is gone (manually
deleted, reaped, or never existed). Retrying will always 404, so treat
it as terminal: drop the era5_cds_jobs row, delete the sibling CDS job,
re-enqueue Era5SubmitWorker for the tile-month, and discard the Oban
poll job. Fixes 30 retryable poll jobs left behind by the interrupted
manual-cleanup script.
Ports the same two overlay toggles from /map to /weather:
- Maidenhead grid squares via the shared updateGridOverlay helper,
redrawn on moveend when visible.
- ECCC GeoMet WMS radar (CONUS + Canada dBZ), lazy-created on first
toggle with a 6-minute tile refresh timer.
Both toggles live in the desktop sidebar below the layer description
and in the mobile pulldown panel. Hook cleanup clears the radar
refresh timer on destroyed().
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Four independent wins from a perf audit of the era5_batch worker path:
1. Parallel CDS submits per job. fetch_single_level_month and
fetch_pressure_level_month have no ordering dependency — each blocks
30+ min on CDS queue/download — so running them in two linked Tasks
halves wall time for every job in the backlog.
2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
A month-tile GRIB is 50–200 MB and was previously slurped into the
BEAM heap and then written back out to a temp file before wgrib2 ran.
New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
full parallelism. Partial files are removed on HTTP error.
3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
streamed temp file directly. The binary variant still exists for the
single-point path and now delegates through the same helper.
4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
three-range scan used by Weather.find_nearest_era5 (contact detail +
path profiles) and StatusLive.count_era5_done (status page EXISTS
subquery). The unique index on (lat, lon, valid_time) couldn't serve
it efficiently.
Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
Raise local_limit 2→4 (12 concurrent cluster-wide) and rate_limit 10→30
per hour. Each job makes 2 serial CDS requests, so active workers map 1:1
to CDS slots; 12 sits safely under CDS-Beta's ~16 per-user ceiling while
the rate gate stays above steady-state so it stops being the bottleneck.
The Pro Smart engine's global_limit tracker is stored inside
oban_producers.meta and is not reliably released when a pod exits
between job start and completion. After today's rollout two pods died
holding tracked slots for running era5_batch jobs; the survivors refused
to start any new ones because the cluster-wide tracker was stuck at
4/4 — with zero actual jobs executing — and the whole backfill froze
for 2+ hours.
Keeping rate_limit (10/hr) is enough to stay polite to Copernicus CDS.
local_limit stays at 2 per pod, so effective cluster concurrency just
scales with replica count instead of being capped by a global value
whose bookkeeping can't survive a restart.
Contact detail page now renders an atmospheric profile section backed by
HRRR when available and ERA5 (reanalysis) otherwise, labeled with the
data source. When both sources are missing and HRRR is known-unavailable
(pre-2014 contacts), the Era5FetchWorker is enqueued so the month-tile
backfill gets triggered from a user visit.
Status page ERA5 progress previously computed "complete" as
(total_contacts - hrrr_unavailable_contacts), which treated every contact
with any HRRR status as ERA5-complete — inflating the bar to 76% while
era5_profiles had 0 rows. Replaced with a per-candidate EXISTS query
against era5_profiles using the same 0.15°/30-min tolerance
Weather.find_nearest_era5/3 uses at lookup time.
CDS rejects `reanalysis-era5-pressure-levels` requests that include
dewpoint_temperature because it's only a single-level 2m variable; no
pressure-level dewpoint exists in ERA5. The failure mode is a 2–3 second
`{"status": "failed"}` response with no message field, which is why
prod was burning through Era5MonthBatchWorker attempts without anything
landing in era5_profiles.
Switch the pressure-level moisture variable to specific_humidity (SPFH),
update the wgrib2 extraction regex, and convert SPFH → dewpoint °C via
Weather.ThetaE.dewpoint_from_spfh/2 in the profile builder. Surface
dewpoint keeps using the single-level 2m dewpoint_temperature, which is
valid and already working.
Three new analysis passes in scripts/recalibrate_algo.py that match the
scoring factors landed in commit 86364f4:
* NEXRAD composite reflectivity vs distance per band — checks that the
Marshall-Palmer rain rate in Scorer.dbz_to_rain_rate_mmhr/1 moves the
needle in the expected direction at 24+ GHz.
* hrrr_native_profiles.best_duct_band_ghz vs distance per band —
validates the 1.15× boost in Scorer.score_refractivity/4. Current
corpus shows essentially zero contacts with a native duct supporting
≥5 GHz at the target band, consistent with the 0.12% rate in Part 2c;
the boost is physics-correct but almost never fires today.
* Commercial-link rx_power degradation vs contemporaneous DFW contacts —
pearson + binned distance. Empty today because commercial_samples only
covers Mar 30–Apr 13 and contests don't overlap. Scaffolding is in
place so the moment contest-season overlap exists, the signal shows.
Report rebuilt at docs/algo-reports/2026-04-13-recalibration.md.
Three signal sources we already collect but weren't using:
* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
as max of HRRR-derived and NEXRAD-derived rate so fast convective
cells between HRRR hourly analyses can still trigger the rain penalty.
Only active on f00 — forecast hours can't see future radar. New
Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
hail-safe ceiling.
* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
applies a 1.15× boost when the cell's native-resolution duct supports
the target band's frequency. HRRR pressure-level gradients
systematically under-read thin trapping layers the native profile can
resolve. Sub-band ducts do NOT boost — they're evidence that the
gradient we have is all there is at the target frequency.
* Commercial LOS link rx_power fading → inverse tropo sensor.
Commercial.link_degradation_at/3 computes the average 7-day-baseline
vs current delta across enabled links within 75 km, ignoring links
where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
to the composite score for 3+ dB of fading. ~150 km radius around
DFW is the only zone this helps today, but it's the first *measured*
signal in the algorithm vs the model-derived proxies.
Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.
1,359 tests, 0 failures.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
IEM's RAOB endpoint stopped keeping Canadian stations current — most
stalled around Sep 2024 — which leaves a gap in refractivity/ducting
calibration over all the CW*/CY* stations already in weather_stations.
MSC's own tephi CSV is fixed-width and rendering-focused; the WMO TEMP
bulletins would need a full decoder. University of Wyoming serves the
same stations in a clean space-delimited format, is current, and
accepts the 3-letter station code (drop the leading C from Canadian
ICAO). Its output shape matches IemClient.parse_raob_json/1 exactly,
so it drops straight into Weather.upsert_sounding/2.
- New UwyoSoundingClient with URL builder, Req-based fetch_sounding,
and a fixed-width parse_sounding_html that extracts PRES/HGHT/TEMP/
DWPT/DRCT/SKNT from the <PRE> block and DateTime from the <H2> header.
- New CanadianSoundingFetchWorker: Oban batch worker that finds all
sounding stations whose code starts with C, picks the most recently
publishable 00Z or 12Z slot (90 min publish delay), calls UWYO per
station, and upserts with SoundingParams-derived refractivity/
ducting fields.
- Oban cron at 01:30Z and 13:30Z (dev + prod) to drive the worker.
- Req.Test plug wiring in config/test.exs.
- Captured Goose Bay fixture as test/support/fixtures/uwyo_sounding_yyr.html.
Also drops two implementation plans under docs/plans/:
- 2026-04-13-rdps-vertical-profiles.md: ~2 day plan for RDPS 10 km
Canadian ducting outside HRRR's Lambert footprint.
- 2026-04-13-hrdps-canadian-prop-grid.md: ~5 day plan for the full
HRDPS 2.5 km Canadian prop grid. Explicitly recommends doing RDPS
first.
TDD throughout, 19 new tests, 1318/1318 passing, credo strict clean.
