Prod has 0 era5_profiles rows after ~500 submit cycles: every CDS job
either gets marked "rejected" (likely cap mismatch) or runs for 13+h
without transitioning to successful, so the handle_both_done branch
has literally never fired and the ERA5Batch: stored log line has
never been emitted once. Tighten Era5PollWorker's @stuck_after_seconds
from 18h to 8h so dead rows recycle within a work shift instead of a
full day, and pause the era5 / era5_submit / era5_poll / era5_batch
queues in runtime.exs so existing rows stay put for inspection while
we figure out why CDS is rejecting everything. Flip paused: true back
once the root cause is identified.
A brief CDS network blip on 2026-04-15 returned HTTP 5xx from check_status
for multiple concurrently-polling workers. The old `{:error, _}` branches
consumed attempts, max_attempts=10 burned out in seconds, and 13 rows
were discarded with no cleanup — orphaned with no active poll worker,
including two rows that had both CDS legs done and were ready to decode.
Result: zero era5_profiles ever inserted despite 59 tile-months in flight.
Transient HTTP errors are now a `{:snooze, 60}` so a CDS outage pauses
polling instead of killing it. Terminal states (:failed, :rejected,
:not_found, :stuck) still drive the fail/resubmit paths.
Era5SubmitWorker was counting era5_cds_jobs rows 1:1 against the CDS
150-job ceiling, but each row holds TWO CDS job IDs (single-level +
pressure-level). Prod hit 74 rows = 148 in-flight jobs and got stuck
in a reject→resubmit spiral. The cap guard now multiplies row count
by 2 to match what CDS sees.
Era5PollWorker @stuck_after_seconds goes from 4h to 18h. CDS
single-level routinely sits at 'accepted' for 12+h under load while
pressure finishes in ~90 min; the 4h threshold was firing on normal
slow runs and feeding the same spiral.
PipelineStatus now returns `label` + `detail` separately so the map
chip can render "Updating propagation" on one line and the
sub-detail ("HRRR run" / "ASOS nudge" / "+3h" / "now") on a second
line below it. running_label/1 is renamed to running_detail/1 and
returns just the sub-part.
Observed in prod: after the CDS cap guard shipped, jobs started
landing but one leg per tile-month would routinely stay in 'accepted'
state for 16+ hours while the other completed in ~90 minutes. CDS's
per-user queue fairness apparently serializes competing leg requests
and the single-level ones got stranded behind higher-priority work.
Add an age-based stuck detector in Era5PollWorker: if a row has been
submitted for more than 4 hours and neither leg is fully done, treat
both legs as terminal, clean them up from CDS, and re-enqueue the
submit. Re-uses the resubmit_vanished path. 4h is ~8× the normal
completion time so transient queue variance doesn't trip it, but
tight enough that stuck jobs self-heal in a working session.
CDS returns status="rejected" (distinct from "failed") when a submit
is rate-limited past the 150-job per-user cap. The poll worker's
interpret_status_body treated this as an unknown status → generic
retryable error → 10 retries → discarded. 102 jobs hit this path
overnight.
Fix:
* Era5Client.check_status/1 now returns {:rejected, reason} (and
treats "dismissed" the same way — that's the status a manually
deleted job transitions to).
* Era5PollWorker handles :rejected exactly like :not_found: drop the
DB row, clean up the sibling CDS job, re-enqueue the submit, and
discard the current Oban job so it doesn't burn attempts retrying
a terminal state.
* Widened Era5SubmitWorker cap headroom from 10 → 30 (effective
ceiling 120 not 140). The race between concurrent workers all
passing the count check simultaneously meant we were reaching 141
in-flight against the 150 hard cap; 30 slots of headroom tolerates
a ~15-worker race before clipping CDS's ceiling.
Refactored handle_row/1 into handle_row + handle_non_both_done with a
leg_outcome/1 tag helper so the cross-product of leg statuses collapses
to a single pair match (credo complexity limit).
When CDS returns 404 for a poll check, the job is gone (manually
deleted, reaped, or never existed). Retrying will always 404, so treat
it as terminal: drop the era5_cds_jobs row, delete the sibling CDS job,
re-enqueue Era5SubmitWorker for the tile-month, and discard the Oban
poll job. Fixes 30 retryable poll jobs left behind by the interrupted
manual-cleanup script.
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
