The scoring+upsert phase was ~4m40s per forecast hour and dominated
wall time. Three stacked optimizations attack it from different
angles.
replace_scores/2 is a new hot-path writer that does DELETE WHERE
valid_time = $1 followed by a plain insert_all (no ON CONFLICT
resolution). The chain worker rewrites the full (valid_time, all
bands) slice every forecast hour, so conflict detection was pure
waste. AsosAdjustmentWorker still uses upsert_scores because it
only rewrites the subset of cells near a station.
factors is now nullable. Forecast hours f01-f18 pass factors: nil
so the JSONB encode + toast write is skipped entirely — roughly
halves the data volume per run. point_detail/4 coalesces nil to
an empty map so the JS popup renders without a TypeError, and
scorer_diff only pulls the most recent valid_time that still has
factors (the f00 row).
propagation_scores is now UNLOGGED, so inserts bypass WAL entirely.
Durability tradeoff: an unclean shutdown truncates the table, but
PropagationGridWorker rebuilds it from HRRR every 3h so a lost
table is re-populated within one cron cycle.
Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a
speculative plan for a flat-file / DuckDB rewrite with explicit
trigger conditions for when to pick it up (partitioning deferred
too; revisit only if these three don't solve it).
A full f00-f18 sweep takes ~170 min of wall time, longer than the
~90 min gap between deploys on this cluster. Over the last 7 days
every run was killed mid-sweep and zero runs completed. The
propagation_scores table only ever held the scraps from partial
runs that landed before the pod died, which is why the map looked
like it was showing "the current hour" (or nothing).
The worker now processes exactly one forecast hour per perform/1
(~8-10 min) and enqueues the next hour as a fresh Oban job. A cron
fire with empty args seeds the chain at f00; subsequent runs carry
run_time + forecast_hour args. At f18 the chain stops and the
pruner runs. Each step broadcasts propagation:updated immediately
so new hours appear on the map as they land.
Wall time per perform drops from ~170 min to ~10 min, so Lifeline's
2h rescue window is no longer a factor and a pod restart loses at
most one forecast hour instead of the whole sweep. Retries and
max_attempts now describe a single fh, not the whole chain.
Also: bump the :propagation queue from 1 to 2 slots so
PropagationPruneWorker can run alongside the chain job, drop the
worker timeout/1 from 90 min to 20 min to match single-fh runs,
and pull Lifeline rescue_after from 120 min to 45 min to keep the
safety net above the step timeout.
Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the
pipeline chip in both the mobile and desktop sidebars so the user
can tell which valid_time the map is showing when the bottom
timeline isn't visible.
Era5SubmitWorker was counting era5_cds_jobs rows 1:1 against the CDS
150-job ceiling, but each row holds TWO CDS job IDs (single-level +
pressure-level). Prod hit 74 rows = 148 in-flight jobs and got stuck
in a reject→resubmit spiral. The cap guard now multiplies row count
by 2 to match what CDS sees.
Era5PollWorker @stuck_after_seconds goes from 4h to 18h. CDS
single-level routinely sits at 'accepted' for 12+h under load while
pressure finishes in ~90 min; the 4h threshold was firing on normal
slow runs and feeding the same spiral.
PipelineStatus now returns `label` + `detail` separately so the map
chip can render "Updating propagation" on one line and the
sub-detail ("HRRR run" / "ASOS nudge" / "+3h" / "now") on a second
line below it. running_label/1 is renamed to running_detail/1 and
returns just the sub-part.
PropagationGridWorker now broadcasts a {:propagation_pipeline_progress,
%{forecast_hour:, valid_time:}} message on the propagation:pipeline
PubSub topic at the start of each process_forecast_hour/4 call.
MapLive subscribes on mount, stashes the last message in the new
:pipeline_progress assign, and clears it on the refresh tick whenever
PipelineStatus flips out of :running.
The chip component consults a new PipelineStatus.running_label/1
helper that formats the payload as "Updating propagation · now" for
f00 and "Updating propagation · +Nh" for f01-f18, falling back to
the base running label until the first progress message arrives.
Covered by unit tests on the formatter and an integration test in
MapLiveTest that seeds an executing grid job, broadcasts progress,
and asserts the rendered chip.
The linker error from the previous attempt was actual symbol
references, not a flag-combination oddity:
undefined reference to `g2c_enc_jpeg2000'
undefined reference to `g2c_dec_jpeg2000'
Those live in g2c's "new" file-based API, built only when
BUILD_G2C=ON. In v1.9.0 the option defaulted OFF and was marked
"experimental until 2.0.0 release", so I had explicitly disabled
it. In v2.1.0 (released 2025-01-13, before wgrib2 v3.6.0 on
2025-02-19) it defaults ON and the API is stable. Bump
G2C_VERSION to 2.1.0 and drop the BUILD_G2C override.
With the new-API symbols now exported, wgrib2 can be compiled
with every decoder we care about in one shot:
-DUSE_PNG=ON -DUSE_OPENJPEG=ON -DUSE_AEC=ON
`wgrib2 -config` in the final runtime image now reports:
Supported decoding: simple, complex, rle, ieee, png, jpeg2000,
CCSDS AEC
Verified end-to-end by building the full image with podman and
running wgrib2 -config inside it. MrmsFetchWorker's packing
type 41 (PNG) messages will decode once the rolling deploy lands.
Last attempt (-DUSE_PNG=ON -DUSE_AEC=ON on wgrib2) reached 100 %
linking libwgrib2.so and then died with "make: Error 2" on the
parallel wgrib2_exe link. wgrib2's own CI workflow
(.github/workflows/Linux_options.yml) never tests USE_PNG alone —
its matrix uses -DUSE_OPENJPEG=ON, and its NCEPLIBS-g2c is built
with -DUSE_OpenJPEG=ON -DUSE_Jasper=OFF. Copying that known-good
combination:
* g2c now builds with USE_PNG + USE_AEC + USE_OpenJPEG (Jasper
still disabled — libjasper is out of Debian anyway).
* wgrib2 flips to -DUSE_OPENJPEG=ON -DUSE_AEC=ON.
* libopenjp2-7-dev added to the builder apt list, libopenjp2-7
added to the runtime apt list so libg2c.so + wgrib2 can
resolve OpenJPEG symbols at exec time.
wgrib2 delegates actual GRIB2 decompression to g2c, which is
built with PNG support, so DRT 5.41 (MRMS PrecipRate) still
decodes through this link — wgrib2's own USE_PNG knob isn't
required for decoding, only for encoding we don't do.
The previous attempt to flip `-DUSE_PNG=ON -DUSE_AEC=ON` on wgrib2
broke the image build because wgrib2 v3.6.0's CMakeLists.txt
line 94 forces `find_package(g2c 1.9.0 REQUIRED)` whenever PNG or
JPEG2000 is enabled, and NCEPLIBS-g2c wasn't installed in the
wgrib2-builder stage.
Install NCEPLIBS-g2c v1.9.0 (the version wgrib2 asks for) in the
wgrib2-builder stage ahead of wgrib2 itself, with:
- USE_PNG=ON (needs libpng-dev, already present)
- USE_AEC=ON (needs libaec-dev, already present)
- USE_Jasper=OFF (default ON — would need libjasper-dev
and another runtime dep we don't need)
- USE_OpenJPEG=OFF (same rationale)
- BUILD_G2C=OFF (new experimental API, not needed here)
Then flip wgrib2's PNG/AEC flags back on. wgrib2's binary links
dynamically against libg2c.so so the final stage copies
/usr/local/lib/libg2c.so* from the builder and refreshes the
loader cache with ldconfig.
With the decoder restored, re-add MrmsFetchWorker to the prod
cron in runtime.exs. That worker pulls NOAA MRMS PrecipRate grids
every 2 minutes, which use GRIB2 DRT 5.41 (PNG-compressed) — the
whole reason we needed the PNG decoder in the first place. ASOS
nudges will now have a live radar overlay again.
The previous commit that added -DUSE_PNG=ON -DUSE_AEC=ON to the
wgrib2 cmake invocation broke the image build: wgrib2 3.6.0 turns
find_package(g2c 1.9.0 REQUIRED) on when those flags are enabled,
and NCEPLIBS-g2c isn't installed in the wgrib2-builder stage.
Revert to the stock wgrib2 cmake config so CI is green again and
pull MrmsFetchWorker out of the prod cron instead. Without the
PNG decoder the worker can't read MRMS PrecipRate (GRIB2 DRT
5.41) and was logging "packing type 41 not supported" every two
minutes. AsosAdjustmentWorker's MRMS path already handles an
empty cache gracefully — the ASOS nudge still runs, it just
drops the live radar overlay.
Re-add MrmsFetchWorker once the Dockerfile installs NCEPLIBS-g2c
and rebuilds wgrib2 with PNG/AEC support.
The running label "Updating propagation (HRRR run)" didn't fit in
the 224 px desktop sidebar alongside the NTMS title block, so it
was being truncated. Move the chip to the bottom of the sidebar
(below the auth menu) where it can use the full sidebar width,
let the text wrap naturally instead of truncating, and drop the
trailing ellipsis. Mobile chip stays at the top since the mobile
layout collapses menu + auth into a dropdown.
AsosAdjustmentWorker fires every 10 minutes and loads every row of
`hrrr_profiles` on the grid for the latest valid_time. The old query
selected `profile: h.profile` — ~1.3 KB of JSONB × 92k grid points ≈
120 MB of JSONB per tick. Postgrex's Jason.decode! ran inline for
each row and blew past the 15 s pool checkout window, so every tick
was killing connections with:
DBConnection.ConnectionError: client timed out because it queued
and checked out the connection for longer than 15000ms
`score_grid_point/4` only touched the profile array to re-derive
`min_refractivity_gradient`, but `hrrr_profiles` already persists
that value as a scalar column at ingestion time. Teach
`derive_from_hrrr/1` to honour the persisted scalar when it's
present and drop `h.profile` from the worker's SELECT list. Net
effect: same score math, ~1% of the JSONB transfer, tick stays
under the pool deadline.
Covered by a new scorer test that feeds a profile map with no
`:profile` list and asserts the refractivity factor still reflects
the persisted gradient instead of the neutral baseline.
Prod wgrib2 3.6.0 was built without PNG support (cmake had no
-DUSE_PNG flag) so MrmsFetchWorker crashed on every tick with
'packing type 41 not supported' — MRMS PrecipRate tiles are
PNG-compressed (GRIB2 DRT 5.41). libpng-dev was already in the
builder stage; the cmake invocation just never asked wgrib2 to
use it.
Also turn on -DUSE_AEC=ON since libaec-dev is already installed
and some HRRR native-grid variables ship with AEC/CCSDS 121
compression.
Oban's Lifeline plugin was rescuing PropagationGridWorker from
executing back to retryable every 10 minutes (runtime.exs) or
30 minutes (config.exs / dev.exs), but one full f00–f18 sweep
takes ~95 minutes and the worker's own timeout/1 is already
90 minutes. The rescue fired mid-run, the retry restarted at
f00, and after 3 rescues the job was discarded with errors: []
(Lifeline rescues don't write an error summary). prod's
propagation_scores never advanced beyond ~5 valid_times, so
the forecast timeline on /map only showed 3–4 hours instead of
the intended ~17.
Raise rescue_after to 120 minutes in all three configs — past
the worker's own 90-minute cap with headroom — and document
the contract so the next concurrency tweak doesn't regress it.
Also restore MrmsFetchWorker and AsosAdjustmentWorker to the
prod cron in runtime.exs: runtime.exs replaces config.exs's
plugin list wholesale and those two had been silently dropped
in the last rewrite. AsosAdjustmentWorker depends on the MRMS
precip cache, so both move together.
Adds a small chip below the NTMS title in both the mobile and
desktop sidebars of /map that collapses PropagationGridWorker +
AsosAdjustmentWorker Oban state into one of four states —
running / idle / stale / unknown — with a plain-language label
("Updating propagation (HRRR run)…", "Up to date · 12m ago").
The chip refreshes on a 15s timer and whenever the pipeline
broadcasts propagation:updated, so a long HRRR sweep is visible
from the moment a visitor opens the page.
Observed in prod: after the CDS cap guard shipped, jobs started
landing but one leg per tile-month would routinely stay in 'accepted'
state for 16+ hours while the other completed in ~90 minutes. CDS's
per-user queue fairness apparently serializes competing leg requests
and the single-level ones got stranded behind higher-priority work.
Add an age-based stuck detector in Era5PollWorker: if a row has been
submitted for more than 4 hours and neither leg is fully done, treat
both legs as terminal, clean them up from CDS, and re-enqueue the
submit. Re-uses the resubmit_vanished path. 4h is ~8× the normal
completion time so transient queue variance doesn't trip it, but
tight enough that stuck jobs self-heal in a working session.
CDS returns status="rejected" (distinct from "failed") when a submit
is rate-limited past the 150-job per-user cap. The poll worker's
interpret_status_body treated this as an unknown status → generic
retryable error → 10 retries → discarded. 102 jobs hit this path
overnight.
Fix:
* Era5Client.check_status/1 now returns {:rejected, reason} (and
treats "dismissed" the same way — that's the status a manually
deleted job transitions to).
* Era5PollWorker handles :rejected exactly like :not_found: drop the
DB row, clean up the sibling CDS job, re-enqueue the submit, and
discard the current Oban job so it doesn't burn attempts retrying
a terminal state.
* Widened Era5SubmitWorker cap headroom from 10 → 30 (effective
ceiling 120 not 140). The race between concurrent workers all
passing the count check simultaneously meant we were reaching 141
in-flight against the 150 hard cap; 30 slots of headroom tolerates
a ~15-worker race before clipping CDS's ceiling.
Refactored handle_row/1 into handle_row + handle_non_both_done with a
leg_outcome/1 tag helper so the cross-product of leg statuses collapses
to a single pair match (credo complexity limit).
Replaces the two-phase (algorithm-scores pretrain + QSO-distance
finetune) pipeline with a single unbiased phase that samples
hrrr_profiles stratified uniformly by calendar month and trains on the
physical algorithm score from Scorer.composite_score/2.
Previous Phase 2 target was 'within-band distance percentile' computed
from the contacts table. Because non-contest months have almost no
QSOs, the model learned 'these conditions → QSO happened → good' and
implicitly 'no QSO → bad', so it systematically under-predicted
propagation during quiet months (Feb, Mar, Nov). The new pipeline
decouples the target from QSO activity entirely.
Stratification uses ROW_NUMBER() OVER (PARTITION BY month ORDER BY
random()) so every month contributes the same number of profile rows
regardless of contest-driven density. Each profile is exploded into
one training row per band.
Trained model: val RMSE=1.66 pts, R²=0.9744 on 840k rows (10k
profiles/month × 7 bands, 50 epochs). Monthly sanity check at fixed
weather conditions (10 GHz, 15°C/10°C) now shows Jan=Feb=Mar=68 — the
model no longer penalizes quiet months.
CDS enforces a per-user cap of 150 queued jobs. Each successful submit
adds 2 rows to era5_cds_jobs (single-level + pressure-level), so when
the current in-flight count + 2 would exceed (150 - 10 headroom), the
worker snoozes for 5 minutes instead of POSTing to CDS. Stops 'Number
of queued requests is limited to 150' rejects from burning Oban
attempts.
Contacts whose weather/hrrr/terrain/iemre status has been :queued for
more than 3 days (144 cron cycles) are flipped to :unavailable. The
set_enrichment_status!/3 helper is a no-op when the status is unchanged,
so a stuck row retains a stale updated_at — which is exactly the signal
we need for 'the cron has already tried and the data doesnt exist
upstream' (dead ASOS station, pre-2014 HRRR, out-of-grid IEMRE). Clears
~1,492 pre-existing stuck contacts and stops them from churning the
backfill queue.
When CDS returns 404 for a poll check, the job is gone (manually
deleted, reaped, or never existed). Retrying will always 404, so treat
it as terminal: drop the era5_cds_jobs row, delete the sibling CDS job,
re-enqueue Era5SubmitWorker for the tile-month, and discard the Oban
poll job. Fixes 30 retryable poll jobs left behind by the interrupted
manual-cleanup script.
Ports the same two overlay toggles from /map to /weather:
- Maidenhead grid squares via the shared updateGridOverlay helper,
redrawn on moveend when visible.
- ECCC GeoMet WMS radar (CONUS + Canada dBZ), lazy-created on first
toggle with a 6-minute tile refresh timer.
Both toggles live in the desktop sidebar below the layer description
and in the mobile pulldown panel. Hook cleanup clears the radar
refresh timer on destroyed().
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Four independent wins from a perf audit of the era5_batch worker path:
1. Parallel CDS submits per job. fetch_single_level_month and
fetch_pressure_level_month have no ordering dependency — each blocks
30+ min on CDS queue/download — so running them in two linked Tasks
halves wall time for every job in the backlog.
2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
A month-tile GRIB is 50–200 MB and was previously slurped into the
BEAM heap and then written back out to a temp file before wgrib2 ran.
New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
full parallelism. Partial files are removed on HTTP error.
3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
streamed temp file directly. The binary variant still exists for the
single-point path and now delegates through the same helper.
4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
three-range scan used by Weather.find_nearest_era5 (contact detail +
path profiles) and StatusLive.count_era5_done (status page EXISTS
subquery). The unique index on (lat, lon, valid_time) couldn't serve
it efficiently.
Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
Raise local_limit 2→4 (12 concurrent cluster-wide) and rate_limit 10→30
per hour. Each job makes 2 serial CDS requests, so active workers map 1:1
to CDS slots; 12 sits safely under CDS-Beta's ~16 per-user ceiling while
the rate gate stays above steady-state so it stops being the bottleneck.
The Pro Smart engine's global_limit tracker is stored inside
oban_producers.meta and is not reliably released when a pod exits
between job start and completion. After today's rollout two pods died
holding tracked slots for running era5_batch jobs; the survivors refused
to start any new ones because the cluster-wide tracker was stuck at
4/4 — with zero actual jobs executing — and the whole backfill froze
for 2+ hours.
Keeping rate_limit (10/hr) is enough to stay polite to Copernicus CDS.
local_limit stays at 2 per pod, so effective cluster concurrency just
scales with replica count instead of being capped by a global value
whose bookkeeping can't survive a restart.
Contact detail page now renders an atmospheric profile section backed by
HRRR when available and ERA5 (reanalysis) otherwise, labeled with the
data source. When both sources are missing and HRRR is known-unavailable
(pre-2014 contacts), the Era5FetchWorker is enqueued so the month-tile
backfill gets triggered from a user visit.
Status page ERA5 progress previously computed "complete" as
(total_contacts - hrrr_unavailable_contacts), which treated every contact
with any HRRR status as ERA5-complete — inflating the bar to 76% while
era5_profiles had 0 rows. Replaced with a per-candidate EXISTS query
against era5_profiles using the same 0.15°/30-min tolerance
Weather.find_nearest_era5/3 uses at lookup time.
