Prod hit "no parseable values in cdo output" for a 2007 record.
Without the actual cdo output it's guesswork whether it's a different
parameter-table format, an empty result, or something else. Capture
stderr (stderr_to_stdout: true) and include first 500 chars of output
so the error message has enough signal to diagnose.
Post-2014 contacts with hrrr_status=:unavailable were being dispatched to
NARR, which 404s because NCEI's archive ends 2014-10-02. 14K of the 14.3K
candidates in prod are post-2014 — these are HRRR's responsibility.
- NarrClient.in_coverage?/1 returns true only inside 1979-01-01 → 2014-10-02
- narr_jobs_for_contact and maybe_enqueue_narr (contact_live) bail out of
coverage returning []
- BackfillEnqueueWorker.type_filter scopes :narr to qso_timestamp <
coverage_end so the cron doesn't keep picking post-2014 candidates
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.
Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs
Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
(the symbol still means "historical backfill", just routed to NARR now)
Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).
Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
Swap the body of era5_jobs_for_contact/1 to enqueue NarrFetchWorker
instead of Era5FetchWorker, and snap qso_timestamp via
NarrClient.snap_to_analysis_hour/1 so it lands on a 3-hourly NARR
analysis slot (00/03/06/09/12/15/18/21 UTC) — NarrFetchWorker raises
on anything else.
The :era5 type key, the era5_jobs_for_contact function name, and
the era5_status field are kept as historical artifacts — they
still refer to the era5_profiles table, which is reused 1:1 by
NARR. A schema rename is a follow-up PR.
The CDS Era5* workers (Era5FetchWorker, Era5SubmitWorker,
Era5PollWorker, Era5MonthBatchWorker, Era5BatchClient) are now
unreachable from the live submission flow. They stay in tree until
the deletion follow-up PR.
Per-contact Oban worker that fetches one NARR profile via
NarrClient.fetch_profile_at/2 and inserts it into era5_profiles.
Replaces Era5FetchWorker for pre-2014 contacts. Unlike the ERA5
router, NARR fetches are per-point so this worker does the fetch
+ insert directly — no downstream batch worker, no routing.
Args shape matches Era5FetchWorker: %{"lat", "lon", "valid_time"}.
Lat/lon get snapped to 0.25° (same dedup granularity as ERA5).
valid_time MUST already be on a 3-hourly NARR analysis slot
(00/03/06/09/12/15/18/21 UTC on the hour) — the worker raises
ArgumentError via NarrClient.url_for/1 otherwise. Caller-side
snapping (in ContactWeatherEnqueueWorker) lands in the next task.
Existence check tightened to ±15 minutes (vs ERA5's ±30) since NARR
is exact 3-hourly. On error, logs a warning and returns {:error,
reason} so Oban retries with backoff (max_attempts: 3).
The :narr Oban queue config and the ContactWeatherEnqueueWorker
swap are deliberately NOT touched in this commit — those are the
next two tasks of the plan.
extract_profile_from_file shells out to cdo:
cdo -outputtab,name,lev,value -remapnn,lon=X_lat=Y <merged.grb>
parses the text output (varNNN ↔ NCEP GRIB1 param number lookup),
maps surface records to era5_profiles fields, builds the pressure
profile from each (TMP, HGT, SPFH) triple, and merges in the derived
fields from SoundingParams.derive/1. Returns the same attrs shape as
Era5BatchClient.build_profile_attrs/5 so the existing
bulk_insert_profiles path can ingest NARR rows unchanged.
fetch_profile_at picks records from a fetched inventory, byte-range
fetches each one into a per-record temp file, runs cdo -merge to build
a composite, calls extract_profile_from_file, and cleans up via
try/after.
Drive-by fix: ThetaE.dewpoint_from_spfh/2 returns Kelvin (its existing
test asserts that explicitly). Era5BatchClient was passing that
Kelvin value straight into "dwpc" (Celsius), which would then crash
SoundingParams.compute_refractivity_profile when it called Buck's
saturation-vapor-pressure equation with t=294 instead of t=21. The
bug never surfaced because era5_profiles is empty in production —
no ERA5 backfill has ever succeeded. Fixed in both era5_batch_client
and the new narr_client by subtracting 273.15 in the wrapper. Both
wrappers now have a comment pointing at the K→C conversion.
Test fixture test/fixtures/narr/narr_dfw_2010-06-15_12z.grb is the
spike-captured composite (1.1 MB — Lambert conformal projection
isn't sellonlatbox-subsettable so we can't shrink it further).
parse_inventory walks the wgrib1-format .inv text and returns a
{var, level} -> {byte_offset, length} index, computing each record's
length from the next record's offset (or file_size for the last one).
fetch_inventory does a GET on the .inv URL plus a HEAD on the .grb URL
to learn its size, then delegates to parse_inventory.
Both stubbable via Req.Test (config/test.exs adds narr_req_options
matching the existing era5_req_options / giro_req_options pattern).
Spike fixture at test/fixtures/narr/narr-a_221_20100615_1200_000.inv
backs the parser test — verifies the ("TMP", "2 m above gnd") and
("WVVFLX", "atmos col") records line up with the real bytes.
First slice of the NARR backfill client: url_for/1,
url_for_inventory/1, and snap_to_analysis_hour/1. Pure functions,
no network. Validates that NARR analyses only exist at 3-hourly
slots (00/03/06/09/12/15/18/21 UTC). The byte-range fetch and
cdo-driven point extraction land in follow-up commits per
docs/plans/2026-04-15-merra2-historical-backfill.md.
The persisted grid_data files contain atoms (:base_m, :top_m,
:thickness_m, :min_freq_ghz, :native_min_gradient, :ducts, etc.) that
live in modules which aren't eagerly loaded during application start.
warm_grid_cache_from_latest_profile runs in Application.start/2 before
those modules get pulled in by their first call sites, so [:safe] was
rejecting our own legitimate files at boot — every weather page hit
crashed with "invalid or unsafe external representation of a term".
The whole point of [:safe] is to refuse atoms from untrusted external
ETF input. ProfilesFile.write!/2 in this same module is the only writer
of these files, so the input is trusted by definition — drop the flag.
The test additions round-trip the production data shape including duct
fields as a regression guard.
Prod has 0 era5_profiles rows after ~500 submit cycles: every CDS job
either gets marked "rejected" (likely cap mismatch) or runs for 13+h
without transitioning to successful, so the handle_both_done branch
has literally never fired and the ERA5Batch: stored log line has
never been emitted once. Tighten Era5PollWorker's @stuck_after_seconds
from 18h to 8h so dead rows recycle within a work shift instead of a
full day, and pause the era5 / era5_submit / era5_poll / era5_batch
queues in runtime.exs so existing rows stay put for inspection while
we figure out why CDS is rejecting everything. Flip paused: true back
once the root cause is identified.
Timeline scrubs on /weather were reading a 21 MB NFS ProfilesFile and
then calling SoundingParams.derive on all 92k CONUS points before
filtering to the viewport. Flip the order: filter the raw grid_data map
by bounds first, so derivation work is bounded by the viewport (~4k
points on a DFW box) instead of the full grid.
Add build_grid_cache_rows/3 with an optional bounds argument; the /2
arity is unchanged so PropagationGridWorker still derives the full grid
on f00.
First HF prediction building block. GIRO publishes MUFD — the F2-layer
maximum usable frequency at the 3000 km reference distance — directly
from station measurements, already calibrated against CCIR M(3000)F2
climatology. Rather than recomputing MUF from scratch (which would
require the ~2 MB CCIR coefficient tables), this module treats the
measured MUFD as the anchor and scales it to the actual path distance
using a thin-layer secant-of-incidence ratio.
That keeps the measurement calibration (~3.3× foF2 at our fixture, vs
~5.1× from a naive thin-layer formula) and only uses the thin-layer
math for the relative distance correction.
* adjust_mufd/2 — scales MUFD(3000) to an arbitrary hop distance.
At D=3000 it's a no-op; shorter paths get lower MUF (near-vertical),
longer paths within the single-hop window get higher MUF.
* fot/1 — standard 0.85 × MUF Frequency of Optimum Traffic.
* hf_score/2 — 0-100 band-vs-MUF score with 5-tier curve:
100 (≤ 0.75×MUF) / 80 (FOT) / 50 (≤ MUF) / 20 (fringe) / 0 (dead).
Limitations are in the moduledoc: single-hop only, nearest-station
bias, no LUF/D-layer absorption, no Kp geomagnetic storm modeling.
Those are separate commits once this has bake time.
Not yet wired into PathLive — follow-up commit will add HF bands to
BandConfig and wire the panel.
Adds the second ionospheric data source layer. Polls NOAA SWPC's free
public JSON services every 5 minutes for three products:
* Planetary K-index (1-min cadence) → geomagnetic_observations
* 10.7 cm solar flux (hourly) → solar_flux_observations
* GOES X-ray flux (1-min, 0.1-0.8nm long-wave band only) →
solar_xray_observations
All three products feed HF / Es scoring inputs that the propagation
engine will start using in a follow-up commit:
* Kp drives HF absorption and Es damping (memory notes Kp is inversely
correlated with tropo/Es stability).
* F10.7 is the SSN proxy for ITU-R P.533 HF MUF prediction.
* GOES X-ray long-wave flux is the short-wave fade (SWF) / D-layer
absorption proxy — a C-class flare starts attenuating HF within
seconds of the X-ray peak.
New context `Microwaveprop.SpaceWeather` exposes `upsert_*/1` and
`latest_*/0` per product; `SpaceWeatherFetchWorker` pulls all three
independently so one endpoint's outage doesn't block the others.
Fixtures captured from live SWPC endpoints on 2026-04-15 for the
parser tests.
Not yet wired into scoring — that's a separate commit.
Adds the missing link between the GIRO ionosonde data we just started
ingesting and the VHF/UHF band scoring:
* `Microwaveprop.Propagation.SporadicE` — ITU-R P.534-6 / Davies 1990
thin-layer Es MUF (sec(i) · foEs with h=110 km), plus an `es_score/3`
that maps (foEs, target band, hop distance) into a 0-100 single-hop
propagation likelihood. Calibrated against literature: 50 MHz Es at
2000 km needs foEs ≳ 5.5 MHz (routine summer); 144 MHz needs
foEs ≳ 15.8 MHz (rare Jun/Jul peaks only); 440 MHz Es does not occur
at physical foEs values. Multi-hop Es and Es-scatter are separate
factors and explicitly out of scope.
* `Ionosphere.nearest_foes/3` — given a lat/lon, returns the latest
observation from the nearest polled GIRO station (Millstone Hill or
Alpena for now), with a configurable staleness cutoff (default 2h).
Returns `{:error, :stale}` or `{:error, :no_data}` so callers can
choose whether to apply the Es factor at all.
Neither is wired into the grid scorer yet — that's a separate commit
so the integration can be reviewed on its own.
First step toward physics-based HF / sporadic-E scoring. Polls GIRO's
DIDBase tabular endpoint every 10 minutes for two US ionosondes
(Millstone Hill MHJ45, Alpena AL945), parses the plain-text response
into observation rows, and upserts into a new ionosonde_observations
table keyed on (station_code, valid_time).
This gets foEs (E-layer sporadic critical frequency) into the database
as a direct measurement — the key input for predicting 144 MHz Es
openings via ITU-R P.534-6. foF2 + MUFD are the F2-layer inputs for
HF MUF predictions.
Not yet wired into scoring. Boulder/Wallops/Austin/Idaho/Point Arguello
are in the GIRO catalog but were silent when probed — add them back
if/when they come online.
Next steps: SWPC JSON (Kp, F10.7, sunspot), GOES X-ray flux, D-RAP text,
and the P.534-6 Es scoring factor that uses foEs at midpoint for the
144/440 band configs.
Both bands share the physics of low microwave: humidity and refractivity
gradient build ducts the same way, and rain + gas absorption are
effectively zero at VHF/UHF. humidity_effect is :beneficial, rain_k /
rain_alpha are (0, 1), and o2_db_km / h2o_coeff are 0.
Range estimates reflect tropo reality: 2m ducts can reach 2500+ km
under exceptional conditions (e.g. documented Hawaii ↔ California
openings), 70cm a bit less. The seasonal base matches the low-microwave
summer-peak curve.
These configs do NOT capture sporadic-E, meteor scatter, or aurora —
those need ionospheric data (GIRO ionosondes, NOAA SWPC) we don't yet
ingest. The scoring on these bands is therefore "tropo-only" and will
underestimate openings driven by ionospheric modes.
Two related UI changes driven by the same data:
* Move the selected-layer description out of the sidebar and into a
dedicated top-right overlay on the map. The sidebar kept the same
layer buttons but dropped the descriptive text; the new overlay
shows group + layer name + description in one card. Mobile still
shows the description in the collapsible panel since it has no
top-right free real estate.
* Add a forecast-hour timeline bar at the bottom of the map, matching
the propagation map. WeatherMapLive enumerates ProfilesFile on mount
(the grid worker has been persisting f00..f18 on disk since
commit 07ffcf5), pushes data-valid-times for the JS hook to render
as buttons, and handles a new select_time event by reading the
per-hour ProfilesFile on demand via Weather.weather_grid_at/2.
weather_grid_at/2 deliberately skips the GridCache write path for
non-latest hours — caching 18 × 92k rows would add ~300 MB per pod.
A ~2 MB ETF decode per scrub is fast enough for a click.
Subscribes to propagation:pipeline so the timeline picks up new
forecast hours live as they land, without waiting for the full chain
to finish.
A brief CDS network blip on 2026-04-15 returned HTTP 5xx from check_status
for multiple concurrently-polling workers. The old `{:error, _}` branches
consumed attempts, max_attempts=10 burned out in seconds, and 13 rows
were discarded with no cleanup — orphaned with no active poll worker,
including two rows that had both CDS legs done and were ready to decode.
Result: zero era5_profiles ever inserted despite 59 tile-months in flight.
Transient HTTP errors are now a `{:snooze, 60}` so a CDS outage pauses
polling instead of killing it. Terminal states (:failed, :rejected,
:not_found, :stuck) still drive the fail/resubmit paths.
The grid worker stopped writing HRRR profiles to the DB on April 14
when the forecast grid moved to /data/scores binary files, but the
24/72h-windowed prune left every is_grid_point=true row older than
72 hours sitting in the table (billions of rows across every
partition back to 2016, ~90% of the 146 GB table size). Nothing
reads them anymore.
Replace prune_old_grid_profiles/0 with purge_grid_point_profiles/0,
which walks every hrrr_profiles partition directly and deletes
is_grid_point=true rows regardless of age. Per-partition DELETEs
avoid a parent-table full scan. QSO-linked rows (is_grid_point=false)
are untouched — those still back contact enrichment and /path.
Call the new purge from the grid worker's end-of-chain hook so any
grid-point row that somehow slips back in gets cleaned up within the
next hour. Add `mix hrrr.purge_grid_points` for a one-shot historical
cleanup.
point_forecast used a strict \`>= now\` cutoff, so the leftmost
"now" sample was always dropped — the HRRR publishing lag means the
newest analysis file is typically 30–60 min behind the wall clock,
which made the filter evict the very hour the chart most wanted to
anchor on. If the chain ever fell behind by even one step the whole
forecast went empty and the chart disappeared.
Use the same window as available_valid_times: keep everything from
one hour before now onward, and fall back to the single newest file
when nothing is fresh enough. Both the cache-path and store-path
flows go through a shared forecast_window helper so the behavior
stays in lockstep with the map timeline.
The "Updating propagation +Nh" chip was double-misleading in prod:
the label's +Nh was relative to the HRRR run time (which lags wall
clock by ~2h), and the progress broadcast fired before the hour was
fetched + persisted — so the chip could read "+8h" while the map
timeline only extended to +4h.
Reframe the label as "through now" / "through +Nh" / "through Nh ago"
by computing the offset from valid_time → now in PipelineStatus.
running_detail, so it matches the semantic the user reads off the
timeline. Move the PubSub broadcast in PropagationGridWorker to fire
after Propagation.replace_scores/2 succeeds, so the chip only
advances once that forecast hour is readable from ScoresFile.
Two related optimizations in the propagation chain hot path. Both
land on the f01..f18 step that was previously OOM-killing prod
pods after the HRRR pressure-level footprint halved wasn't enough
to fit inside 4 Gi.
1. Skip the GridCache broadcast on forecast hours.
/weather only ever renders the analysis hour (latest_grid_valid_time
feeds the map). Building 92k rows, serializing them through PubSub,
and rebuilding the {lat,lon}→row map on all three replicas was
pure waste for f01..f18 — no consumer was reading that data. Only
f00 now calls build_grid_cache_rows + broadcast_put. Point lookups
for non-analysis hours still work through ProfilesFile on disk
(weather_point_detail_from_profiles/3) exactly as before.
2. Fold replace_scores into a single streaming pass.
The old path did `Enum.to_list/1` on the ~460k-entry score stream
followed by `Enum.group_by/2`, holding two full copies of the grid
before any file was written. A single `Enum.reduce/3` that folds
each score into a per-band accumulator keeps only one copy and
eliminates the group_by intermediate entirely. The public
signature — an Enumerable in, {:ok, count} out — is unchanged.
The skew-T commit (30c1018) doubled @pressure_levels from 13 to 25 so
new contact fetches would cover the full troposphere. That list is
also what PropagationGridWorker pulls per forecast hour, which
doubled the GRIB footprint (~57 MB compressed + 92k points × 25
levels × 3 vars decoded through wgrib2) and pushed prod pods over
their 4 Gi OOMKill threshold. Every chain died during f00 and the
map timeline never got beyond now and now+1h because the .ntms files
for f02-f18 were never written.
Split the constant:
* @profile_pressure_levels (25 levels, 1000-100 mb) drives the
per-contact HrrrClient.fetch_profile path so the skew-T plot
keeps its full-atmosphere trace.
* @grid_pressure_levels (13 levels, 1000-700 mb) drives the grid
hot path. That's the band SoundingParams.derive reads for
min_refractivity_gradient, and native hybrid-sigma data
(native_min_gradient) takes priority over the pressure-level
fallback anyway, so upper-air levels contribute nothing to
scoring — pure memory waste on this path.
build_profile/1 still iterates the full 25-level list; grid fetches
simply populate the 13 near-surface slots and skip the rest.
Makes HrrrClient.pressure_messages public with a :grid | :profile
variant so the split is testable from outside the module.
Two related fixes so the main map reliably picks up new binary
propagation score files as soon as PropagationGridWorker writes them.
1. Propagation.available_valid_times/1 previously preferred ScoreCache
over ScoresFile, using the cache as an index of what was available.
The cache is a lazy ETS of whatever hours have been fetched or
broadcast, which is a strict subset of what's on disk. A new
forecast hour landing on disk while the cache was warm with older
entries was invisible to the timeline until the cache happened to
catch up. Read directly from ScoresFile so the disk store is the
source of truth.
2. Add Propagation.scores_at_fresh/3 that always reads the .ntms file
and overwrites the cache entry, and use it from MapLive's
propagation_updated handler. PropagationGridWorker publishes the
cache_refresh on `propagation:cache` and the timeline ping on
`propagation:updated` as separate PubSub broadcasts, so by the time
MapLive runs through scores_at the ScoreCache GenServer may not
have processed the refresh yet — fetch_bounds then returns the
previous chain's bytes. scores_at_fresh takes disk as the source
of truth for the refresh path and warms the cache as a side effect
so subsequent readers see the new data.
Historical contacts showed a skew-T log-P diagram that stopped at 700 mb
because HrrrClient and Era5Client only fetched pressure-level data down
to the top of the boundary layer. The chart canvas already ran up to
100 mb, so the trace clipped mid-atmosphere.
Two complementary fixes:
1. Extend @pressure_levels in HrrrClient and Era5Client with
650/600/550/500/450/400/350/300/250/200/150/100 mb so new fetches
cover the full troposphere + lower stratosphere.
2. Prefer the native hybrid-sigma profile for the contact-detail
skew-T when one has been backfilled for the contact's hour. The
native profile already stores all 50 hybrid levels up to ~19 km,
so historical contacts covered by the native backfill get a full
trace without re-hitting S3. A new HrrrNativeProfile.to_skew_t_profile/1
converts the parallel arrays into the %{"pres","tmpc","dwpc","hght"}
list shape the renderer expects, deriving dewpoint from SPFH via the
Magnus inverse. Weather.find_nearest_native_profile/3 mirrors
find_nearest_hrrr/3 for the lookup.
For a 1 W / 432 MHz beacon, RangeEstimate.estimate/1 was returning
~126 k cells because free-space loss dominates over atmospheric
loss at low UHF, letting the rx_dbm filter pass almost the whole
bbox. Jason-encoding that payload into the map's data-cells
attribute was locking up browsers on the beacon detail page for
low-band beacons even though the coverage toggle is already
disabled for them. Return an empty estimate below the same 5760
MHz threshold the UI already enforces.
Replace the timer-based Oban.Plugins.Lifeline (45 min rescue_after)
with Oban.Pro.Plugins.DynamicLifeline, which watches producer
records and rescues orphaned jobs within one rescue_interval (30s)
after the owning pod disappears. This cuts PropagationGridWorker
chain recovery from up to 55 min (wait for next hourly cron) down
to ~30s after a rolling deploy kills a mid-flight step. Bump the
worker's max_attempts 3 → 5 so a couple of rescues during a deploy
don't exhaust the chain's retry budget.
Extend the f00 profile persistence to cover every forecast hour
(f00-f18) so /weather keeps working across pod restarts and shows
forecast-hour atmospheric data without a fresh database. Route the
Weather cold path (latest_grid_valid_time, load_weather_grid,
weather_point_detail) through ProfilesFile first, with the legacy
hrrr_profiles table as a last-resort historical fallback. Warm
GridCache from the latest profile on app start so /weather is hot
the moment a pod boots.
The propagation_scores → binary files cutover dropped the factors
JSONB column, which left point_detail returning factors: %{} and
broke the analysis breakdown popup on map clicks. Persist the
enriched f00 grid_data to /data/scores/profiles/{iso}.etf.gz and
rescore on demand at click time so the factor block renders again.
Flip the cron from every-3-hours to every-hour now that a full
f00-f18 chain runs in ~45-60 min instead of ~170 min. With the
:propagation queue's concurrency-of-2, a slow chain won't block
the next one — they interleave, and the file store's last-writer
-wins semantics give the newer run_time's analysis data naturally.
Add ScoresFile.retain_window/2 that deletes any file whose
valid_time falls outside [run_time, run_time + 18h]. Call it at
fh=18 of the chain so leftover files from the previous chain
(specifically the old f00 whose valid_time the new chain doesn't
overwrite because it advanced by one hour) are discarded
immediately instead of waiting for the 2h prune cron.
The step transition path is extracted to handle_step_transition/2
to keep the nesting under credo's limit and make the final-step
logic easier to read.
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:
1. New migration drops the propagation_scores table and its indexes
(the earlier tuning migrations for it were already applied and
are now no-ops against a missing table, which is fine — Ecto
just runs them on fresh environments).
2. Propagation context is gutted of every GridScore reference.
replace_scores/2 writes files only. upsert_scores/2 is deleted.
load_scores_from_db, available_valid_times_from_db,
point_detail_from_db, point_forecast_from_db, fetch_factors,
coalesce_factors, the Postgres side of prune_old_scores, and
the Postgres fallbacks in latest/earliest_valid_time are all
removed. point_detail always returns an empty factors map now
since factor breakdowns were retired with the table.
3. Deleted modules:
- Propagation.GridScore (the schema)
- Propagation.ScorerDiff (read factors from the table)
- Propagation.AsosNudge (helper for AsosAdjustmentWorker)
- Workers.AsosAdjustmentWorker (its cron was already disabled)
- Mix.Tasks.ScorerDiff (wrapper around the deleted module)
And their tests. AdminTaskWorker's scorer_diff task is a
logging no-op so any queued Oban rows drain cleanly.
Release.scorer_diff stays as a stub that tells the operator.
propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
PropagationGridWorker.merge_commercial_link_data/2 called
Commercial.link_degradation_at twice per grid cell (once to count,
once to merge), and each call re-ran enabled_links plus two sample
queries per in-range link. That was ~500k SQL queries per forecast
hour — fine for the DB but catastrophic for log volume in dev
iex sessions trying to watch a chain step run.
Split into two new functions:
* build_link_lookup/2 does all the DB work up front — one
enabled_links query + one link_degradation per enabled link
(~10 queries total).
* link_degradation_from_lookup/3 is pure: takes a (lat, lon)
and the precomputed lookup, returns the aggregate or nil.
The worker now calls build_link_lookup once per forecast hour and
the per-cell path is haversine-only. Net: 500k queries → ~10.
PipelineStatus freshness detection moves off oban_jobs.completed_at
onto ScoresFile.latest_valid_time(). The on-disk files ARE the data
the map renders from, so the chip's "Up to date · Nm ago" and the
2h stale threshold now track actual data presence. Running-state
detection still comes from oban_jobs since the chain step is still
an Oban row. Tests updated to seed a ScoresFile instead of a
completed Oban row for the idle/stale cases.
Read-side cutover for the binary scores store and a companion
cleanup that removes the biggest remaining DB write from the hot
path.
Propagation.scores_at/3, available_valid_times/1, latest_valid_time/0,
latest_valid_time/1, earliest_valid_time/1, point_detail/4, and
point_forecast/3 all now prefer ScoresFile and fall back to the
propagation_scores table when a file is missing. The map render
path reads from /data/scores first; Postgres stays as a safety
net while dual-write is on. point_detail still pulls factors from
Postgres (analysis-hour rows only) and coalesces nil to an empty
map so the JS popup iterates cleanly.
replace_scores/2 is now gated by a postgres_writes_enabled? flag
(runtime env MICROWAVEPROP_SCORES_POSTGRES=false, or the
:propagation_scores_postgres app env key) so the binary-only path
can be benchmarked locally without the DB insert. Default stays
true.
PropagationGridWorker no longer calls store_hrrr_profiles —
persisting 92k grid rows × 19 forecast hours of JSONB profiles
was ~12 min of wall time per chain for a table only
AsosAdjustmentWorker read from. Per-contact HRRR enrichment
through HrrrFetchWorker still writes its own (is_grid_point:
false) rows. AsosAdjustmentWorker is disabled in all three cron
configs since its data source is gone.
DataCase resets the scores tree between tests so per-test
ScoresFile writes don't leak across cases, and ScoresFileTest
switches to async: false because it mutates the global
:propagation_scores_dir env.
First step of the disk-backed scores migration from the DuckDB plan
doc. Ended up shipping the raw-binary variant instead of Parquet
because the data is disposable after ~2h — the ecosystem benefits
of Parquet only pay off for long-lived datasets, and the binary
path has zero new dependencies.
Microwaveprop.Propagation.ScoresFile writes one file per
(band_mhz, valid_time) tuple under the configured scores dir,
default /data/scores in prod and priv/dev_scores in dev. Layout is
a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is
the no-data sentinel). The whole CONUS grid serializes to ~93 KB
per band, and writes use the temp-then-rename pattern so NFSv4
concurrent readers never see a partial file.
Propagation.replace_scores/2 now materializes the score stream
once and dual-writes: Postgres on the primary path, then one
ScoresFile per band as a best-effort follow-up (any file error is
logged but doesn't fail the DB write, so we can verify the file
path in prod before cutting readers over).
Propagation.prune_old_scores/0 also clears expired score files so
the existing 15-minute prune cron covers both storage layers.
Dev configuration points at priv/dev_scores/, added to .gitignore.
Test configuration points at a per-run tmp directory.
The :propagation Oban queue runs with concurrency 2, so the grid
worker chain step and the 10-minute ASOS nudge can (and do) execute
in parallel. PipelineStatus.current/0 used to return only the
most-recently-started worker, which made the chip label flip to
"ASOS nudge" at every :00 / :10 tick even though HRRR was still
chugging through forecast hours behind it.
Replace the single :detail field with a list of running_detail
maps, each tagged with :grid or :asos so the chip can:
- Render one sub-line per currently-executing worker
- Sort them deterministically (grid before asos) so the eye
doesn't see rows swap
- Still override the grid entry with the forecast-hour progress
label when a propagation_pipeline_progress broadcast arrives
The scoring+upsert phase was ~4m40s per forecast hour and dominated
wall time. Three stacked optimizations attack it from different
angles.
replace_scores/2 is a new hot-path writer that does DELETE WHERE
valid_time = $1 followed by a plain insert_all (no ON CONFLICT
resolution). The chain worker rewrites the full (valid_time, all
bands) slice every forecast hour, so conflict detection was pure
waste. AsosAdjustmentWorker still uses upsert_scores because it
only rewrites the subset of cells near a station.
factors is now nullable. Forecast hours f01-f18 pass factors: nil
so the JSONB encode + toast write is skipped entirely — roughly
halves the data volume per run. point_detail/4 coalesces nil to
an empty map so the JS popup renders without a TypeError, and
scorer_diff only pulls the most recent valid_time that still has
factors (the f00 row).
propagation_scores is now UNLOGGED, so inserts bypass WAL entirely.
Durability tradeoff: an unclean shutdown truncates the table, but
PropagationGridWorker rebuilds it from HRRR every 3h so a lost
table is re-populated within one cron cycle.
Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a
speculative plan for a flat-file / DuckDB rewrite with explicit
trigger conditions for when to pick it up (partitioning deferred
too; revisit only if these three don't solve it).
A full f00-f18 sweep takes ~170 min of wall time, longer than the
~90 min gap between deploys on this cluster. Over the last 7 days
every run was killed mid-sweep and zero runs completed. The
propagation_scores table only ever held the scraps from partial
runs that landed before the pod died, which is why the map looked
like it was showing "the current hour" (or nothing).
The worker now processes exactly one forecast hour per perform/1
(~8-10 min) and enqueues the next hour as a fresh Oban job. A cron
fire with empty args seeds the chain at f00; subsequent runs carry
run_time + forecast_hour args. At f18 the chain stops and the
pruner runs. Each step broadcasts propagation:updated immediately
so new hours appear on the map as they land.
Wall time per perform drops from ~170 min to ~10 min, so Lifeline's
2h rescue window is no longer a factor and a pod restart loses at
most one forecast hour instead of the whole sweep. Retries and
max_attempts now describe a single fh, not the whole chain.
Also: bump the :propagation queue from 1 to 2 slots so
PropagationPruneWorker can run alongside the chain job, drop the
worker timeout/1 from 90 min to 20 min to match single-fh runs,
and pull Lifeline rescue_after from 120 min to 45 min to keep the
safety net above the step timeout.
Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the
pipeline chip in both the mobile and desktop sidebars so the user
can tell which valid_time the map is showing when the bottom
timeline isn't visible.
Era5SubmitWorker was counting era5_cds_jobs rows 1:1 against the CDS
150-job ceiling, but each row holds TWO CDS job IDs (single-level +
pressure-level). Prod hit 74 rows = 148 in-flight jobs and got stuck
in a reject→resubmit spiral. The cap guard now multiplies row count
by 2 to match what CDS sees.
Era5PollWorker @stuck_after_seconds goes from 4h to 18h. CDS
single-level routinely sits at 'accepted' for 12+h under load while
pressure finishes in ~90 min; the 4h threshold was firing on normal
slow runs and feeding the same spiral.
PipelineStatus now returns `label` + `detail` separately so the map
chip can render "Updating propagation" on one line and the
sub-detail ("HRRR run" / "ASOS nudge" / "+3h" / "now") on a second
line below it. running_label/1 is renamed to running_detail/1 and
returns just the sub-part.
PropagationGridWorker now broadcasts a {:propagation_pipeline_progress,
%{forecast_hour:, valid_time:}} message on the propagation:pipeline
PubSub topic at the start of each process_forecast_hour/4 call.
MapLive subscribes on mount, stashes the last message in the new
:pipeline_progress assign, and clears it on the refresh tick whenever
PipelineStatus flips out of :running.
The chip component consults a new PipelineStatus.running_label/1
helper that formats the payload as "Updating propagation · now" for
f00 and "Updating propagation · +Nh" for f01-f18, falling back to
the base running label until the first progress message arrives.
Covered by unit tests on the formatter and an integration test in
MapLiveTest that seeds an executing grid job, broadcasts progress,
and asserts the rendered chip.
The running label "Updating propagation (HRRR run)" didn't fit in
the 224 px desktop sidebar alongside the NTMS title block, so it
was being truncated. Move the chip to the bottom of the sidebar
(below the auth menu) where it can use the full sidebar width,
let the text wrap naturally instead of truncating, and drop the
trailing ellipsis. Mobile chip stays at the top since the mobile
layout collapses menu + auth into a dropdown.
AsosAdjustmentWorker fires every 10 minutes and loads every row of
`hrrr_profiles` on the grid for the latest valid_time. The old query
selected `profile: h.profile` — ~1.3 KB of JSONB × 92k grid points ≈
120 MB of JSONB per tick. Postgrex's Jason.decode! ran inline for
each row and blew past the 15 s pool checkout window, so every tick
was killing connections with:
DBConnection.ConnectionError: client timed out because it queued
and checked out the connection for longer than 15000ms
`score_grid_point/4` only touched the profile array to re-derive
`min_refractivity_gradient`, but `hrrr_profiles` already persists
that value as a scalar column at ingestion time. Teach
`derive_from_hrrr/1` to honour the persisted scalar when it's
present and drop `h.profile` from the worker's SELECT list. Net
effect: same score math, ~1% of the JSONB transfer, tick stays
under the pool deadline.
Covered by a new scorer test that feeds a profile map with no
`:profile` list and asserts the refractivity factor still reflects
the persisted gradient instead of the neutral baseline.
Adds a small chip below the NTMS title in both the mobile and
desktop sidebars of /map that collapses PropagationGridWorker +
AsosAdjustmentWorker Oban state into one of four states —
running / idle / stale / unknown — with a plain-language label
("Updating propagation (HRRR run)…", "Up to date · 12m ago").
The chip refreshes on a 15s timer and whenever the pipeline
broadcasts propagation:updated, so a long HRRR sweep is visible
from the moment a visitor opens the page.
Observed in prod: after the CDS cap guard shipped, jobs started
landing but one leg per tile-month would routinely stay in 'accepted'
state for 16+ hours while the other completed in ~90 minutes. CDS's
per-user queue fairness apparently serializes competing leg requests
and the single-level ones got stranded behind higher-priority work.
Add an age-based stuck detector in Era5PollWorker: if a row has been
submitted for more than 4 hours and neither leg is fully done, treat
both legs as terminal, clean them up from CDS, and re-enqueue the
submit. Re-uses the resubmit_vanished path. 4h is ~8× the normal
completion time so transient queue variance doesn't trip it, but
tight enough that stuck jobs self-heal in a working session.
CDS returns status="rejected" (distinct from "failed") when a submit
is rate-limited past the 150-job per-user cap. The poll worker's
interpret_status_body treated this as an unknown status → generic
retryable error → 10 retries → discarded. 102 jobs hit this path
overnight.
Fix:
* Era5Client.check_status/1 now returns {:rejected, reason} (and
treats "dismissed" the same way — that's the status a manually
deleted job transitions to).
* Era5PollWorker handles :rejected exactly like :not_found: drop the
DB row, clean up the sibling CDS job, re-enqueue the submit, and
discard the current Oban job so it doesn't burn attempts retrying
a terminal state.
* Widened Era5SubmitWorker cap headroom from 10 → 30 (effective
ceiling 120 not 140). The race between concurrent workers all
passing the count check simultaneously meant we were reaching 141
in-flight against the 150 hard cap; 30 slots of headroom tolerates
a ~15-worker race before clipping CDS's ceiling.
Refactored handle_row/1 into handle_row + handle_non_both_done with a
leg_outcome/1 tag helper so the cross-product of leg statuses collapses
to a single pair match (credo complexity limit).
CDS enforces a per-user cap of 150 queued jobs. Each successful submit
adds 2 rows to era5_cds_jobs (single-level + pressure-level), so when
the current in-flight count + 2 would exceed (150 - 10 headroom), the
worker snoozes for 5 minutes instead of POSTing to CDS. Stops 'Number
of queued requests is limited to 150' rejects from burning Oban
attempts.
Contacts whose weather/hrrr/terrain/iemre status has been :queued for
more than 3 days (144 cron cycles) are flipped to :unavailable. The
set_enrichment_status!/3 helper is a no-op when the status is unchanged,
so a stuck row retains a stale updated_at — which is exactly the signal
we need for 'the cron has already tried and the data doesnt exist
upstream' (dead ASOS station, pre-2014 HRRR, out-of-grid IEMRE). Clears
~1,492 pre-existing stuck contacts and stops them from churning the
backfill queue.
When CDS returns 404 for a poll check, the job is gone (manually
deleted, reaped, or never existed). Retrying will always 404, so treat
it as terminal: drop the era5_cds_jobs row, delete the sibling CDS job,
re-enqueue Era5SubmitWorker for the tile-month, and discard the Oban
poll job. Fixes 30 retryable poll jobs left behind by the interrupted
manual-cleanup script.
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Four independent wins from a perf audit of the era5_batch worker path:
1. Parallel CDS submits per job. fetch_single_level_month and
fetch_pressure_level_month have no ordering dependency — each blocks
30+ min on CDS queue/download — so running them in two linked Tasks
halves wall time for every job in the backlog.
2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
A month-tile GRIB is 50–200 MB and was previously slurped into the
BEAM heap and then written back out to a temp file before wgrib2 ran.
New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
full parallelism. Partial files are removed on HTTP error.
3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
streamed temp file directly. The binary variant still exists for the
single-point path and now delegates through the same helper.
4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
three-range scan used by Weather.find_nearest_era5 (contact detail +
path profiles) and StatusLive.count_era5_done (status page EXISTS
subquery). The unique index on (lat, lon, valid_time) couldn't serve
it efficiently.
Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
