The skew-T commit (30c1018) doubled @pressure_levels from 13 to 25 so
new contact fetches would cover the full troposphere. That list is
also what PropagationGridWorker pulls per forecast hour, which
doubled the GRIB footprint (~57 MB compressed + 92k points × 25
levels × 3 vars decoded through wgrib2) and pushed prod pods over
their 4 Gi OOMKill threshold. Every chain died during f00 and the
map timeline never got beyond now and now+1h because the .ntms files
for f02-f18 were never written.
Split the constant:
* @profile_pressure_levels (25 levels, 1000-100 mb) drives the
per-contact HrrrClient.fetch_profile path so the skew-T plot
keeps its full-atmosphere trace.
* @grid_pressure_levels (13 levels, 1000-700 mb) drives the grid
hot path. That's the band SoundingParams.derive reads for
min_refractivity_gradient, and native hybrid-sigma data
(native_min_gradient) takes priority over the pressure-level
fallback anyway, so upper-air levels contribute nothing to
scoring — pure memory waste on this path.
build_profile/1 still iterates the full 25-level list; grid fetches
simply populate the 13 near-surface slots and skip the rest.
Makes HrrrClient.pressure_messages public with a :grid | :profile
variant so the split is testable from outside the module.
Two related fixes so the main map reliably picks up new binary
propagation score files as soon as PropagationGridWorker writes them.
1. Propagation.available_valid_times/1 previously preferred ScoreCache
over ScoresFile, using the cache as an index of what was available.
The cache is a lazy ETS of whatever hours have been fetched or
broadcast, which is a strict subset of what's on disk. A new
forecast hour landing on disk while the cache was warm with older
entries was invisible to the timeline until the cache happened to
catch up. Read directly from ScoresFile so the disk store is the
source of truth.
2. Add Propagation.scores_at_fresh/3 that always reads the .ntms file
and overwrites the cache entry, and use it from MapLive's
propagation_updated handler. PropagationGridWorker publishes the
cache_refresh on `propagation:cache` and the timeline ping on
`propagation:updated` as separate PubSub broadcasts, so by the time
MapLive runs through scores_at the ScoreCache GenServer may not
have processed the refresh yet — fetch_bounds then returns the
previous chain's bytes. scores_at_fresh takes disk as the source
of truth for the refresh path and warms the cache as a side effect
so subsequent readers see the new data.
With the native hybrid-sigma profile now storing levels all the way up
to ~19 km, the scorer can finally see the mid-to-upper troposphere.
The five factors that would plausibly add signal — 500 mb dewpoint
depression, 300 mb wind speed, 850→500 mb dθ/dz, tropopause height,
and 500 mb height anomaly — get rationale, derivation notes, and
placeholder weights. Actual weights have to wait for the native
backfill to catch up enough QSO hours for a representative training
sample, at which point the existing gradient-descent recalibration
pass refits the full factor set.
Historical contacts showed a skew-T log-P diagram that stopped at 700 mb
because HrrrClient and Era5Client only fetched pressure-level data down
to the top of the boundary layer. The chart canvas already ran up to
100 mb, so the trace clipped mid-atmosphere.
Two complementary fixes:
1. Extend @pressure_levels in HrrrClient and Era5Client with
650/600/550/500/450/400/350/300/250/200/150/100 mb so new fetches
cover the full troposphere + lower stratosphere.
2. Prefer the native hybrid-sigma profile for the contact-detail
skew-T when one has been backfilled for the contact's hour. The
native profile already stores all 50 hybrid levels up to ~19 km,
so historical contacts covered by the native backfill get a full
trace without re-hitting S3. A new HrrrNativeProfile.to_skew_t_profile/1
converts the parallel arrays into the %{"pres","tmpc","dwpc","hght"}
list shape the renderer expects, deriving dewpoint from SPFH via the
Magnus inverse. Weather.find_nearest_native_profile/3 mirrors
find_nearest_hrrr/3 for the lookup.
Previously RangeEstimate.estimate/1 ran on every beacon page mount,
even though the cells data was only rendered after the user flipped
the coverage toggle — a wasted bbox grid pass per page load. Start
with estimate: nil, compute lazily in handle_event("toggle_coverage")
when flipping on, and push the cells payload to the Leaflet hook via
a new load_coverage event. The JS hook no longer reads data-cells on
mount and only builds the cellLayer when the server sends it. Cached
in the socket so re-toggling is instant.
The atmospheric profile header was a single-line flex row with no
wrap, so on narrower viewports the HRRR (3 km) badge and the unit
suffixes were breaking mid-phrase — the badge showed "HRRR (3 /
km)" with borders crossing through the text. Add flex-wrap plus
whitespace-nowrap on each cell so the whole row wraps cleanly
onto a second line instead of shrinking each cell vertically.
For a 1 W / 432 MHz beacon, RangeEstimate.estimate/1 was returning
~126 k cells because free-space loss dominates over atmospheric
loss at low UHF, letting the rx_dbm filter pass almost the whole
bbox. Jason-encoding that payload into the map's data-cells
attribute was locking up browsers on the beacon detail page for
low-band beacons even though the coverage toggle is already
disabled for them. Return an empty estimate below the same 5760
MHz threshold the UI already enforces.
Render the HRRR pressure-level profile as a full skew-T log-P
diagram on the contact detail page: skewed isotherms, isobars,
dry adiabats, saturation mixing ratio lines, plus the T and Td
traces. Math lives in MicrowavepropWeb.SkewT (Magnus formula,
dry adiabat potential temperature, log-P projection) and is
exercised by a dedicated test module. The atmospheric profile
section now expands by default so the chart is visible without
an extra click.
The estimated current coverage map is only meaningful on bands
where atmospheric attenuation shapes the reach — below 5.76 GHz
it's dominated by free-space loss and misleads. Disable the
toggle, show a helper note explaining the limit, and guard the
handle_event against clients that try to force it.
Replace the timer-based Oban.Plugins.Lifeline (45 min rescue_after)
with Oban.Pro.Plugins.DynamicLifeline, which watches producer
records and rescues orphaned jobs within one rescue_interval (30s)
after the owning pod disappears. This cuts PropagationGridWorker
chain recovery from up to 55 min (wait for next hourly cron) down
to ~30s after a rolling deploy kills a mid-flight step. Bump the
worker's max_attempts 3 → 5 so a couple of rescues during a deploy
don't exhaust the chain's retry budget.
Extend the f00 profile persistence to cover every forecast hour
(f00-f18) so /weather keeps working across pod restarts and shows
forecast-hour atmospheric data without a fresh database. Route the
Weather cold path (latest_grid_valid_time, load_weather_grid,
weather_point_detail) through ProfilesFile first, with the legacy
hrrr_profiles table as a last-resort historical fallback. Warm
GridCache from the latest profile on app start so /weather is hot
the moment a pod boots.
The propagation_scores → binary files cutover dropped the factors
JSONB column, which left point_detail returning factors: %{} and
broke the analysis breakdown popup on map clicks. Persist the
enriched f00 grid_data to /data/scores/profiles/{iso}.etf.gz and
rescore on demand at click time so the factor block renders again.
The propagation_scores → binary files cutover dropped the factors
JSONB column, which left point_detail returning factors: %{} and
broke the analysis breakdown popup on map clicks. Persist the
enriched f00 grid_data to /data/scores/profiles/{iso}.etf.gz and
rescore on demand at click time so the factor block renders again.
Flip the cron from every-3-hours to every-hour now that a full
f00-f18 chain runs in ~45-60 min instead of ~170 min. With the
:propagation queue's concurrency-of-2, a slow chain won't block
the next one — they interleave, and the file store's last-writer
-wins semantics give the newer run_time's analysis data naturally.
Add ScoresFile.retain_window/2 that deletes any file whose
valid_time falls outside [run_time, run_time + 18h]. Call it at
fh=18 of the chain so leftover files from the previous chain
(specifically the old f00 whose valid_time the new chain doesn't
overwrite because it advanced by one hour) are discarded
immediately instead of waiting for the 2h prune cron.
The step transition path is extracted to handle_step_transition/2
to keep the nesting under credo's limit and make the final-step
logic easier to read.
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:
1. New migration drops the propagation_scores table and its indexes
(the earlier tuning migrations for it were already applied and
are now no-ops against a missing table, which is fine — Ecto
just runs them on fresh environments).
2. Propagation context is gutted of every GridScore reference.
replace_scores/2 writes files only. upsert_scores/2 is deleted.
load_scores_from_db, available_valid_times_from_db,
point_detail_from_db, point_forecast_from_db, fetch_factors,
coalesce_factors, the Postgres side of prune_old_scores, and
the Postgres fallbacks in latest/earliest_valid_time are all
removed. point_detail always returns an empty factors map now
since factor breakdowns were retired with the table.
3. Deleted modules:
- Propagation.GridScore (the schema)
- Propagation.ScorerDiff (read factors from the table)
- Propagation.AsosNudge (helper for AsosAdjustmentWorker)
- Workers.AsosAdjustmentWorker (its cron was already disabled)
- Mix.Tasks.ScorerDiff (wrapper around the deleted module)
And their tests. AdminTaskWorker's scorer_diff task is a
logging no-op so any queued Oban rows drain cleanly.
Release.scorer_diff stays as a stub that tells the operator.
propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
PropagationGridWorker.merge_commercial_link_data/2 called
Commercial.link_degradation_at twice per grid cell (once to count,
once to merge), and each call re-ran enabled_links plus two sample
queries per in-range link. That was ~500k SQL queries per forecast
hour — fine for the DB but catastrophic for log volume in dev
iex sessions trying to watch a chain step run.
Split into two new functions:
* build_link_lookup/2 does all the DB work up front — one
enabled_links query + one link_degradation per enabled link
(~10 queries total).
* link_degradation_from_lookup/3 is pure: takes a (lat, lon)
and the precomputed lookup, returns the aggregate or nil.
The worker now calls build_link_lookup once per forecast hour and
the per-cell path is haversine-only. Net: 500k queries → ~10.
PipelineStatus freshness detection moves off oban_jobs.completed_at
onto ScoresFile.latest_valid_time(). The on-disk files ARE the data
the map renders from, so the chip's "Up to date · Nm ago" and the
2h stale threshold now track actual data presence. Running-state
detection still comes from oban_jobs since the chain step is still
an Oban row. Tests updated to seed a ScoresFile instead of a
completed Oban row for the idle/stale cases.
Read-side cutover for the binary scores store and a companion
cleanup that removes the biggest remaining DB write from the hot
path.
Propagation.scores_at/3, available_valid_times/1, latest_valid_time/0,
latest_valid_time/1, earliest_valid_time/1, point_detail/4, and
point_forecast/3 all now prefer ScoresFile and fall back to the
propagation_scores table when a file is missing. The map render
path reads from /data/scores first; Postgres stays as a safety
net while dual-write is on. point_detail still pulls factors from
Postgres (analysis-hour rows only) and coalesces nil to an empty
map so the JS popup iterates cleanly.
replace_scores/2 is now gated by a postgres_writes_enabled? flag
(runtime env MICROWAVEPROP_SCORES_POSTGRES=false, or the
:propagation_scores_postgres app env key) so the binary-only path
can be benchmarked locally without the DB insert. Default stays
true.
PropagationGridWorker no longer calls store_hrrr_profiles —
persisting 92k grid rows × 19 forecast hours of JSONB profiles
was ~12 min of wall time per chain for a table only
AsosAdjustmentWorker read from. Per-contact HRRR enrichment
through HrrrFetchWorker still writes its own (is_grid_point:
false) rows. AsosAdjustmentWorker is disabled in all three cron
configs since its data source is gone.
DataCase resets the scores tree between tests so per-test
ScoresFile writes don't leak across cases, and ScoresFileTest
switches to async: false because it mutates the global
:propagation_scores_dir env.
First step of the disk-backed scores migration from the DuckDB plan
doc. Ended up shipping the raw-binary variant instead of Parquet
because the data is disposable after ~2h — the ecosystem benefits
of Parquet only pay off for long-lived datasets, and the binary
path has zero new dependencies.
Microwaveprop.Propagation.ScoresFile writes one file per
(band_mhz, valid_time) tuple under the configured scores dir,
default /data/scores in prod and priv/dev_scores in dev. Layout is
a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is
the no-data sentinel). The whole CONUS grid serializes to ~93 KB
per band, and writes use the temp-then-rename pattern so NFSv4
concurrent readers never see a partial file.
Propagation.replace_scores/2 now materializes the score stream
once and dual-writes: Postgres on the primary path, then one
ScoresFile per band as a best-effort follow-up (any file error is
logged but doesn't fail the DB write, so we can verify the file
path in prod before cutting readers over).
Propagation.prune_old_scores/0 also clears expired score files so
the existing 15-minute prune cron covers both storage layers.
Dev configuration points at priv/dev_scores/, added to .gitignore.
Test configuration points at a per-run tmp directory.
node3's NVMe SSD is the new shared-storage home. Mount the whole
/data export from 10.0.15.103 at /data inside the pod (instead of
the old skippy export of /data/srtm at /srtm) so there's room for
future shared stuff — Parquet score files, maybe a DuckDB database,
or whatever the next thing is — without re-exporting a new path.
SRTM tiles now live at /data/srtm rather than /srtm. Update the
srtm_tiles_dir runtime config to match, and fix the two doc
pointers (CLAUDE.md, duckdb plan) that referenced the old path.
Already applied to the cluster and rolled out; pods confirmed
running with tiles visible at /data/srtm.
The :propagation Oban queue runs with concurrency 2, so the grid
worker chain step and the 10-minute ASOS nudge can (and do) execute
in parallel. PipelineStatus.current/0 used to return only the
most-recently-started worker, which made the chip label flip to
"ASOS nudge" at every :00 / :10 tick even though HRRR was still
chugging through forecast hours behind it.
Replace the single :detail field with a list of running_detail
maps, each tagged with :grid or :asos so the chip can:
- Render one sub-line per currently-executing worker
- Sort them deterministically (grid before asos) so the eye
doesn't see rows swap
- Still override the grid entry with the forecast-hour progress
label when a propagation_pipeline_progress broadcast arrives
The scoring+upsert phase was ~4m40s per forecast hour and dominated
wall time. Three stacked optimizations attack it from different
angles.
replace_scores/2 is a new hot-path writer that does DELETE WHERE
valid_time = $1 followed by a plain insert_all (no ON CONFLICT
resolution). The chain worker rewrites the full (valid_time, all
bands) slice every forecast hour, so conflict detection was pure
waste. AsosAdjustmentWorker still uses upsert_scores because it
only rewrites the subset of cells near a station.
factors is now nullable. Forecast hours f01-f18 pass factors: nil
so the JSONB encode + toast write is skipped entirely — roughly
halves the data volume per run. point_detail/4 coalesces nil to
an empty map so the JS popup renders without a TypeError, and
scorer_diff only pulls the most recent valid_time that still has
factors (the f00 row).
propagation_scores is now UNLOGGED, so inserts bypass WAL entirely.
Durability tradeoff: an unclean shutdown truncates the table, but
PropagationGridWorker rebuilds it from HRRR every 3h so a lost
table is re-populated within one cron cycle.
Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a
speculative plan for a flat-file / DuckDB rewrite with explicit
trigger conditions for when to pick it up (partitioning deferred
too; revisit only if these three don't solve it).
A full f00-f18 sweep takes ~170 min of wall time, longer than the
~90 min gap between deploys on this cluster. Over the last 7 days
every run was killed mid-sweep and zero runs completed. The
propagation_scores table only ever held the scraps from partial
runs that landed before the pod died, which is why the map looked
like it was showing "the current hour" (or nothing).
The worker now processes exactly one forecast hour per perform/1
(~8-10 min) and enqueues the next hour as a fresh Oban job. A cron
fire with empty args seeds the chain at f00; subsequent runs carry
run_time + forecast_hour args. At f18 the chain stops and the
pruner runs. Each step broadcasts propagation:updated immediately
so new hours appear on the map as they land.
Wall time per perform drops from ~170 min to ~10 min, so Lifeline's
2h rescue window is no longer a factor and a pod restart loses at
most one forecast hour instead of the whole sweep. Retries and
max_attempts now describe a single fh, not the whole chain.
Also: bump the :propagation queue from 1 to 2 slots so
PropagationPruneWorker can run alongside the chain job, drop the
worker timeout/1 from 90 min to 20 min to match single-fh runs,
and pull Lifeline rescue_after from 120 min to 45 min to keep the
safety net above the step timeout.
Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the
pipeline chip in both the mobile and desktop sidebars so the user
can tell which valid_time the map is showing when the bottom
timeline isn't visible.
Era5SubmitWorker was counting era5_cds_jobs rows 1:1 against the CDS
150-job ceiling, but each row holds TWO CDS job IDs (single-level +
pressure-level). Prod hit 74 rows = 148 in-flight jobs and got stuck
in a reject→resubmit spiral. The cap guard now multiplies row count
by 2 to match what CDS sees.
Era5PollWorker @stuck_after_seconds goes from 4h to 18h. CDS
single-level routinely sits at 'accepted' for 12+h under load while
pressure finishes in ~90 min; the 4h threshold was firing on normal
slow runs and feeding the same spiral.
PipelineStatus now returns `label` + `detail` separately so the map
chip can render "Updating propagation" on one line and the
sub-detail ("HRRR run" / "ASOS nudge" / "+3h" / "now") on a second
line below it. running_label/1 is renamed to running_detail/1 and
returns just the sub-part.
