Two related fixes so the main map reliably picks up new binary
propagation score files as soon as PropagationGridWorker writes them.
1. Propagation.available_valid_times/1 previously preferred ScoreCache
over ScoresFile, using the cache as an index of what was available.
The cache is a lazy ETS of whatever hours have been fetched or
broadcast, which is a strict subset of what's on disk. A new
forecast hour landing on disk while the cache was warm with older
entries was invisible to the timeline until the cache happened to
catch up. Read directly from ScoresFile so the disk store is the
source of truth.
2. Add Propagation.scores_at_fresh/3 that always reads the .ntms file
and overwrites the cache entry, and use it from MapLive's
propagation_updated handler. PropagationGridWorker publishes the
cache_refresh on `propagation:cache` and the timeline ping on
`propagation:updated` as separate PubSub broadcasts, so by the time
MapLive runs through scores_at the ScoreCache GenServer may not
have processed the refresh yet — fetch_bounds then returns the
previous chain's bytes. scores_at_fresh takes disk as the source
of truth for the refresh path and warms the cache as a side effect
so subsequent readers see the new data.
Historical contacts showed a skew-T log-P diagram that stopped at 700 mb
because HrrrClient and Era5Client only fetched pressure-level data down
to the top of the boundary layer. The chart canvas already ran up to
100 mb, so the trace clipped mid-atmosphere.
Two complementary fixes:
1. Extend @pressure_levels in HrrrClient and Era5Client with
650/600/550/500/450/400/350/300/250/200/150/100 mb so new fetches
cover the full troposphere + lower stratosphere.
2. Prefer the native hybrid-sigma profile for the contact-detail
skew-T when one has been backfilled for the contact's hour. The
native profile already stores all 50 hybrid levels up to ~19 km,
so historical contacts covered by the native backfill get a full
trace without re-hitting S3. A new HrrrNativeProfile.to_skew_t_profile/1
converts the parallel arrays into the %{"pres","tmpc","dwpc","hght"}
list shape the renderer expects, deriving dewpoint from SPFH via the
Magnus inverse. Weather.find_nearest_native_profile/3 mirrors
find_nearest_hrrr/3 for the lookup.
Previously RangeEstimate.estimate/1 ran on every beacon page mount,
even though the cells data was only rendered after the user flipped
the coverage toggle — a wasted bbox grid pass per page load. Start
with estimate: nil, compute lazily in handle_event("toggle_coverage")
when flipping on, and push the cells payload to the Leaflet hook via
a new load_coverage event. The JS hook no longer reads data-cells on
mount and only builds the cellLayer when the server sends it. Cached
in the socket so re-toggling is instant.
For a 1 W / 432 MHz beacon, RangeEstimate.estimate/1 was returning
~126 k cells because free-space loss dominates over atmospheric
loss at low UHF, letting the rx_dbm filter pass almost the whole
bbox. Jason-encoding that payload into the map's data-cells
attribute was locking up browsers on the beacon detail page for
low-band beacons even though the coverage toggle is already
disabled for them. Return an empty estimate below the same 5760
MHz threshold the UI already enforces.
Render the HRRR pressure-level profile as a full skew-T log-P
diagram on the contact detail page: skewed isotherms, isobars,
dry adiabats, saturation mixing ratio lines, plus the T and Td
traces. Math lives in MicrowavepropWeb.SkewT (Magnus formula,
dry adiabat potential temperature, log-P projection) and is
exercised by a dedicated test module. The atmospheric profile
section now expands by default so the chart is visible without
an extra click.
Extend the f00 profile persistence to cover every forecast hour
(f00-f18) so /weather keeps working across pod restarts and shows
forecast-hour atmospheric data without a fresh database. Route the
Weather cold path (latest_grid_valid_time, load_weather_grid,
weather_point_detail) through ProfilesFile first, with the legacy
hrrr_profiles table as a last-resort historical fallback. Warm
GridCache from the latest profile on app start so /weather is hot
the moment a pod boots.
The propagation_scores → binary files cutover dropped the factors
JSONB column, which left point_detail returning factors: %{} and
broke the analysis breakdown popup on map clicks. Persist the
enriched f00 grid_data to /data/scores/profiles/{iso}.etf.gz and
rescore on demand at click time so the factor block renders again.
Flip the cron from every-3-hours to every-hour now that a full
f00-f18 chain runs in ~45-60 min instead of ~170 min. With the
:propagation queue's concurrency-of-2, a slow chain won't block
the next one — they interleave, and the file store's last-writer
-wins semantics give the newer run_time's analysis data naturally.
Add ScoresFile.retain_window/2 that deletes any file whose
valid_time falls outside [run_time, run_time + 18h]. Call it at
fh=18 of the chain so leftover files from the previous chain
(specifically the old f00 whose valid_time the new chain doesn't
overwrite because it advanced by one hour) are discarded
immediately instead of waiting for the 2h prune cron.
The step transition path is extracted to handle_step_transition/2
to keep the nesting under credo's limit and make the final-step
logic easier to read.
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:
1. New migration drops the propagation_scores table and its indexes
(the earlier tuning migrations for it were already applied and
are now no-ops against a missing table, which is fine — Ecto
just runs them on fresh environments).
2. Propagation context is gutted of every GridScore reference.
replace_scores/2 writes files only. upsert_scores/2 is deleted.
load_scores_from_db, available_valid_times_from_db,
point_detail_from_db, point_forecast_from_db, fetch_factors,
coalesce_factors, the Postgres side of prune_old_scores, and
the Postgres fallbacks in latest/earliest_valid_time are all
removed. point_detail always returns an empty factors map now
since factor breakdowns were retired with the table.
3. Deleted modules:
- Propagation.GridScore (the schema)
- Propagation.ScorerDiff (read factors from the table)
- Propagation.AsosNudge (helper for AsosAdjustmentWorker)
- Workers.AsosAdjustmentWorker (its cron was already disabled)
- Mix.Tasks.ScorerDiff (wrapper around the deleted module)
And their tests. AdminTaskWorker's scorer_diff task is a
logging no-op so any queued Oban rows drain cleanly.
Release.scorer_diff stays as a stub that tells the operator.
propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
PropagationGridWorker.merge_commercial_link_data/2 called
Commercial.link_degradation_at twice per grid cell (once to count,
once to merge), and each call re-ran enabled_links plus two sample
queries per in-range link. That was ~500k SQL queries per forecast
hour — fine for the DB but catastrophic for log volume in dev
iex sessions trying to watch a chain step run.
Split into two new functions:
* build_link_lookup/2 does all the DB work up front — one
enabled_links query + one link_degradation per enabled link
(~10 queries total).
* link_degradation_from_lookup/3 is pure: takes a (lat, lon)
and the precomputed lookup, returns the aggregate or nil.
The worker now calls build_link_lookup once per forecast hour and
the per-cell path is haversine-only. Net: 500k queries → ~10.
PipelineStatus freshness detection moves off oban_jobs.completed_at
onto ScoresFile.latest_valid_time(). The on-disk files ARE the data
the map renders from, so the chip's "Up to date · Nm ago" and the
2h stale threshold now track actual data presence. Running-state
detection still comes from oban_jobs since the chain step is still
an Oban row. Tests updated to seed a ScoresFile instead of a
completed Oban row for the idle/stale cases.
Read-side cutover for the binary scores store and a companion
cleanup that removes the biggest remaining DB write from the hot
path.
Propagation.scores_at/3, available_valid_times/1, latest_valid_time/0,
latest_valid_time/1, earliest_valid_time/1, point_detail/4, and
point_forecast/3 all now prefer ScoresFile and fall back to the
propagation_scores table when a file is missing. The map render
path reads from /data/scores first; Postgres stays as a safety
net while dual-write is on. point_detail still pulls factors from
Postgres (analysis-hour rows only) and coalesces nil to an empty
map so the JS popup iterates cleanly.
replace_scores/2 is now gated by a postgres_writes_enabled? flag
(runtime env MICROWAVEPROP_SCORES_POSTGRES=false, or the
:propagation_scores_postgres app env key) so the binary-only path
can be benchmarked locally without the DB insert. Default stays
true.
PropagationGridWorker no longer calls store_hrrr_profiles —
persisting 92k grid rows × 19 forecast hours of JSONB profiles
was ~12 min of wall time per chain for a table only
AsosAdjustmentWorker read from. Per-contact HRRR enrichment
through HrrrFetchWorker still writes its own (is_grid_point:
false) rows. AsosAdjustmentWorker is disabled in all three cron
configs since its data source is gone.
DataCase resets the scores tree between tests so per-test
ScoresFile writes don't leak across cases, and ScoresFileTest
switches to async: false because it mutates the global
:propagation_scores_dir env.
First step of the disk-backed scores migration from the DuckDB plan
doc. Ended up shipping the raw-binary variant instead of Parquet
because the data is disposable after ~2h — the ecosystem benefits
of Parquet only pay off for long-lived datasets, and the binary
path has zero new dependencies.
Microwaveprop.Propagation.ScoresFile writes one file per
(band_mhz, valid_time) tuple under the configured scores dir,
default /data/scores in prod and priv/dev_scores in dev. Layout is
a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is
the no-data sentinel). The whole CONUS grid serializes to ~93 KB
per band, and writes use the temp-then-rename pattern so NFSv4
concurrent readers never see a partial file.
Propagation.replace_scores/2 now materializes the score stream
once and dual-writes: Postgres on the primary path, then one
ScoresFile per band as a best-effort follow-up (any file error is
logged but doesn't fail the DB write, so we can verify the file
path in prod before cutting readers over).
Propagation.prune_old_scores/0 also clears expired score files so
the existing 15-minute prune cron covers both storage layers.
Dev configuration points at priv/dev_scores/, added to .gitignore.
Test configuration points at a per-run tmp directory.
The :propagation Oban queue runs with concurrency 2, so the grid
worker chain step and the 10-minute ASOS nudge can (and do) execute
in parallel. PipelineStatus.current/0 used to return only the
most-recently-started worker, which made the chip label flip to
"ASOS nudge" at every :00 / :10 tick even though HRRR was still
chugging through forecast hours behind it.
Replace the single :detail field with a list of running_detail
maps, each tagged with :grid or :asos so the chip can:
- Render one sub-line per currently-executing worker
- Sort them deterministically (grid before asos) so the eye
doesn't see rows swap
- Still override the grid entry with the forecast-hour progress
label when a propagation_pipeline_progress broadcast arrives
The scoring+upsert phase was ~4m40s per forecast hour and dominated
wall time. Three stacked optimizations attack it from different
angles.
replace_scores/2 is a new hot-path writer that does DELETE WHERE
valid_time = $1 followed by a plain insert_all (no ON CONFLICT
resolution). The chain worker rewrites the full (valid_time, all
bands) slice every forecast hour, so conflict detection was pure
waste. AsosAdjustmentWorker still uses upsert_scores because it
only rewrites the subset of cells near a station.
factors is now nullable. Forecast hours f01-f18 pass factors: nil
so the JSONB encode + toast write is skipped entirely — roughly
halves the data volume per run. point_detail/4 coalesces nil to
an empty map so the JS popup renders without a TypeError, and
scorer_diff only pulls the most recent valid_time that still has
factors (the f00 row).
propagation_scores is now UNLOGGED, so inserts bypass WAL entirely.
Durability tradeoff: an unclean shutdown truncates the table, but
PropagationGridWorker rebuilds it from HRRR every 3h so a lost
table is re-populated within one cron cycle.
Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a
speculative plan for a flat-file / DuckDB rewrite with explicit
trigger conditions for when to pick it up (partitioning deferred
too; revisit only if these three don't solve it).
A full f00-f18 sweep takes ~170 min of wall time, longer than the
~90 min gap between deploys on this cluster. Over the last 7 days
every run was killed mid-sweep and zero runs completed. The
propagation_scores table only ever held the scraps from partial
runs that landed before the pod died, which is why the map looked
like it was showing "the current hour" (or nothing).
The worker now processes exactly one forecast hour per perform/1
(~8-10 min) and enqueues the next hour as a fresh Oban job. A cron
fire with empty args seeds the chain at f00; subsequent runs carry
run_time + forecast_hour args. At f18 the chain stops and the
pruner runs. Each step broadcasts propagation:updated immediately
so new hours appear on the map as they land.
Wall time per perform drops from ~170 min to ~10 min, so Lifeline's
2h rescue window is no longer a factor and a pod restart loses at
most one forecast hour instead of the whole sweep. Retries and
max_attempts now describe a single fh, not the whole chain.
Also: bump the :propagation queue from 1 to 2 slots so
PropagationPruneWorker can run alongside the chain job, drop the
worker timeout/1 from 90 min to 20 min to match single-fh runs,
and pull Lifeline rescue_after from 120 min to 45 min to keep the
safety net above the step timeout.
Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the
pipeline chip in both the mobile and desktop sidebars so the user
can tell which valid_time the map is showing when the bottom
timeline isn't visible.
Era5SubmitWorker was counting era5_cds_jobs rows 1:1 against the CDS
150-job ceiling, but each row holds TWO CDS job IDs (single-level +
pressure-level). Prod hit 74 rows = 148 in-flight jobs and got stuck
in a reject→resubmit spiral. The cap guard now multiplies row count
by 2 to match what CDS sees.
Era5PollWorker @stuck_after_seconds goes from 4h to 18h. CDS
single-level routinely sits at 'accepted' for 12+h under load while
pressure finishes in ~90 min; the 4h threshold was firing on normal
slow runs and feeding the same spiral.
PipelineStatus now returns `label` + `detail` separately so the map
chip can render "Updating propagation" on one line and the
sub-detail ("HRRR run" / "ASOS nudge" / "+3h" / "now") on a second
line below it. running_label/1 is renamed to running_detail/1 and
returns just the sub-part.
PropagationGridWorker now broadcasts a {:propagation_pipeline_progress,
%{forecast_hour:, valid_time:}} message on the propagation:pipeline
PubSub topic at the start of each process_forecast_hour/4 call.
MapLive subscribes on mount, stashes the last message in the new
:pipeline_progress assign, and clears it on the refresh tick whenever
PipelineStatus flips out of :running.
The chip component consults a new PipelineStatus.running_label/1
helper that formats the payload as "Updating propagation · now" for
f00 and "Updating propagation · +Nh" for f01-f18, falling back to
the base running label until the first progress message arrives.
Covered by unit tests on the formatter and an integration test in
MapLiveTest that seeds an executing grid job, broadcasts progress,
and asserts the rendered chip.
AsosAdjustmentWorker fires every 10 minutes and loads every row of
`hrrr_profiles` on the grid for the latest valid_time. The old query
selected `profile: h.profile` — ~1.3 KB of JSONB × 92k grid points ≈
120 MB of JSONB per tick. Postgrex's Jason.decode! ran inline for
each row and blew past the 15 s pool checkout window, so every tick
was killing connections with:
DBConnection.ConnectionError: client timed out because it queued
and checked out the connection for longer than 15000ms
`score_grid_point/4` only touched the profile array to re-derive
`min_refractivity_gradient`, but `hrrr_profiles` already persists
that value as a scalar column at ingestion time. Teach
`derive_from_hrrr/1` to honour the persisted scalar when it's
present and drop `h.profile` from the worker's SELECT list. Net
effect: same score math, ~1% of the JSONB transfer, tick stays
under the pool deadline.
Covered by a new scorer test that feeds a profile map with no
`:profile` list and asserts the refractivity factor still reflects
the persisted gradient instead of the neutral baseline.
Adds a small chip below the NTMS title in both the mobile and
desktop sidebars of /map that collapses PropagationGridWorker +
AsosAdjustmentWorker Oban state into one of four states —
running / idle / stale / unknown — with a plain-language label
("Updating propagation (HRRR run)…", "Up to date · 12m ago").
The chip refreshes on a 15s timer and whenever the pipeline
broadcasts propagation:updated, so a long HRRR sweep is visible
from the moment a visitor opens the page.
Observed in prod: after the CDS cap guard shipped, jobs started
landing but one leg per tile-month would routinely stay in 'accepted'
state for 16+ hours while the other completed in ~90 minutes. CDS's
per-user queue fairness apparently serializes competing leg requests
and the single-level ones got stranded behind higher-priority work.
Add an age-based stuck detector in Era5PollWorker: if a row has been
submitted for more than 4 hours and neither leg is fully done, treat
both legs as terminal, clean them up from CDS, and re-enqueue the
submit. Re-uses the resubmit_vanished path. 4h is ~8× the normal
completion time so transient queue variance doesn't trip it, but
tight enough that stuck jobs self-heal in a working session.
CDS returns status="rejected" (distinct from "failed") when a submit
is rate-limited past the 150-job per-user cap. The poll worker's
interpret_status_body treated this as an unknown status → generic
retryable error → 10 retries → discarded. 102 jobs hit this path
overnight.
Fix:
* Era5Client.check_status/1 now returns {:rejected, reason} (and
treats "dismissed" the same way — that's the status a manually
deleted job transitions to).
* Era5PollWorker handles :rejected exactly like :not_found: drop the
DB row, clean up the sibling CDS job, re-enqueue the submit, and
discard the current Oban job so it doesn't burn attempts retrying
a terminal state.
* Widened Era5SubmitWorker cap headroom from 10 → 30 (effective
ceiling 120 not 140). The race between concurrent workers all
passing the count check simultaneously meant we were reaching 141
in-flight against the 150 hard cap; 30 slots of headroom tolerates
a ~15-worker race before clipping CDS's ceiling.
Refactored handle_row/1 into handle_row + handle_non_both_done with a
leg_outcome/1 tag helper so the cross-product of leg statuses collapses
to a single pair match (credo complexity limit).
CDS enforces a per-user cap of 150 queued jobs. Each successful submit
adds 2 rows to era5_cds_jobs (single-level + pressure-level), so when
the current in-flight count + 2 would exceed (150 - 10 headroom), the
worker snoozes for 5 minutes instead of POSTing to CDS. Stops 'Number
of queued requests is limited to 150' rejects from burning Oban
attempts.
Contacts whose weather/hrrr/terrain/iemre status has been :queued for
more than 3 days (144 cron cycles) are flipped to :unavailable. The
set_enrichment_status!/3 helper is a no-op when the status is unchanged,
so a stuck row retains a stale updated_at — which is exactly the signal
we need for 'the cron has already tried and the data doesnt exist
upstream' (dead ASOS station, pre-2014 HRRR, out-of-grid IEMRE). Clears
~1,492 pre-existing stuck contacts and stops them from churning the
backfill queue.
When CDS returns 404 for a poll check, the job is gone (manually
deleted, reaped, or never existed). Retrying will always 404, so treat
it as terminal: drop the era5_cds_jobs row, delete the sibling CDS job,
re-enqueue Era5SubmitWorker for the tile-month, and discard the Oban
poll job. Fixes 30 retryable poll jobs left behind by the interrupted
manual-cleanup script.
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Contact detail page now renders an atmospheric profile section backed by
HRRR when available and ERA5 (reanalysis) otherwise, labeled with the
data source. When both sources are missing and HRRR is known-unavailable
(pre-2014 contacts), the Era5FetchWorker is enqueued so the month-tile
backfill gets triggered from a user visit.
Status page ERA5 progress previously computed "complete" as
(total_contacts - hrrr_unavailable_contacts), which treated every contact
with any HRRR status as ERA5-complete — inflating the bar to 76% while
era5_profiles had 0 rows. Replaced with a per-candidate EXISTS query
against era5_profiles using the same 0.15°/30-min tolerance
Weather.find_nearest_era5/3 uses at lookup time.
Three signal sources we already collect but weren't using:
* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
as max of HRRR-derived and NEXRAD-derived rate so fast convective
cells between HRRR hourly analyses can still trigger the rain penalty.
Only active on f00 — forecast hours can't see future radar. New
Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
hail-safe ceiling.
* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
applies a 1.15× boost when the cell's native-resolution duct supports
the target band's frequency. HRRR pressure-level gradients
systematically under-read thin trapping layers the native profile can
resolve. Sub-band ducts do NOT boost — they're evidence that the
gradient we have is all there is at the target frequency.
* Commercial LOS link rx_power fading → inverse tropo sensor.
Commercial.link_degradation_at/3 computes the average 7-day-baseline
vs current delta across enabled links within 75 km, ignoring links
where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
to the composite score for 3+ dB of fading. ~150 km radius around
DFW is the only zone this helps today, but it's the first *measured*
signal in the algorithm vs the model-derived proxies.
Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.
1,359 tests, 0 failures.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
IEM's RAOB endpoint stopped keeping Canadian stations current — most
stalled around Sep 2024 — which leaves a gap in refractivity/ducting
calibration over all the CW*/CY* stations already in weather_stations.
MSC's own tephi CSV is fixed-width and rendering-focused; the WMO TEMP
bulletins would need a full decoder. University of Wyoming serves the
same stations in a clean space-delimited format, is current, and
accepts the 3-letter station code (drop the leading C from Canadian
ICAO). Its output shape matches IemClient.parse_raob_json/1 exactly,
so it drops straight into Weather.upsert_sounding/2.
- New UwyoSoundingClient with URL builder, Req-based fetch_sounding,
and a fixed-width parse_sounding_html that extracts PRES/HGHT/TEMP/
DWPT/DRCT/SKNT from the <PRE> block and DateTime from the <H2> header.
- New CanadianSoundingFetchWorker: Oban batch worker that finds all
sounding stations whose code starts with C, picks the most recently
publishable 00Z or 12Z slot (90 min publish delay), calls UWYO per
station, and upserts with SoundingParams-derived refractivity/
ducting fields.
- Oban cron at 01:30Z and 13:30Z (dev + prod) to drive the worker.
- Req.Test plug wiring in config/test.exs.
- Captured Goose Bay fixture as test/support/fixtures/uwyo_sounding_yyr.html.
Also drops two implementation plans under docs/plans/:
- 2026-04-13-rdps-vertical-profiles.md: ~2 day plan for RDPS 10 km
Canadian ducting outside HRRR's Lambert footprint.
- 2026-04-13-hrdps-canadian-prop-grid.md: ~5 day plan for the full
HRDPS 2.5 km Canadian prop grid. Explicitly recommends doing RDPS
first.
TDD throughout, 19 new tests, 1318/1318 passing, credo strict clean.
warm_grid_cache_and_broadcast was re-SELECTing 92k hrrr_profiles rows
with JSONB profile/duct_characteristics columns on every forecast hour.
Row decoding exceeded the 15s default DB connection timeout, killing
the worker before f01-f18 could run. Oban retried from f00 and got
stuck in a loop: for the last several hours, no forecast hours were
being written and even f00 was stale.
Build the weather cache rows directly from the in-memory grid_data
the worker already has (Weather.build_grid_cache_rows/2) and
GridCache.broadcast_put directly. No DB round trip, no JSONB decode.
Also widen the cron from hourly to every 3 hours so a full f00-f18
sweep (~95 min) can actually complete before the next run starts,
and drop the redundant prune_old_scores() at worker start
(PropagationPruneWorker already runs every 15 min). Add a 60s query
timeout to load_weather_grid_from_db as defense-in-depth for the
cold-cache fill path that still uses it.
Instead of shelling out to https://gridmap.org/locate/:callsign for
callsign → lat/lon lookups, ports the resolver pipeline from gridmap-web
into this project so the whole flow runs in-process.
New modules:
- Microwaveprop.Qrz — cache facade over the QRZ.com XML callsign API.
Looks up from qrz_callsigns first, falls back to a live fetch, and
upserts the result with a configurable cache_ttl_hours (default
168h / 7 days).
- Microwaveprop.Qrz.Client — HTTP/XML client against
https://xmldata.qrz.com/xml/current/. Holds the session key in an
Agent, transparently re-logs-in on :session_expired, and parses
responses via xmerl.
- Microwaveprop.Qrz.Callsign — Ecto schema for the qrz_callsigns
cache table, binary_id primary key per project convention.
- Microwaveprop.Qrz.Record — slim struct with only the 11 fields
we actually consume (identity, name, grid, address, lat/lon).
The full XML payload stays in the raw :data jsonb column for
anyone who wants the other ~40 QRZ fields.
- Microwaveprop.Geocoder — Req-based client against the Google Maps
Geocoding API. Only called as a fallback when QRZ has no explicit
<lat>/<lon> for the callsign.
- Microwaveprop.CallsignLocation — orchestrator. Reads the
callsign_locations cache, on miss calls Qrz then either uses QRZ's
coords directly or geocodes the formatted address, snaps to an
8-char Maidenhead grid via Microwaveprop.Radio.Maidenhead, and
upserts the result.
Microwaveprop.Radio.CallsignClient.locate/1 is rewritten to delegate
to CallsignLocation.lookup/1 and shape the response back to the
existing {:ok, %{callsign, gridsquare, lat, lon}} contract so the
callers in PathLive and RoverLive don't change.
Wiring:
- priv/repo/migrations/20260413000000_create_qrz_callsigns_and_callsign_locations.exs
creates qrz_callsigns and callsign_locations with unique indexes
on :callsign.
- Microwaveprop.Qrz.Client added to the application supervision tree
so the session Agent is started.
- :xmerl added to extra_applications so the release bundles it.
- config/test.exs wires Req.Test plugs for Microwaveprop.Qrz.Client
and Microwaveprop.Geocoder, forces cache_ttl_hours: 0 so the cache
never short-circuits test-level stubs, and supplies dummy QRZ
credentials.
- config/runtime.exs pulls QRZ_USERNAME / QRZ_PASSWORD / QRZ_AGENT
and GOOGLE_API_KEY from the environment so prod and dev can
configure both upstream keys out of band.
Tests (ported verbatim from gridmap-web):
- test/microwaveprop/qrz/callsign_test.exs — schema/changeset
- test/microwaveprop/qrz_test.exs — cache hit, cache miss, TTL
behavior, upsert on stale, error passthrough, case-insensitive
input
- test/microwaveprop/geocoder_test.exs — success, zero results,
request denied, transport error
- test/microwaveprop/callsign_location_test.exs — end-to-end flow
including the QRZ-lat/lon shortcut and the missing-address error
path
All 1294 tests still pass. Credo strict clean.
Replaces the hand-rolled sort/search/pagination on /contacts with
LiveTable.LiveResource. Sortable columns for station1/2, grid1/2,
band, mode, distance_km, qso_timestamp, and inserted_at; searchable
across both stations, both grids, and mode. Custom renderers preserve
the enrichment summary badge (with per-status tooltip), the invalid
flag column, and the qso/inserted_at formatters.
The reciprocal-grouping block has been dropped per the brainstorming
decision — every QSO now shows as its own row, which avoids fighting
live_table's one-row-per-record model. A "View" action links to the
detail page instead of the previous row-click handler (live_table's
stream dom_ids are random UUIDs so row-click by record id is no
longer practical without forking the library).
Test updates:
- sort test switched to ?sort_params[station1]=asc (live_table URL
format; the old ?sort_by/?sort_order params are gone)
- pair-search test removed since live_table does a single ILIKE
across searchable fields and no longer supports the two-callsign
intersection that Radio.list_contacts/1 used to do; single-term
search is exercised instead
- pagination test asserts the presence of "Page" (live_table's
pagination text is "Page N" without the total count)
Also converts the stray cond-with-single-clause in
UserManagementLive.Index's delete handler to if/else (credo fix).
Wire up gurujada/live_table 0.4.1 as a dependency (alongside its
transitive sutra_ui and optional igniter requirement) with the
:live_table config pointing at our Repo and PubSub. Forced oban_web
override so the path-pinned vendored copy still wins over live_table's
hex constraint.
Convert the admin /users page to use LiveTable.LiveResource with
sortable, searchable columns for callsign, name, and email. Custom
renderers preserve the daisyUI admin badge and the pending/confirmed
date cell. Hidden id field stays in the query so the edit and delete
actions can target a record by id.
Updated the delete test to push the delete event directly with an id
payload; the stream dom_ids under live_table are random UUIDs so
tr#users-<id> selectors no longer work.
The profile page's involving-contacts query now limits to the 100
most recent rows (ordered by qso_timestamp desc). A small note is
shown under the heading when the cap is hit so users know there's
more data they're not seeing.
Add a new card on the profile page listing every contact where the
callsign appears as either station1 or station2, case-insensitive,
newest first. Backed by a new `Radio.list_contacts_involving_callsign/1`
query with unit coverage for station1/station2 matching, case handling,
ordering, and blank input.
Profile page:
* New MicrowavepropWeb.UserProfileLive at /u/:callsign is a public
page showing a user's contacts and beacons. Resolves case-
insensitively so /u/w5isp and /u/W5ISP are the same thing; unknown
callsigns redirect to /. Uses daisyUI card / stats / table
components with an avatar-placeholder initial and hero icons.
* Accounts.get_user_by_callsign/1 (case-insensitive) plus
Radio.list_contacts_for_user/1 and Beacons.list_beacons_for_user/1
back the page. Beacons list includes both approved and pending so
owners see their drafts.
* The top nav bar and the three LiveView sidebars (MapLive,
WeatherMapLive, ContactMapLive) now render the logged-in callsign
as a navigate link to /u/:callsign instead of a static label.
* Nine new tests cover the lookup, the LiveView render, and the
ownership-scoped queries.
Flexible band input:
* New Microwaveprop.Radio.BandResolver module converts any of:
ADIF wavelength labels ("33cm", "1.25cm", "6mm", case/whitespace
insensitive), numeric frequency strings ("903.100", "10368.000"),
and canonical MHz integers into the one of the site's known bands.
Returns the nearest allowed band for numeric inputs >= 900 MHz,
nil otherwise.
* 902 MHz is added to Contact.@allowed_bands, ContactEdit.@allowed_bands,
AdifImport.@allowed_bands, and the BandResolver list so "33cm"
round-trips end-to-end.
* AdifImport and CsvImport now delegate band resolution to
BandResolver, and Radio.create_contact/2 normalizes the :band attr
on the way in so the manual form and any API callers benefit too.
CsvImport's "invalid band" tests previously used 99999 MHz which
the new resolver snaps to the nearest allowed band; swapped to
"notaband" which is truly unresolvable.
Contacts and beacons list UX:
* Remove the "Submitted" column from /contacts — it duplicated info
already visible on the detail page and was pushing the real
columns off narrow viewports. submitted_cell/1 and its three
column-specific tests go with it.
* Hide the Lat / Lon columns from /beacons — six decimal places of
coordinates weren't useful next to the grid square and took a
disproportionate amount of row width.
Submit and contact-detail forms now use a plain text input for the QSO
timestamp with a "YYYY-MM-DD HH:MM" placeholder and regex pattern. The
native datetime-local input ignores lang="en-GB" on macOS/iOS when the
system clock is 12-hour, so it was rendering AM/PM against the user's
wishes. Ecto's :utc_datetime cast already accepts this format (verified
space+seconds, T+seconds, and both with and without seconds/Z).
Contact edit workflow grows a third branch: admins still apply directly,
non-owners still go through the admin review queue, and now logged-in
owners of a contact (user_id == current_user.id) can apply edits
directly too. New helpers Radio.owner?/2 and Radio.apply_owner_edit/3
reuse the existing normalize + diff + apply_edit_to_contact path so the
grid-change → enrichment re-enqueue pipeline kicks in automatically.
Anonymous contacts (user_id nil) and other users' contacts both return
{:error, :not_owner}.
Nine new tests cover owner?/2 and apply_owner_edit/3 including the
no-changes, wrong-user, and anonymous-contact paths.
Regex.scan with `return: :index` reports BYTE offsets, but String.slice/3
uses CHARACTER offsets. When a record contained a multi-byte UTF-8
character in an earlier field (e.g. "café" in NOTES), every subsequent
field's slice shifted one byte left, eventually landing on a literal
">" that crashed String.to_integer/1 with ArgumentError.
Switch to :binary.part/3 which is byte-based and matches what Regex.scan
reports. Also simplify the value offset calculation to use the scanned
tag_len directly instead of re-splitting the tag text. Reproduced in a
new regression test.
Prod stack trace: AdifImport.parse_fields/1 crashed on upload with
binary_to_integer(">") -- a FreeDV/N1MM logger that embedded UTF-8
characters in a NOTES field triggered it.
The three BackfillEnqueueWorkerTest cases were timing out because the
default types list includes :era5, which cascades through inline Oban
into Era5Client.poll_and_download where Process.sleep blocks for the
full 60s test timeout. Era5Client uses bare Req.get with no plug hook,
so it can't be stubbed via Req.Test the way the other clients are. Pass
explicit non-ERA5 types in the three affected cases — ERA5 has its own
coverage and these tests don't assert anything era5-specific.
Also replace two `length(results) > 0` checks in asos_nudge_test with
`results != []` to silence credo warnings.
MRMS
----
Layer the NOAA MRMS PrecipRate product onto the score grid so rain fade
updates every 2 minutes instead of every hour alongside HRRR. New modules:
- Microwaveprop.Weather.MrmsClient: fetches the latest .grib2.gz off the
NCEP mirror (Req auto-decompresses so no gunzip step), writes the raw
GRIB2 to a temp file, and calls the existing wgrib2 wrapper with the
0.125 propagation grid spec to get interpolated cells. Returns a
%{{lat, lon} => mm_per_hour} map with missing-value sentinels dropped.
- Microwaveprop.Weather.MrmsCache: ETS-backed GenServer mirroring
ScoreCache/GridCache. Caches a single "current" entry keyed by
valid_time with PubSub broadcast so peer nodes stay in sync and only
the Oban leader pays the fetch + regrid cost.
- Microwaveprop.Workers.MrmsFetchWorker: cron every 2 minutes, short-
circuits when the cached valid_time already matches the newest file.
Microwaveprop.Propagation.AsosNudge.compute/4 now takes an optional
rain_grid. When a cell has MRMS rain >= 0.1 mm/hr it gets patched onto
the HRRR profile's `precip_mm` field (the scorer already reads it there)
and the cell is re-scored even with no ASOS station nearby. Cells with
MRMS rain below the threshold aren't touched so dry cells keep their
raw HRRR scores (which have the wind/sky/native-gradient signal that
isn't persisted on HrrrProfile rows and would otherwise be lost).
AsosAdjustmentWorker pulls MrmsCache on every tick and passes the grid
through to AsosNudge.compute/4. Also skips the IemClient error branch
that can never happen and handles the ASOS-empty + MRMS-empty case
explicitly. MrmsCache wired into the supervision tree; MrmsFetchWorker
cron entry added to config.exs and dev.exs.
Four new AsosNudge cases cover MRMS-only re-scoring, threshold gating,
and the wet/dry score delta.
Beacons 500
-----------
Beacon.format_freq/1 and format_mw/1 crashed on whole-number floats
(e.g. 24192.0) because `frac == 0.0` could become false under float
rounding while `trim_trailing_zeros/1` stripped the decimal point,
leaving a 1-element list that couldn't be destructured as [_, frac].
Shared format_number/1 helper handles integer input directly and
pattern-matches both the "int-only" and "int + frac" shapes.
Added stream_data property tests covering the whole microwave range for
both integers and floats to catch this class of bug before prod.
UTC clock flash
---------------
The /weather and /map UTC clocks were empty until the JS hook mounted
post-WebSocket, producing a several-second blank spot on initial load
and a clobber risk on sidebar re-renders. Mount now computes a
server-rendered `initial_utc_clock` string and the template seeds the
element with that plus `phx-update="ignore"` so LiveView morphdom won't
overwrite what the hook writes.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
The /weather LiveView was timing out on pod restart because
latest_weather_grid/1 ran load_weather_grid_from_db/1 synchronously on
cache miss — that query reads 176k hrrr_profiles rows with two huge
JSONB columns (profile array + duct_characteristics), and JSONB
decoding blocked the LiveView process past the 15-second pool
ownership timeout.
On cache miss latest_weather_grid/1 now returns empty immediately and
kicks off a deduped background fill through GridCache.claim_fill/1 (a
per-valid_time ETS lock so N concurrent mounts after restart don't each
fire the slow query). When the fill completes it broadcasts
weather:updated, and the handle_info in WeatherMapLive hits the warm
cache and pushes rows over the socket.
Added load_weather_grid/1 as a synchronous sibling for tests and any
caller that genuinely needs inline data. Tests updated to use it and
to clear GridCache in setup so ETS state doesn't leak between cases.
Also untrack k8s/secret.yaml and add both it and k8s/*-secret.yaml to
.gitignore — those manifests hold plaintext SMTP and database
credentials that should not live in the repo. Secrets already in git
history should be rotated separately.
- AdifImport.normalize_mode/2 maps ADIF MODE (and SUBMODE when present)
onto the fixed set of modes the site accepts: CW/SSB/FM/FT8/FT4/Q65.
SUBMODE takes precedence so `MODE=MFSK, SUBMODE=FT8` correctly imports
as FT8. USB/LSB/DSB all become SSB; NBFM/WFM become FM.
- Unknown modes (RTTY, PSK31, etc.) map to nil so the row still imports
— mode is optional on the submission path now.
- AdifImport.mode_mapping_reference/0 returns the user-facing list of
recognized inputs and their mappings.
- Submit page's ADIF tab renders that list in a table so users can see
upfront what will happen to their file before uploading.
- Add Weather.GridCache: ETS cache of derived HRRR grid rows keyed by
valid_time, cluster-synced via PubSub. Eagerly warmed from
PropagationGridWorker after each upsert so /weather map pan/zoom and
weather_point_detail hit zero DB on warm cache.
- Replace latest_weather_grid DB query path with cache-first lookup +
DB fallback. hrrr_profiles is 42M rows partitioned; pulling 3-10k
rows per viewport on every pan was the main cost.
- ContactLive.Show: defer the heavy enrichment loads (weather, solar,
HRRR, terrain, IEMRE, elevation profile, ITU-R propagation analysis,
data_sources) into a handle_info(:hydrate) that runs after the shell
renders. Initial mount now returns nil placeholders; template already
had :if guards for all of them. Shell-to-first-paint goes from
~500ms-2s down to ~20ms.
- Cache fetch_queue_counts for 5s in ContactLive.Show — oban_jobs group
by query was running on every contact page view.
- Backfill stats: wrap count_unprocessed, fetch_stats, fetch_db_stats
in Microwaveprop.Cache with 2-5s TTLs; bump refresh debounce from 1s
to 2s so bulk enrichment events don't thrash the DB.
- Add contacts_inserted_at_desc_id_desc_idx so /contacts list runs as an
Index Scan (~0.4ms) instead of a Seq Scan + top-N heapsort (~77ms on
58k rows)
- Add Microwaveprop.Cache: tiny ETS-backed TTL cache for memoizing
expensive-but-stale-tolerant values
- Cache total_entries for unsearched /contacts page loads (30s TTL,
invalidated on insert)
- Cache the entire /contacts/map payload (pre-encoded JSON + band list),
moving load_contacts into Radio.contact_map_payload; invalidated on
new contact insert. Mount goes from ~150-300ms to ~5ms.
- Fix /map overlay blinking on load: reuse the Leaflet GridLayer across
renderScores calls and just redraw() against the updated ScoreGrid,
instead of removeLayer + new layer which flashed between tile regens
- Make mode optional on submission: schema change (null: true),
submission_changeset no longer requires it, blank strings normalise to
nil, CSV import accepts 6-column (no mode) or 7-column layouts
- core_components .input adds a red asterisk to labels when required,
giving users a visual cue for which fields must be filled on /submit
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
immediately with a pending marker; push rain_scatter_update when
NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
viewport after mount/band change/propagation_updated; client caches
them and renders timeline scrubs instantly without a server roundtrip.
Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
timeline click uses preloaded cache when available
- Add ScoreCache GenServer with node-local ETS table keyed by
{band, valid_time}, subscribed to "propagation:cache" PubSub topic so
every pod stays in sync with a single hourly compute
- scores_at/3 checks cache first, falls back to DB and populates on miss
- PropagationGridWorker warms and broadcasts the cache for each band
after every forecast hour upsert; prunes >2h old entries
- Replace per-pixel string-keyed Map with flat Int8Array over the CONUS
grid in propagation_map_hook.ts to eliminate allocations in the tile
rasterization hot loop (interpolateScore / propagationReach)
Aliases: add module aliases for 9 nested module references
Apply: replace apply/3 with direct module attribute calls
Line length: break 1 long spec line
Refactoring: extract helpers to reduce complexity and nesting
in show.ex, radio.ex, weather workers, terrain, duct detection,
backfill dashboard, contact map, and mix tasks
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
