- NexradCache: replace :ets.tab2list() with foldl to avoid ~1.3 GB heap
allocation on full cache (OOM risk)
- GridCache: replace :ets.select full-table copies with foldl in
ets_latest_valid_time and ets_prune_keep_latest
- Weather.backfill_hrrr_batch: batch N individual UPDATEs into single
VALUES-based SQL statement eliminating ~500 round-trips per batch
- Add 9 missing database indexes across 6 tables for hot-path queries
(contacts, contact_edits, hrrr_profiles, hrrr_native_profiles,
grid_tasks, rover_mission_paths, fixed_stations, beacons)
- grid.ex: memoize conus_points/0 with module attribute (was ~94k recompute per call)
- path_compute.ex: reduce 6 list traversals to single Enum.reduce pass
- radio.ex + application.ex: fix unused variable e and redundant case warnings
- priv: add 5 partial indexes on (qso_timestamp) for enrichment queries (weather, hrrr,
terrain, iemre, radar) matching the WHERE status IN ('pending','failed') AND pos1 IS NOT NULL
pattern used by contacts_needing_enrichment
- findings.md: mark all performance hotspots and corresponding items as fixed
POST /api/v1/beacon-monitor/measurements accepts one measurement per
integration window from a propmonitor client, authenticated by the
BeaconMonitor token. Records noise floor, signal peak/avg dBFS, SNR,
signal-active-fraction, gain, and frequency for later correlation
against weather and propagation scores.
Beacon UUID must resolve to an approved + on-the-air beacon (404
otherwise — the client drops 404s without retry). Retries with the
same (monitor, beacon, measured_at) are idempotent: a unique index
makes the second insert a 409. Every accepted upload stamps the
monitor's last_seen_at — measurements *are* the heartbeat.
MonitorAuth is a separate plug from the user API-token Auth plug;
monitors are not users. The rate limiter buckets each monitor by id so
retry storms after a 5xx don't burn the shared anon-IP bucket.
Adds bearer-token authenticated REST API at /api/v1 covering every
action a non-admin user can perform on the website: contact + beacon
submission, beacon-monitor management, propagation queries, profile
read/update, and self-service API token issuance/revocation.
Security: SHA-256-hashed bearer tokens (mwp_ prefix, plaintext shown
once at creation), RFC 9457 problem+json error responses, RFC 9651
RateLimit-* headers backed by an ETS bucket (600/min per token,
60/min per anonymous IP, 30/min on /auth/tokens), private-contact
filtering by viewer.
Docs at docs/api/README.md (prose reference) and docs/api/openapi.yaml
(OpenAPI 3.1 spec covering every endpoint, response, and schema).
Tests: 124 new tests across schema, plug, error renderer, rate
limiter, fallback, and every controller. 16/17 API modules at 100%
line coverage; FallbackController at 87.5% (one defmodule line, an
Erlang-cover artifact for action_fallback-only modules).
`Pskr.Recalibrator.run/0` reads `pskr_calibration_samples`, bins
each sample per (band × feature), and writes spot-density stats to
`pskr_feature_bins` so an operator can read whether a feature
actually discriminates propagation at the threshold granularity
the scorer uses.
Bins, not regression: `BandConfig` already encodes scoring as
discrete thresholds, so the bin output matches that shape and an
operator can copy adjusted thresholds directly without translating
from regression coefficients.
Self-healing: corpus too thin ⇒ run row written with status
`skipped_insufficient_data` and the analysis is a no-op until next
fire. `min_total_samples = 1000` (≈ 4-5 days of CONUS PSKR
activity); per-band threshold is 100. Both surface in the run row's
`notes`.
Auto-applies nothing. Weight changes still go through human review
of `BandConfig.@band_configs` and a code commit. The recalibrator
is a read-only analyst that stays out of the production scoring
path.
Features binned (matching the scorer's discriminating fields):
* pwat_mm — humidity U-shape candidate
* hpbl_m — boundary layer (mechanism vs scoring re-eval)
* min_refractivity_gradient — refractivity threshold validation
* surface_pressure_mb — pressure-front proxy
* kp_index — aurora boost magnitude tuning
Schema: two tables.
* `pskr_recalibration_runs` — one row per fire with corpus
stats, status, notes
* `pskr_feature_bins` — one row per (run, band, feature, bin)
with sample_count, spot_count_total/avg/p50/p90
Cron: `0 4 * * 0` (Sundays 04:00 UTC, off-peak, post-climatology).
Manual reruns enqueue with no args.
Tests cover the empty-corpus skip path, sub-threshold totals,
per-band threshold gating, the actual bin emission, nil-feature
handling, spot-count averaging, and the always-records-a-run
audit invariant. 8 new tests, 3282 total passing.
Backfill pipeline untouched.
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:
* `pskr_spots_hourly` — observed spot density (truth signal)
* `hrrr_profiles` nearest match at the cell midpoint at hour
boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
ducting flag)
* `geomagnetic_observations` latest Kp at the hour (space weather)
Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.
Components:
* Migration `20260504210756_create_pskr_calibration_samples` —
new table + unique index on (hour, band, midpoint_lat, lon),
plus a midpoint spatial index on `pskr_spots_hourly` to keep
the join cheap.
* `Pskr.CalibrationSample` — schema mirror of the table with
the same `(hour, band, midpoint_lat, lon)` unique constraint.
* `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
for the hour, snaps midpoints to the propagation grid (0.125°),
builds an in-memory HRRR index over a ±0.07°/±60 min window,
and bulk-upserts samples. Idempotent — re-runs upsert.
* `Workers.PskrCalibrationWorker` — Oban cron entry on the
`:backfill_enqueue` queue. Default args target the previous
full hour; explicit `hour_utc` arg reruns any hour.
* Cron `25 * * * *` — fires past HRRR analysis publish window
(~HH:50→HH:05) and PSKR aggregator's 60s flush.
Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.
Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).
Backfill pipeline untouched — none of these changes feed into
contact enrichment.
Both RTMA and METAR5 schemas + clients + workers were defined but
never had a consumer. `find_nearest_rtma/3` and `recent_surface_obs/3`
existed in `Microwaveprop.Weather` with zero callers cluster-wide;
RtmaFetchWorker had test coverage but was never enqueued anywhere;
no IEM 5-min ASOS fetcher was ever written. The tables sat empty in
prod and the recalibrate audit was permanently flagging them BROKEN
even though the empty state was the steady state.
Drop the lot — schemas, clients, workers, queue config, Req.Test
stubs, audit checks, and the per-table unique tests in the
observation-changesets suite. New migration drops the now-unreferenced
`rtma_observations` and `metar_5min_observations` tables (both 0
rows in prod). Trim the `:rtma` slot out of `backfill_only_queues`
in runtime.exs so Oban Pro's Smart engine stops trying to manage a
queue with no producer.
Audit thresholds reset on the surviving NARR check: drop the
"WARN < 5000 absolute rows" rule that didn't account for
NarrFetchWorker's 0.13° spatial dedup, and point remediation at
the existing BackfillEnqueueWorker cron instead of the dev-only
`mix narr.backfill` task.
scripts/recalibrate_algo.py: ROW_COUNT_SOURCES + DATA_GAP_SQL +
DATA_GAP_RULES all stop referencing the dropped tables.
Two related changes to the contact detail page:
1. The "Flagged invalid" badge was wrapping to two lines on
narrow widths because the daisyUI badge has no explicit
no-wrap class. Add `whitespace-nowrap`, drop the redundant
capitalization on "Invalid", and append the flagger's
callsign so the badge reads "Flagged invalid by W5XYZ".
Hover tooltip carries the timestamp.
2. New `flagged_by_user_id` + `flagged_at` columns on contacts.
`Radio.toggle_flagged_invalid!/2` now takes the admin User
doing the flagging; on unflag it clears both fields so a
subsequent re-flag gets a fresh attribution and the badge
never shows a stale flagger.
3. `Radio.delete_contact!/1` + a Delete button next to Flag.
Admin-only, gated in both the event handler and the template.
Uses LV's `data-confirm` for the destructive-action prompt.
All FK references to `contacts` already declare
`on_delete: :delete_all` (terrain_profiles, contact_edits,
contact_common_volume_radar) so a single Repo.delete!
cascades cleanly.
Subscribes to mqtt.pskreporter.info:1883 over plain TCP and folds
every reception report into hourly aggregates per
(band, sender_grid_4, receiver_grid_4). Each spot is the empirical
"propagation actually occurred on this path right now" signal we'll
correlate against HRRR atmospheric state to recalibrate the scoring
weights. Aggregating at the path-hour bucket collapses 50-reporter
QSOs to one row instead of 50.
Topic filter anchors on USA (DXCC 291) on either sender or receiver
side so the broker pre-filters out everything outside our HRRR
domain. Bands subscribed: VHF and up (6m, 2m, 70cm, 23cm, +
microwave) — HF is dominated by ionospheric propagation, which this
project doesn't model.
Pskr.Client cluster-elects a singleton via :global.register_name —
all replicas start the GenServer but only one connects to MQTT;
the rest stay in :standby and watch for the leader's nodedown to
re-run election. Off in dev/test, on by default in prod
(PSKR_MQTT_ENABLED=false as kill-switch).
Pskr.Aggregator buffers in memory keyed by Pskr.path_key/1 and
flushes every 60s via Repo.insert_all/3 with an additive
ON CONFLICT (count summed, SNR envelope by GREATEST/LEAST,
modes unioned via unnest+DISTINCT). Idempotent across overlapping
flushes and across leader handover.
Dockerfile sets BUILD_WITHOUT_QUIC=1 — emqtt's transitive quicer
dep wants CMake + OpenSSL headers we'd otherwise have to add to
the builder image just to never use QUIC over plain MQTT. Base
image is unchanged; the new dep compiles cleanly into the existing
prop-base runtime.
Mission now carries bands_mhz ({:array, :integer}) — operator picks
one or more bands as multi-checkboxes. enqueue_paths_for builds the
cross product (rover x station x band) and persists each tuple as its
own Path row keyed by (mission_id, rover_location_id, station_id,
band_mhz). The path-profile worker reads band_mhz from the path
itself (legacy single-band jobs without band_mhz in args still resolve
to their unique row).
replace_mission_paths/1 is now a thin alias for reconcile_mission_paths/1
which diffs desired vs actual: stale tuples (old band that the user
unchecked, station they removed, rover-site they deleted) get dropped,
new tuples become :pending and enqueue, surviving :complete rows are
left in place — no more wholesale destruction of already-computed
paths on every edit.
The show table gains a Band column, and band_label() in the mission
summary becomes bands_label() (joins the list with commas).
Renames the rover-location status enum and every label that referenced
it. Existing rows are migrated in place. Also touches the consumers:
- Rover.Location enum + default
- RoverLocationsLive index, status filter, form, show, badges
- Rover.Compute and rover_live (only_ideal_locations → only_good_locations)
- RoverPlanning context candidate filter (status == :good)
- RoverPlanning.Show / Form copy
- All rover-related tests
Adds a new globally-scoped, owner-mutable rover-mission tracker:
- /rover-planning paginated table (LiveTable) with View/Edit/Delete
- /rover-planning/new + /:id/edit form: name, band, antenna heights,
notes, "only check against known good locations" toggle (default on),
and a dynamic list of stationary stations entered as callsigns,
Maidenhead grids, or lat,lon pairs (Station changeset geocodes via
LocationResolver, lat/lon stays editable after add)
- /rover-planning/:id show page renders the station list, scope, and a
matrix of computed path profiles (distance, min clearance, diffraction,
verdict) populated as the worker completes each pairing
After save, RoverPlanning enqueues one RoverPathProfileWorker job per
(rover-location × station) pairing. The worker mirrors PathLive's
synchronous compute (ElevationClient + TerrainAnalysis at the mission's
band + heights) and stores the result on the matching path row.
PubSub broadcast on completion lets the show page live-refresh.
Admins can edit/delete any mission; owners can edit their own.
Oban Pro 1.7.0 ships database-backed workflow tracking, replaces the
generated `uniq_key` / `partition_key` columns with expression
indexes, and adds new partial indexes for query performance.
Switches oban_pro from the vendor/ tree to the licensed `oban` hex
repo (auth already configured via `mix hex.repo`). vendor/oban_pro
deleted — the vendoring was a 1.6.x stopgap, 1.7 is the first
version we land directly from hex.
oban_met (1.1.0) and oban_web (2.12.1) stay vendored — they're
not published to the licensed `oban` repo (verified by 404 on
`mix hex.package fetch`).
Migration 20260430142341 runs `Oban.Pro.Migration.up(version: "1.7.0")`
which creates the workflow tables and the new indexes. Verified on
dev + test DBs. Production is well below the size threshold the v1.7
guide flags for split-migration handling, so the inline default
suffices.
No code changes for the breaking-change list — the codebase only
references Oban.Pro.Engines.Smart and Oban.Pro.Plugins.DynamicLifeline,
both unchanged in 1.7. We don't use Workflow.after_cancelled/2 or
Oban.Pro.Workers.Chunk.
Test suite stays at 3132/3132.
Stage 3 (Elixir foundation) of the HRDPS plan. Lands the scaffolding
that lets the Rust prop-grid-rs HRDPS branch be wired up cleanly when
it ships, without further DB or Elixir-side changes.
What ships:
- `grid_tasks.source` column (default 'hrrr', backfill is a no-op).
Replaces the (run_time, forecast_hour, kind) unique index with a
4-column key (..., source) so HRRR and HRDPS analysis/forecast rows
for the same cycle coexist.
- `Grid.hrdps_only_points/0` — Canadian cells inside HRDPS bbox
(49-60°N, -141 to -52°W) but outside HRRR's CONUS bbox. Disjoint
from `conus_points/0` by construction so the two grids never
double-write the same (lat, lon).
- `Grid.point_source/1` — `:hrrr | :hrdps | :neither` classification
for a (lat, lon) pair. Routes per-QSO enrichment to the right NWP
source.
- `GridTaskEnqueuer.seed_with_analysis/2` and `seed/2` accept a
`:source` option; defaults to "hrrr" so existing callers are
unchanged.
- `HrdpsGridWorker` cycle seeder mirroring PropagationGridWorker's
shape but for HRDPS's 4×/day cadence. `:hrdps` queue added to base
Oban config.
What's deferred:
- The Rust `prop-grid-rs` worker doesn't yet branch on
`task.source`. Until that ships, `HrdpsGridWorker` stays out of
runtime.exs cron — it would just produce rows that the Rust worker
mishandles as HRRR. Module doc explicitly notes the dormant state.
- Map overlay extension and FreshnessMonitor HRDPS staleness deferred
to follow-up sessions once real data flows.
Activation path documented in the updated plan file.
Coverage cap recorded as Stage 8 decision: 60°N for v1 (SRTM stops
there). Arctic CDEM deferred to a follow-up plan.
Stage 2 of the HRDPS plan. Mirror of hrrr_profiles for HRDPS-derived
rows so HRDPS can be dropped/reprocessed without disturbing the 4500+
HRRR profiles already in prod. Same column set, same indexes,
PARTITION BY RANGE (valid_time) with quarterly partitions starting at
the deploy quarter.
Schema follows the @primary_key {:id, :binary_id, autogenerate: true}
convention and the same set of derived fields HrrrProfile carries
(surface_refractivity, min_refractivity_gradient, ducting_detected,
duct_characteristics, etc.) so the rest of the pipeline can consume
HRDPS-derived rows interchangeably with HRRR-derived rows.
Unique constraint on (lat, lon, valid_time) is enforced at the DB
level via the partitioned index; schema-level unique_constraint/2
matching against changeset errors doesn't apply because the
partition's index name varies per partition. Production upserts go
through `on_conflict: {:replace_all_except, ...}` per project
convention.
New globally-shared list of rover-friendly (or off-limits) parking
spots. Anyone can view; logged-in users can add new locations and
edit/delete their own. Each location stores lat/lon, free-text notes,
and a status enum (:ideal | :off_limits).
* Microwaveprop.Rover.Location schema + migration
* Rover.list_locations/0, create_location/2, update_location/3,
delete_location/2 (mutations require User; ownership-scoped)
* /rover-locations LiveView in the public live_session — Add button
is disabled for anonymous visitors, Edit/Delete buttons render
only on the user's own entries
* Tests for context + LiveView covering anonymous read, owner-only
edit/delete, and form submission
Two issues spotted in pg_stat_user_tables:
1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
The partitions are append-only from the hourly grid worker, so
autovacuum never crosses the analyze threshold — but without
stats the planner falls back to estimates and picks bad joins
on per-QSO lookups.
2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
`contact_common_volume_radar` see high UPDATE volumes
(enrichment status flips, queue claim/complete cycles, mechanism
classifier writes) but only accrue ~5% dead before the default
20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
LOCKED claim scan both want fresher stats than that.
Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
four tables above. Aggressive enough to keep stats current;
tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
`ANALYZE` on every table not auto-analyzed in the last 6 h,
reports counts. Meant to be invoked once via
`bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
after deploy to seed statistics on the cold partitions;
idempotent on re-run.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
Weather.find_nearest_hrrr/3 and Weather.find_nearest_native_profile/3
run a three-range scan gated first by a ±1h valid_time window, then
by a ±0.07° lat/lon box. The existing unique `(lat, lon, valid_time)`
index is only usable on its `lat` prefix for these queries, forcing
Postgres to scan wide lat slices before applying the valid_time
filter — expensive on the Turing Pi 2 node.
Add a `(valid_time, lat, lon)` composite so valid_time leads,
collapsing the search to a narrow btree slice inside a single
partition before the lat/lon box filter kicks in.
`hrrr_profiles` is RANGE-partitioned by valid_time, so
CREATE INDEX CONCURRENTLY is applied per partition and the resulting
indexes are attached to a parent index created ON ONLY the partitioned
table. `hrrr_native_profiles` is a regular table, so the concurrent
build runs directly on it.
Column name now matches the type key used everywhere else:
:hrrr -> hrrr_status, :weather -> weather_status,
:terrain -> terrain_status, :mechanism -> mechanism_status
Drops the @status_column_overrides remap in BackfillEnqueueWorker
(status_column/1 is now a straight :"#{type}_status" derivation) and
the long-form references in Contact, ContactWeatherEnqueueWorker,
and MechanismClassifyWorker. Migration is a metadata-only RENAME
COLUMN + RENAME INDEX — safe to land in prod (no table rewrite,
brief catalog lock only).
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
Prep for Stream A of Phase 3 rust migration: f00's native-duct merge,
NEXRAD, and commercial-link enrichment move to Rust via the same
grid_tasks work queue. kind='analysis' (single row per run) vs
kind='forecast' (18 rows per run) lets one Rust worker dispatch on
payload type without a second table.
Forecast default keeps existing Rust claim_next behaviour unchanged
during cutover.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
Contacts can be marked private at submit time (single form, CSV
import, ADIF import) and edit time. Private contacts are hidden
from the public map, the public /u/callsign profile, and the
browse table for anonymous and non-owning viewers. The original
submitter and admins see them inline on the browse table with a
"Yes" badge, and the detail page shows a lock icon. Non-authorized
viewers get a 404 on the detail page.
Add optional height1_ft/height2_ft to contact schema, submit form, and
edit form (direct + suggest-edit paths). Heights flow through to the
elevation-profile terrain analysis so clearance and diffraction are
computed from the actual antenna height instead of a 10-ft default.
Editing heights resets terrain_status and re-enqueues
TerrainProfileWorker so the stored path analysis picks up the new
geometry.
Also fix mark_likely_duct/3: the pattern was grabbing the element
instead of the index, so no duct ever got flagged as the likely
propagation path even when extract_ducts returned a valid layer.
Lays the groundwork for a Day 2-7 propagation outlook driven by NCEP
GEFS ensemble-mean output, picking up where HRRR's 18-hour horizon
leaves off.
- GefsClient: NOMADS URL builder, forecast-hour list (3h to +240,
6h to +384), ensemble-mean message inventory, and a Magnus
dewpoint derivation (pgrb2a publishes RH rather than Td)
- gefs_profiles table + GefsProfile schema mirroring hrrr_profiles
so the scorer can run over rows from either source
- Weather.upsert_gefs_profile/1 and upsert_gefs_profiles_batch/1
No worker or UI yet — those land in follow-up commits.
Classifies every contact's likely non-LOS propagation mechanism and
persists the result on contacts.propagation_mechanism. Mechanism is
determined in priority order:
1. user_declared_prop_mode (ADIF PROP_MODE from the operator log)
2. EME — moon-ephemeris check, ≥2m band, >1800 km path
3. aurora — Kp≥5 + high-lat path, 50-432 MHz
4. sporadic-E — foEs × 5 ≥ band_mhz, 400-2500 km path
5. meteor_scatter — ±3 days of a shower peak, VHF/UHF
6. rain_scatter — common-volume radar heavy rain, 5-11 GHz ≤800 km
7. tropo_duct — HRRR native_best_duct ≥ band or ducting_detected
8. line_of_sight — ≤50 km path
9. troposcatter — default
Persisted via MechanismClassifyWorker (queue: :mechanism, unique on
contact_id). Submit-time enqueue path includes :mechanism by default;
BackfillEnqueueWorker cron now handles :mechanism alongside existing
types so prod continuously backfills any contact with
propagation_mechanism_status in (:pending, :queued, :failed). Also
added :radar to the cron's type list so common-volume radar backfill
runs automatically rather than only via `mix radar_backfill`.
New modules:
- Microwaveprop.Propagation.MoonEphemeris — Meeus low-precision moon
position, accuracy ±1° — enough for the mutual-visibility EME test
- Microwaveprop.Propagation.MechanismClassifier — plug-in priority
chain over the evidence map
- Microwaveprop.Workers.MechanismClassifyWorker — assembles inputs
from HRRR / native profiles / common-volume radar / solar_indices /
ionosonde + calls the classifier
ADIF importer now reads PROP_MODE into user_declared_prop_mode so
operator-tagged mechanisms (EME/ES/MS/RS/AS/AUR) become ground truth.
Add a propagation_run_timings table so the wall-clock duration of each
(run_time, forecast_hour) step is queryable long after the run is over.
Keyed by (run_time, forecast_hour) with a status column that captures
whether the step succeeded or bailed out, and an error string on
failure.
PropagationGridWorker stamps every step (ok and failed) via
Propagation.record_run_timing/1. Timing inserts are wrapped in rescue +
changeset-error handling so the instrumentation can never brick the
chain.
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.
- New import_runs table: stores preview rows + running counters (total,
processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
its slice via CsvImport.commit_rows/1, atomically increments counters
in one update_all, flips status, and broadcasts progress on
'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.
Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.
LiveView UI for /imports/:id is a separate commit.
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).
- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
(was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
The US 9cm amateur allocation was cut from 3300-3500 to 3300-3450 MHz
in 2020, so 3456 (the legacy weak-signal calling frequency) is no
longer in-band. Move the canonical key to 3400, migrate existing
contacts, and keep 3456 input resolving to 3400 via the
nearest-band snap so historical ADIF/CSV logs still import cleanly.
Adds the second ionospheric data source layer. Polls NOAA SWPC's free
public JSON services every 5 minutes for three products:
* Planetary K-index (1-min cadence) → geomagnetic_observations
* 10.7 cm solar flux (hourly) → solar_flux_observations
* GOES X-ray flux (1-min, 0.1-0.8nm long-wave band only) →
solar_xray_observations
All three products feed HF / Es scoring inputs that the propagation
engine will start using in a follow-up commit:
* Kp drives HF absorption and Es damping (memory notes Kp is inversely
correlated with tropo/Es stability).
* F10.7 is the SSN proxy for ITU-R P.533 HF MUF prediction.
* GOES X-ray long-wave flux is the short-wave fade (SWF) / D-layer
absorption proxy — a C-class flare starts attenuating HF within
seconds of the X-ray peak.
New context `Microwaveprop.SpaceWeather` exposes `upsert_*/1` and
`latest_*/0` per product; `SpaceWeatherFetchWorker` pulls all three
independently so one endpoint's outage doesn't block the others.
Fixtures captured from live SWPC endpoints on 2026-04-15 for the
parser tests.
Not yet wired into scoring — that's a separate commit.
First step toward physics-based HF / sporadic-E scoring. Polls GIRO's
DIDBase tabular endpoint every 10 minutes for two US ionosondes
(Millstone Hill MHJ45, Alpena AL945), parses the plain-text response
into observation rows, and upserts into a new ionosonde_observations
table keyed on (station_code, valid_time).
This gets foEs (E-layer sporadic critical frequency) into the database
as a direct measurement — the key input for predicting 144 MHz Es
openings via ITU-R P.534-6. foF2 + MUFD are the F2-layer inputs for
HF MUF predictions.
Not yet wired into scoring. Boulder/Wallops/Austin/Idaho/Point Arguello
are in the GIRO catalog but were silent when probed — add them back
if/when they come online.
Next steps: SWPC JSON (Kp, F10.7, sunspot), GOES X-ray flux, D-RAP text,
and the P.534-6 Es scoring factor that uses foEs at midpoint for the
144/440 band configs.
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:
1. New migration drops the propagation_scores table and its indexes
(the earlier tuning migrations for it were already applied and
are now no-ops against a missing table, which is fine — Ecto
just runs them on fresh environments).
2. Propagation context is gutted of every GridScore reference.
replace_scores/2 writes files only. upsert_scores/2 is deleted.
load_scores_from_db, available_valid_times_from_db,
point_detail_from_db, point_forecast_from_db, fetch_factors,
coalesce_factors, the Postgres side of prune_old_scores, and
the Postgres fallbacks in latest/earliest_valid_time are all
removed. point_detail always returns an empty factors map now
since factor breakdowns were retired with the table.
3. Deleted modules:
- Propagation.GridScore (the schema)
- Propagation.ScorerDiff (read factors from the table)
- Propagation.AsosNudge (helper for AsosAdjustmentWorker)
- Workers.AsosAdjustmentWorker (its cron was already disabled)
- Mix.Tasks.ScorerDiff (wrapper around the deleted module)
And their tests. AdminTaskWorker's scorer_diff task is a
logging no-op so any queued Oban rows drain cleanly.
Release.scorer_diff stays as a stub that tells the operator.
propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
The scoring+upsert phase was ~4m40s per forecast hour and dominated
wall time. Three stacked optimizations attack it from different
angles.
replace_scores/2 is a new hot-path writer that does DELETE WHERE
valid_time = $1 followed by a plain insert_all (no ON CONFLICT
resolution). The chain worker rewrites the full (valid_time, all
bands) slice every forecast hour, so conflict detection was pure
waste. AsosAdjustmentWorker still uses upsert_scores because it
only rewrites the subset of cells near a station.
factors is now nullable. Forecast hours f01-f18 pass factors: nil
so the JSONB encode + toast write is skipped entirely — roughly
halves the data volume per run. point_detail/4 coalesces nil to
an empty map so the JS popup renders without a TypeError, and
scorer_diff only pulls the most recent valid_time that still has
factors (the f00 row).
propagation_scores is now UNLOGGED, so inserts bypass WAL entirely.
Durability tradeoff: an unclean shutdown truncates the table, but
PropagationGridWorker rebuilds it from HRRR every 3h so a lost
table is re-populated within one cron cycle.
Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a
speculative plan for a flat-file / DuckDB rewrite with explicit
trigger conditions for when to pick it up (partitioning deferred
too; revisit only if these three don't solve it).
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Four independent wins from a perf audit of the era5_batch worker path:
1. Parallel CDS submits per job. fetch_single_level_month and
fetch_pressure_level_month have no ordering dependency — each blocks
30+ min on CDS queue/download — so running them in two linked Tasks
halves wall time for every job in the backlog.
2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
A month-tile GRIB is 50–200 MB and was previously slurped into the
BEAM heap and then written back out to a temp file before wgrib2 ran.
New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
full parallelism. Partial files are removed on HTTP error.
3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
streamed temp file directly. The binary variant still exists for the
single-point path and now delegates through the same helper.
4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
three-range scan used by Weather.find_nearest_era5 (contact detail +
path profiles) and StatusLive.count_era5_done (status page EXISTS
subquery). The unique index on (lat, lon, valid_time) couldn't serve
it efficiently.
Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
Instead of shelling out to https://gridmap.org/locate/:callsign for
callsign → lat/lon lookups, ports the resolver pipeline from gridmap-web
into this project so the whole flow runs in-process.
New modules:
- Microwaveprop.Qrz — cache facade over the QRZ.com XML callsign API.
Looks up from qrz_callsigns first, falls back to a live fetch, and
upserts the result with a configurable cache_ttl_hours (default
168h / 7 days).
- Microwaveprop.Qrz.Client — HTTP/XML client against
https://xmldata.qrz.com/xml/current/. Holds the session key in an
Agent, transparently re-logs-in on :session_expired, and parses
responses via xmerl.
- Microwaveprop.Qrz.Callsign — Ecto schema for the qrz_callsigns
cache table, binary_id primary key per project convention.
- Microwaveprop.Qrz.Record — slim struct with only the 11 fields
we actually consume (identity, name, grid, address, lat/lon).
The full XML payload stays in the raw :data jsonb column for
anyone who wants the other ~40 QRZ fields.
- Microwaveprop.Geocoder — Req-based client against the Google Maps
Geocoding API. Only called as a fallback when QRZ has no explicit
<lat>/<lon> for the callsign.
- Microwaveprop.CallsignLocation — orchestrator. Reads the
callsign_locations cache, on miss calls Qrz then either uses QRZ's
coords directly or geocodes the formatted address, snaps to an
8-char Maidenhead grid via Microwaveprop.Radio.Maidenhead, and
upserts the result.
Microwaveprop.Radio.CallsignClient.locate/1 is rewritten to delegate
to CallsignLocation.lookup/1 and shape the response back to the
existing {:ok, %{callsign, gridsquare, lat, lon}} contract so the
callers in PathLive and RoverLive don't change.
Wiring:
- priv/repo/migrations/20260413000000_create_qrz_callsigns_and_callsign_locations.exs
creates qrz_callsigns and callsign_locations with unique indexes
on :callsign.
- Microwaveprop.Qrz.Client added to the application supervision tree
so the session Agent is started.
- :xmerl added to extra_applications so the release bundles it.
- config/test.exs wires Req.Test plugs for Microwaveprop.Qrz.Client
and Microwaveprop.Geocoder, forces cache_ttl_hours: 0 so the cache
never short-circuits test-level stubs, and supplies dummy QRZ
credentials.
- config/runtime.exs pulls QRZ_USERNAME / QRZ_PASSWORD / QRZ_AGENT
and GOOGLE_API_KEY from the environment so prod and dev can
configure both upstream keys out of band.
Tests (ported verbatim from gridmap-web):
- test/microwaveprop/qrz/callsign_test.exs — schema/changeset
- test/microwaveprop/qrz_test.exs — cache hit, cache miss, TTL
behavior, upsert on stale, error passthrough, case-insensitive
input
- test/microwaveprop/geocoder_test.exs — success, zero results,
request denied, transport error
- test/microwaveprop/callsign_location_test.exs — end-to-end flow
including the QRZ-lat/lon shortcut and the missing-address error
path
All 1294 tests still pass. Credo strict clean.
- Add contacts_inserted_at_desc_id_desc_idx so /contacts list runs as an
Index Scan (~0.4ms) instead of a Seq Scan + top-N heapsort (~77ms on
58k rows)
- Add Microwaveprop.Cache: tiny ETS-backed TTL cache for memoizing
expensive-but-stale-tolerant values
- Cache total_entries for unsearched /contacts page loads (30s TTL,
invalidated on insert)
- Cache the entire /contacts/map payload (pre-encoded JSON + band list),
moving load_contacts into Radio.contact_map_payload; invalidated on
new contact insert. Mount goes from ~150-300ms to ~5ms.
- Fix /map overlay blinking on load: reuse the Leaflet GridLayer across
renderScores calls and just redraw() against the updated ScoreGrid,
instead of removeLayer + new layer which flashed between tile regens
- Make mode optional on submission: schema change (null: true),
submission_changeset no longer requires it, blank strings normalise to
nil, CSV import accepts 6-column (no mode) or 7-column layouts
- core_components .input adds a red asterisk to labels when required,
giving users a visual cue for which fields must be filled on /submit
Registered users can suggest edits to any contact's core fields
(callsigns, grids, band, mode, timestamp). Edits enter an admin
approval queue with field-by-field diff view. On approve, changes
are applied and enrichment re-enqueued if grids/band changed.
Users receive email notification on approve or reject.
Also updates dependabot.yml for mix ecosystem.
