- Add Weather.GridCache: ETS cache of derived HRRR grid rows keyed by
valid_time, cluster-synced via PubSub. Eagerly warmed from
PropagationGridWorker after each upsert so /weather map pan/zoom and
weather_point_detail hit zero DB on warm cache.
- Replace latest_weather_grid DB query path with cache-first lookup +
DB fallback. hrrr_profiles is 42M rows partitioned; pulling 3-10k
rows per viewport on every pan was the main cost.
- ContactLive.Show: defer the heavy enrichment loads (weather, solar,
HRRR, terrain, IEMRE, elevation profile, ITU-R propagation analysis,
data_sources) into a handle_info(:hydrate) that runs after the shell
renders. Initial mount now returns nil placeholders; template already
had :if guards for all of them. Shell-to-first-paint goes from
~500ms-2s down to ~20ms.
- Cache fetch_queue_counts for 5s in ContactLive.Show — oban_jobs group
by query was running on every contact page view.
- Backfill stats: wrap count_unprocessed, fetch_stats, fetch_db_stats
in Microwaveprop.Cache with 2-5s TTLs; bump refresh debounce from 1s
to 2s so bulk enrichment events don't thrash the DB.
- Add contacts_inserted_at_desc_id_desc_idx so /contacts list runs as an
Index Scan (~0.4ms) instead of a Seq Scan + top-N heapsort (~77ms on
58k rows)
- Add Microwaveprop.Cache: tiny ETS-backed TTL cache for memoizing
expensive-but-stale-tolerant values
- Cache total_entries for unsearched /contacts page loads (30s TTL,
invalidated on insert)
- Cache the entire /contacts/map payload (pre-encoded JSON + band list),
moving load_contacts into Radio.contact_map_payload; invalidated on
new contact insert. Mount goes from ~150-300ms to ~5ms.
- Fix /map overlay blinking on load: reuse the Leaflet GridLayer across
renderScores calls and just redraw() against the updated ScoreGrid,
instead of removeLayer + new layer which flashed between tile regens
- Make mode optional on submission: schema change (null: true),
submission_changeset no longer requires it, blank strings normalise to
nil, CSV import accepts 6-column (no mode) or 7-column layouts
- core_components .input adds a red asterisk to labels when required,
giving users a visual cue for which fields must be filled on /submit
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
immediately with a pending marker; push rain_scatter_update when
NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
viewport after mount/band change/propagation_updated; client caches
them and renders timeline scrubs instantly without a server roundtrip.
Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
timeline click uses preloaded cache when available
- Add ScoreCache GenServer with node-local ETS table keyed by
{band, valid_time}, subscribed to "propagation:cache" PubSub topic so
every pod stays in sync with a single hourly compute
- scores_at/3 checks cache first, falls back to DB and populates on miss
- PropagationGridWorker warms and broadcasts the cache for each band
after every forecast hour upsert; prunes >2h old entries
- Replace per-pixel string-keyed Map with flat Int8Array over the CONUS
grid in propagation_map_hook.ts to eliminate allocations in the tile
rasterization hot loop (interpolateScore / propagationReach)
Aliases: add module aliases for 9 nested module references
Apply: replace apply/3 with direct module attribute calls
Line length: break 1 long spec line
Refactoring: extract helpers to reduce complexity and nesting
in show.ex, radio.ex, weather workers, terrain, duct detection,
backfill dashboard, contact map, and mix tasks
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
- Add belongs_to :user on Beacon schema, preload in get_beacon!/1
- Show "Submitted by {callsign}" on beacon detail page
- Add NTMS donation link on the about page
- Change grid preference text to 8 characters (e.g. EM12kp37)
Parse ADIF tagged-data format with fuzzy frequency-to-band matching
(nearest microwave band for any freq >= 900 MHz). Supports STATION_CALLSIGN,
OPERATOR, CALL, GRIDSQUARE, MY_GRIDSQUARE, FREQ, BAND, QSO_DATE, TIME_ON.
Same dedup and preview/commit flow as CSV upload.
Also fix select dropdown text alignment in daisyUI.
SRTM profiles no longer abort on missing tiles — water/ocean areas
return 0m elevation instead of failing the entire profile and falling
back to the rate-limited API. Elevation client and viewshed now pass
download: true to auto-fetch missing tiles from S3. NFS mount changed
to writable so downloaded tiles persist across pods.
Registered users can suggest edits to any contact's core fields
(callsigns, grids, band, mode, timestamp). Edits enter an admin
approval queue with field-by-field diff view. On approve, changes
are applied and enrichment re-enqueued if grids/band changed.
Users receive email notification on approve or reject.
Also updates dependabot.yml for mix ecosystem.
Height in feet doesn't need decimal precision. Migrates the DB
column, updates schema type, and strips trailing .0 from form
input so the integer cast succeeds.
- Add `mix backtest --all` for consolidated pass/fail table across all features
- Add Backtest.consolidated_report/2 and to_consolidated_markdown/1
- Add Features.all_features/0 to auto-discover backtestable features
- Add `mix import_contest_logs` for bulk ARRL contest CSV import with dedup
- Fix hrrr_climatology to batch by (month, hour) to avoid query timeout
- Fix Repo.query! result pattern (Postgrex.Result, not tuple)
- Backtest reports for all Phase 1-6 features
FrontalAnalysis module (Weather.FrontalAnalysis):
- detect_fronts/3 computes the Thermal Front Parameter (TFP) from
2D grids of surface temperature and pressure using Nx vectorized
ops. TFP = -nabla|nabla(theta)| . nabla(theta)/|nabla(theta)|.
Most negative values mark cold fronts.
- central_gradient/1 for 2D finite differences with edge handling
- nearest_front/3 finds closest front point with distance and bearing
- path_front_angle/2 computes angle between a QSO path and the
front (0 = parallel = good, 90 = crosses = dead)
Backtest feature stubs for distance_to_front and parallel_to_front
(return nil until the pipeline caches per-cell frontal features from
the hourly HRRR grid run). The FrontalAnalysis module itself is
tested and ready for integration.
NEXRAD spike docs also included in this commit.
NCEI ASOS 5-minute data client (Weather.NceiMetarClient):
- fetch/3 pulls per-station monthly .dat files from NCEI C00418
- parse/1 decodes the fixed-width METAR format including precise
T-group temperatures (T02110094 → 21.1/9.4°C)
- metar_5min_observations table: schema-identical to
surface_observations, separate table to avoid mixing cadences
Weather.recent_surface_obs/3 prefers 5-min data when available,
falls back to the hourly surface_observations table.
Data URL: https://www.ncei.noaa.gov/data/automated-surface-observing-system-five-minute/access/YYYY/MM/asos-5min-KXXX-YYYYMM.dat
Available back to 1996.
Add Propagation.Region module with 8 CONUS climate zones (gulf_coast,
southeast, southern_plains, corn_belt, northeast, desert_southwest,
pacific_northwest, mountain_west) and per-region monthly seasonal
adjustment multipliers.
The scorer's score_season now takes lat/lon and applies a regional
multiplier from Region.seasonal_adjustment on top of the band's
seasonal_base + seasonal_adj. Gulf coast August gets a 1.15x boost
(drier, better for ducting) while Corn Belt August gets a 0.80x
penalty (corn evapotranspiration = miserable dewpoints).
Adjustments are hand-tuned starting points from the meteorologist's
qualitative guidance. Phase 9 recalibration will refine them from
backtest data.
Duct module (Propagation.Duct):
- refractivity_profile/1: ITU-R P.453 N at each native level
- m_profile/1: modified refractivity M = N + 157*h(km)
- detect_ducts/1: find contiguous regions where dM/dh < 0, returning
base/top height, thickness, and M-deficit per duct
- min_trapped_frequency_ghz/1: waveguide approximation (Bean & Dutton)
for the minimum frequency a duct of given geometry can trap
- analyze/1: full pipeline from native profile to duct list + best
trapped frequency across all ducts
Derive task updated to also compute ducts JSONB and best_duct_band_ghz
alongside the Phase 2 turbulence fields.
Backtest features: duct_thickness, best_duct_freq, duct_usable_10ghz,
duct_usable_24ghz, duct_usable_47ghz.
Real-data validation: 2022-08-20 12Z TX profile shows 0 ducts (M
increases monotonically) — correct for a well-mixed boundary layer
on a turbulent August afternoon (Ri=0.16).
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286
Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric
mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.
First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
wgrib2 -lola ... bin writes Fortran unformatted records (4-byte
length header + data + 4-byte length trailer per message).
parse_lola_binary was treating the binary as tightly packed,
causing every message after the first to read from the wrong
offset — values came out as garbage across all grid points.
Fix: account for the 8-byte record overhead per message when
computing the data offset for each message's grid values.
This bug affects both the existing propagation grid extraction
(which may have been producing subtly wrong scores) and the new
native-level extraction (which was producing obviously wrong
values). The fix is a one-line stride change.
Also adds Backtest.Features.native_surface_refractivity for the
Phase 1 sanity check, plus a tighter wgrib2 match pattern that
selects only hybrid-level messages from the native file.
- Spike docs at docs/research/hrrr_native_levels.md confirming files
are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
SPFH needed for Phase 2 turbulence features. Architectural finding:
per-point on-demand fetching is impractical (~530 MB/file), so
the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
scalars and placeholder columns for Phase 2/4 derived fields.
Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
unique at :infinity, pulls distinct (lat, lon) points from contacts
in the ±30 min window, downloads the native grib2, extracts per
point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
contact count.
Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
Add Microwaveprop.Backtest: a feature-evaluation framework that runs
a (lat, lon, valid_time) -> float function over the historical QSO
corpus and a matched random-time baseline, reporting distribution
statistics, distance-binned lift, and band-stratified lift.
Adds four baseline feature wrappers around the current scorer inputs
(naive_gradient, td_depression, time_of_day, pressure), a mix backtest
CLI, and the first set of baseline reports under priv/backtest_reports
so downstream phases have a frozen reference point to compare against.
When a contact is linked to a logged-in user account, display that
user's callsign in the Submitted column instead of a generic "Yes".
Anonymous submissions still show "Yes"; scraped contacts show an em
dash.
Add a per-entry progress bar (with cancel) while a CSV streams in, and
replace the small post-commit alert with a prominent "X contacts
submitted" confirmation card.
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.
- Era5BatchClient builds the monthly CDS requests, extracts every
(lat, lon, hour) from the GRIB2 blob with wgrib2, derives
refractivity params, and bulk-inserts in 2k-row chunks with
on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
→ enqueue the month-batch for the point's tile-month and return.
No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
Uploads now show a review page with three summary cards (valid,
duplicates, invalid), tables for invalid and duplicate rows, and a
sample of the first 20 valid rows. Nothing is written to the database
until the user clicks "Looks good — submit N contacts".
Duplicate detection treats two rows as the same contact if they share
the same pair of (callsign, grid) tuples (direction-agnostic), the
same band, and timestamps within 60 minutes. Dedup runs against both
existing DB contacts (single band/time-scoped query) and earlier rows
in the same upload, and the preview labels which side matched.
CsvImport grows preview/2 + commit/1; the old import/2 is kept as a
thin backward-compat wrapper that skips dedup. Added 11 new CsvImport
tests and reworked the SubmitLive CSV tests for the two-step flow.
Pruning used to only run at the end of a successful PropagationGridWorker
pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM)
stopped prune from running and let the table accumulate ~5h of stale
rows. A dedicated PropagationPruneWorker now runs every 15 minutes on
its own Oban cron, and PropagationGridWorker also calls prune_old_scores
at the start of each run as a second safety net. Bumped the delete
timeout from 2m to 5m so the first catch-up pass has enough headroom.
Beacons can now record a directional bearing ("omni" or 0-360 degrees)
and a beamwidth in degrees. The keying list grows from {on_off, fsk}
to include FM Voice, WSPR, and the full Q65A/B/C/D/E × 15/30/60/120
family, grouped via optgroups in the submission form.
- Anonymous users can submit beacons (held pending admin approval)
- Added notes field to beacons (textarea on form, shown on detail page)
- Added Beacons link to /map sidebar nav (mobile + desktop),
removed stale Rover Planner link
- Moved /backfill to /admin/backfill under the admin live_session
- Beacon detail map zooms in two extra levels after fitBounds
The Era5FetchWorker now declares an Oban unique constraint on
{lat, lon, valid_time} for any job in available/scheduled/executing/
retryable, so two backfill runs targeting the same contact grid point
can no longer spawn parallel CDS requests for the same hour.
Because Oban OSS insert_all doesn't honor unique, ERA5 jobs are now
routed through Oban.insert/1 from ContactWeatherEnqueueWorker and the
era5_backfill mix task. Other worker types still use insert_all.
Non-admin users can now submit beacons but they are held in an
unapproved state until an admin approves them. Only approved beacons
appear in the public list; pending submissions are shown in a separate
admin-only section on /beacons with Approve and Delete actions. The
show page surfaces a pending badge and an admin-only Approve button
when viewing an unapproved beacon.
Also formats EIRP (mW) on the index page without scientific notation.
Replace the idealised concentric-circle range rings with a realistic
per-HRRR-grid-cell reception map. For every 0.125° grid point within
the band's exceptional range, the estimator computes great-circle
distance, FSPL + atmospheric loss, and applies a cell-specific score
adjustment of (50 - score) * 0.3 dB — score 100 gives a 15 dB ducting
boost, score 0 a 15 dB penalty. Cells below the -145 dBm detection
floor are dropped.
The beacon map hook now renders each surviving cell as a 0.125°
filled rectangle with a tooltip showing tier, Rx dBm, distance, and
HRRR score, so the coverage footprint bulges where ducting conditions
are good and cuts off where they aren't, instead of being a perfect
circle. Falls back to score 50 for cells that don't have HRRR data
yet, so the plot is always useful.
Also fixes the prior all-grey-tiles regression by restoring an
explicit setView() at map creation and adding a post-mount
invalidateSize() for when the map is placed inside a flex container.
- Add on_the_air boolean to beacons (default true); surfaced as a
checkbox on the form, a badge on the index, and in the detail list.
- Render a Leaflet map on the beacon show page with a marker at the
beacon's lat/lon tooltipped with callsign + frequency. Marker is
green when on air, gray when off.
- Compute and draw a reception-range estimate as concentric signal-
strength rings. New Microwaveprop.Beacons.RangeEstimate solves a
link budget (FSPL + O2/H2O absorption from BandConfig) at five RX
thresholds (-100 to -145 dBm), then scales by 0.5 + score/100 from
the latest Propagation.point_detail at the beacon's grid square, so
current HRRR conditions shift the rings in or out.
- Re-enable the hourly PropagationGridWorker cron and freshness
monitor in dev.exs so dev actually has HRRR-backed scores to feed
the new estimator.
- Rename beacons.height_m to height_ft; migration converts existing
values (m * 3.28084). Form/list/show labels updated.
- Beacon changeset now fills lat/lon from the grid when they're blank
(previously only the reverse worked). Lat/lon and grid inputs are no
longer marked required in the form — enter one side and the other
populates on blur via phx-change.
- Rename beacons.power_watts to power_mw (migration multiplies existing
values by 1000); form/list/show all relabeled to "Power (mW)"
- Fix Add-monitor button alignment on /users/settings by rendering the
label above and putting the input and button in a plain flex row so
the input wrapper margin no longer offsets the button
- Extend the settings page re-auth window from 10 minutes to 24 hours
Beacons:
- Scaffolded with phx.gen.live then reworked so reads are public
and mutations go through a live_session gated by the new
:require_admin on_mount hook in UserAuth
- Beacon schema stores frequency (MHz), callsign, grid, lat/lon,
power (W), and height above ground (m); grid auto-derives from
lat/lon when left blank via Maidenhead.from_latlon
- Adds Maidenhead.from_latlon/3 so we can compute grids locally
instead of hitting an external API
Users admin page:
- /users and /users/:id/edit (admin-only) for listing, editing
(callsign/name/email/is_admin), and deleting other users
- Adds Accounts.list_users, admin_update_user, delete_user, and
a dedicated admin_changeset on the User schema
- Nav gains a "Users" link for admins and a "Beacons" link for
everyone
Adds a boolean is_admin column (default false) and has the
registration changeset auto-set it to true when the email matches
graham@mcintire.me (case-insensitive). The migration also backfills
the flag for an existing row with that email.
Users can now register beacon monitors on the settings page. Each
monitor gets a unique random token the remote program will use to
authenticate its reports. Name is the only user-supplied field for
now; more will be added as the monitor protocol is defined.
Also adds :gen_smtp to deps. Swoosh's SMTP adapter depends on
:mimemail which lives in that package, and production was 500'ing
when trying to send confirmation emails without it.
- Register/Log in (or callsign/Settings/Log out) now live next to
the Map/Path/Rover links in the main header instead of a separate
top menu bar
- Add on_mount in UserAuth to assign current_scope on LiveView mount,
and pass current_scope through to Layouts.app from every LiveView
- Drop the old top <ul> from root.html.heex
- Password minimum lowered from 12 to 8 characters
Generated Accounts context, User schema, and controllers via
phx.gen.auth. Adapted it for password-only login with required
email confirmation:
- Users have callsign (unique, uppercased), name, email, password
- Registration form fields: callsign, name, email, password, confirm
- Magic-link login path removed; login is email + password only
- After register, a confirmation email is sent and login is blocked
until the account is confirmed via the token URL
- Confirmation link logs the user in on first use
- SMTP2GO configured as the outgoing mailer in k8s prod
New BandConfig entries for 902, 1296, 2304, 3456, 5760 MHz:
- All beneficial humidity effect (like 10 GHz)
- Near-zero gaseous absorption and rain attenuation
- Ranges: 902 MHz typical 400 km, 5760 MHz typical 220 km
- Same seasonal curves as 10 GHz (ducting-driven)
BandConfig.band_options/0 generates dropdown options from configs.
All pages (path, rover, submit) use centralized band_options instead
of hardcoded lists. Map page already used BandConfig.all_bands().
10 GHz remains the default on all pages.
