wgrib2 -lola ... bin writes Fortran unformatted records (4-byte
length header + data + 4-byte length trailer per message).
parse_lola_binary was treating the binary as tightly packed,
causing every message after the first to read from the wrong
offset — values came out as garbage across all grid points.
Fix: account for the 8-byte record overhead per message when
computing the data offset for each message's grid values.
This bug affects both the existing propagation grid extraction
(which may have been producing subtly wrong scores) and the new
native-level extraction (which was producing obviously wrong
values). The fix is a one-line stride change.
Also adds Backtest.Features.native_surface_refractivity for the
Phase 1 sanity check, plus a tighter wgrib2 match pattern that
selects only hybrid-level messages from the native file.
- Spike docs at docs/research/hrrr_native_levels.md confirming files
are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
SPFH needed for Phase 2 turbulence features. Architectural finding:
per-point on-demand fetching is impractical (~530 MB/file), so
the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
scalars and placeholder columns for Phase 2/4 derived fields.
Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
unique at :infinity, pulls distinct (lat, lon) points from contacts
in the ±30 min window, downloads the native grib2, extracts per
point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
contact count.
Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
Add Microwaveprop.Backtest: a feature-evaluation framework that runs
a (lat, lon, valid_time) -> float function over the historical QSO
corpus and a matched random-time baseline, reporting distribution
statistics, distance-binned lift, and band-stratified lift.
Adds four baseline feature wrappers around the current scorer inputs
(naive_gradient, td_depression, time_of_day, pressure), a mix backtest
CLI, and the first set of baseline reports under priv/backtest_reports
so downstream phases have a frozen reference point to compare against.
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.
- Era5BatchClient builds the monthly CDS requests, extracts every
(lat, lon, hour) from the GRIB2 blob with wgrib2, derives
refractivity params, and bulk-inserts in 2k-row chunks with
on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
→ enqueue the month-batch for the point's tile-month and return.
No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
Uploads now show a review page with three summary cards (valid,
duplicates, invalid), tables for invalid and duplicate rows, and a
sample of the first 20 valid rows. Nothing is written to the database
until the user clicks "Looks good — submit N contacts".
Duplicate detection treats two rows as the same contact if they share
the same pair of (callsign, grid) tuples (direction-agnostic), the
same band, and timestamps within 60 minutes. Dedup runs against both
existing DB contacts (single band/time-scoped query) and earlier rows
in the same upload, and the preview labels which side matched.
CsvImport grows preview/2 + commit/1; the old import/2 is kept as a
thin backward-compat wrapper that skips dedup. Added 11 new CsvImport
tests and reworked the SubmitLive CSV tests for the two-step flow.
Pruning used to only run at the end of a successful PropagationGridWorker
pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM)
stopped prune from running and let the table accumulate ~5h of stale
rows. A dedicated PropagationPruneWorker now runs every 15 minutes on
its own Oban cron, and PropagationGridWorker also calls prune_old_scores
at the start of each run as a second safety net. Bumped the delete
timeout from 2m to 5m so the first catch-up pass has enough headroom.
Beacons can now record a directional bearing ("omni" or 0-360 degrees)
and a beamwidth in degrees. The keying list grows from {on_off, fsk}
to include FM Voice, WSPR, and the full Q65A/B/C/D/E × 15/30/60/120
family, grouped via optgroups in the submission form.
- Anonymous users can submit beacons (held pending admin approval)
- Added notes field to beacons (textarea on form, shown on detail page)
- Added Beacons link to /map sidebar nav (mobile + desktop),
removed stale Rover Planner link
- Moved /backfill to /admin/backfill under the admin live_session
- Beacon detail map zooms in two extra levels after fitBounds
The Era5FetchWorker now declares an Oban unique constraint on
{lat, lon, valid_time} for any job in available/scheduled/executing/
retryable, so two backfill runs targeting the same contact grid point
can no longer spawn parallel CDS requests for the same hour.
Because Oban OSS insert_all doesn't honor unique, ERA5 jobs are now
routed through Oban.insert/1 from ContactWeatherEnqueueWorker and the
era5_backfill mix task. Other worker types still use insert_all.
Non-admin users can now submit beacons but they are held in an
unapproved state until an admin approves them. Only approved beacons
appear in the public list; pending submissions are shown in a separate
admin-only section on /beacons with Approve and Delete actions. The
show page surfaces a pending badge and an admin-only Approve button
when viewing an unapproved beacon.
Also formats EIRP (mW) on the index page without scientific notation.
Replace the idealised concentric-circle range rings with a realistic
per-HRRR-grid-cell reception map. For every 0.125° grid point within
the band's exceptional range, the estimator computes great-circle
distance, FSPL + atmospheric loss, and applies a cell-specific score
adjustment of (50 - score) * 0.3 dB — score 100 gives a 15 dB ducting
boost, score 0 a 15 dB penalty. Cells below the -145 dBm detection
floor are dropped.
The beacon map hook now renders each surviving cell as a 0.125°
filled rectangle with a tooltip showing tier, Rx dBm, distance, and
HRRR score, so the coverage footprint bulges where ducting conditions
are good and cuts off where they aren't, instead of being a perfect
circle. Falls back to score 50 for cells that don't have HRRR data
yet, so the plot is always useful.
Also fixes the prior all-grey-tiles regression by restoring an
explicit setView() at map creation and adding a post-mount
invalidateSize() for when the map is placed inside a flex container.
- Add on_the_air boolean to beacons (default true); surfaced as a
checkbox on the form, a badge on the index, and in the detail list.
- Render a Leaflet map on the beacon show page with a marker at the
beacon's lat/lon tooltipped with callsign + frequency. Marker is
green when on air, gray when off.
- Compute and draw a reception-range estimate as concentric signal-
strength rings. New Microwaveprop.Beacons.RangeEstimate solves a
link budget (FSPL + O2/H2O absorption from BandConfig) at five RX
thresholds (-100 to -145 dBm), then scales by 0.5 + score/100 from
the latest Propagation.point_detail at the beacon's grid square, so
current HRRR conditions shift the rings in or out.
- Re-enable the hourly PropagationGridWorker cron and freshness
monitor in dev.exs so dev actually has HRRR-backed scores to feed
the new estimator.
- Rename beacons.height_m to height_ft; migration converts existing
values (m * 3.28084). Form/list/show labels updated.
- Beacon changeset now fills lat/lon from the grid when they're blank
(previously only the reverse worked). Lat/lon and grid inputs are no
longer marked required in the form — enter one side and the other
populates on blur via phx-change.
- Rename beacons.power_watts to power_mw (migration multiplies existing
values by 1000); form/list/show all relabeled to "Power (mW)"
- Fix Add-monitor button alignment on /users/settings by rendering the
label above and putting the input and button in a plain flex row so
the input wrapper margin no longer offsets the button
- Extend the settings page re-auth window from 10 minutes to 24 hours
Beacons:
- Scaffolded with phx.gen.live then reworked so reads are public
and mutations go through a live_session gated by the new
:require_admin on_mount hook in UserAuth
- Beacon schema stores frequency (MHz), callsign, grid, lat/lon,
power (W), and height above ground (m); grid auto-derives from
lat/lon when left blank via Maidenhead.from_latlon
- Adds Maidenhead.from_latlon/3 so we can compute grids locally
instead of hitting an external API
Users admin page:
- /users and /users/:id/edit (admin-only) for listing, editing
(callsign/name/email/is_admin), and deleting other users
- Adds Accounts.list_users, admin_update_user, delete_user, and
a dedicated admin_changeset on the User schema
- Nav gains a "Users" link for admins and a "Beacons" link for
everyone
Adds a boolean is_admin column (default false) and has the
registration changeset auto-set it to true when the email matches
graham@mcintire.me (case-insensitive). The migration also backfills
the flag for an existing row with that email.
Users can now register beacon monitors on the settings page. Each
monitor gets a unique random token the remote program will use to
authenticate its reports. Name is the only user-supplied field for
now; more will be added as the monitor protocol is defined.
Also adds :gen_smtp to deps. Swoosh's SMTP adapter depends on
:mimemail which lives in that package, and production was 500'ing
when trying to send confirmation emails without it.
- Register/Log in (or callsign/Settings/Log out) now live next to
the Map/Path/Rover links in the main header instead of a separate
top menu bar
- Add on_mount in UserAuth to assign current_scope on LiveView mount,
and pass current_scope through to Layouts.app from every LiveView
- Drop the old top <ul> from root.html.heex
- Password minimum lowered from 12 to 8 characters
Generated Accounts context, User schema, and controllers via
phx.gen.auth. Adapted it for password-only login with required
email confirmation:
- Users have callsign (unique, uppercased), name, email, password
- Registration form fields: callsign, name, email, password, confirm
- Magic-link login path removed; login is email + password only
- After register, a confirmation email is sent and login is blocked
until the account is confirmed via the token URL
- Confirmation link logs the user in on first use
- SMTP2GO configured as the outgoing mailer in k8s prod
New BandConfig entries for 902, 1296, 2304, 3456, 5760 MHz:
- All beneficial humidity effect (like 10 GHz)
- Near-zero gaseous absorption and rain attenuation
- Ranges: 902 MHz typical 400 km, 5760 MHz typical 220 km
- Same seasonal curves as 10 GHz (ducting-driven)
BandConfig.band_options/0 generates dropdown options from configs.
All pages (path, rover, submit) use centralized band_options instead
of hardcoded lists. Map page already used BandConfig.all_bands().
10 GHz remains the default on all pages.
- Extract scoring logic to Microwaveprop.Rover.Coverage for testability
- Two-phase computation: fast pass (distance + propagation) for all grids,
then SRTM terrain analysis for top 20 candidates only
- Cap search radius to 300 km to keep candidate count reasonable
- Run terrain analysis in Task with rescue/fallback for resilience
- Background Task doesn't block LiveView process
- 5 tests covering: ranked results, empty inputs, station details,
band range differences, field presence
When ASOS/RAOB/IEMRE APIs return empty data, store a stub record so
dedup checks (has_surface_observations?, has_sounding?, has_iemre_observation?)
see coverage and don't recreate the same jobs on future backfill runs.
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
- Migration renames table, column, all indexes, and FK constraints
- Contact schema now references "contacts" table
- TerrainProfile foreign key renamed from qso_id to contact_id
- All code and tests updated to use contact_id
- Radio.ensure_positions!/1 computes pos1/pos2/distance from grids
- Called in contact show page mount and enqueue_for_contact
- Migration backfills positions for existing contacts missing them
- Migration corrects enrichment statuses based on actual data presence
- Backfill dashboard includes contacts with grids but no positions
- New fields: hrrr_status, weather_status, terrain_status, iemre_status
with values: pending, queued, processing, complete, failed, unavailable
- EnrichmentStatus module defines valid state transitions
- Migration backfills: queued=true → complete, false → pending
- Workers set status on transitions (queued on enqueue, complete on finish)
- Show page reads status from contact struct, not computed dynamically
- Backfill dashboard queries status fields for accurate progress counts
- Remove cron-scheduled ContactWeatherEnqueueWorker (enrichment only on
submission, page view, or manual backfill)
- Partial indexes on status != 'complete' for fast unprocessed lookups
- Add timex ~> 3.7, downgrade gettext to ~> 0.26 for compatibility
- Parse ISO 8601, US dates (M/D/YYYY), AM/PM, 24h, compact formats
- 22 tests covering all accepted timestamp formats and edge cases
- Integration tests for CSV import with US date and space-separated formats
Adds a tabbed UI with single-contact form and CSV upload. Users can
download a sample CSV, upload their file, and get partial import with
per-row error reporting. Enrichment jobs enqueue for each imported contact.
Rename all modules, functions, variables, routes, and UI text from
qso/qsos to contact/contacts. Database table stays as "qsos" to avoid
migration. Add /qsos -> /contacts redirects for old URLs.
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
normalized distance — reduces noise from operator/equipment variation
