Commit graph

184 commits

Author SHA1 Message Date
ba0f1161a7 Beacon improvements: anon submit, notes, /map nav, admin backfill
- Anonymous users can submit beacons (held pending admin approval)
- Added notes field to beacons (textarea on form, shown on detail page)
- Added Beacons link to /map sidebar nav (mobile + desktop),
  removed stale Rover Planner link
- Moved /backfill to /admin/backfill under the admin live_session
- Beacon detail map zooms in two extra levels after fitBounds
2026-04-08 16:23:59 -05:00
f5b5af171e Add beacon keying field (on/off or FSK) 2026-04-08 16:13:35 -05:00
6476bc8045 Dedupe ERA5 fetch jobs by (lat, lon, valid_time)
The Era5FetchWorker now declares an Oban unique constraint on
{lat, lon, valid_time} for any job in available/scheduled/executing/
retryable, so two backfill runs targeting the same contact grid point
can no longer spawn parallel CDS requests for the same hour.

Because Oban OSS insert_all doesn't honor unique, ERA5 jobs are now
routed through Oban.insert/1 from ContactWeatherEnqueueWorker and the
era5_backfill mix task. Other worker types still use insert_all.
2026-04-08 15:52:53 -05:00
80f2725cd5 Beacon submission approval workflow
Non-admin users can now submit beacons but they are held in an
unapproved state until an admin approves them. Only approved beacons
appear in the public list; pending submissions are shown in a separate
admin-only section on /beacons with Approve and Delete actions. The
show page surfaces a pending badge and an admin-only Approve button
when viewing an unapproved beacon.

Also formats EIRP (mW) on the index page without scientific notation.
2026-04-08 15:42:32 -05:00
cda54e3514 Per-HRRR-cell beacon reception plot
Replace the idealised concentric-circle range rings with a realistic
per-HRRR-grid-cell reception map. For every 0.125° grid point within
the band's exceptional range, the estimator computes great-circle
distance, FSPL + atmospheric loss, and applies a cell-specific score
adjustment of (50 - score) * 0.3 dB — score 100 gives a 15 dB ducting
boost, score 0 a 15 dB penalty. Cells below the -145 dBm detection
floor are dropped.

The beacon map hook now renders each surviving cell as a 0.125°
filled rectangle with a tooltip showing tier, Rx dBm, distance, and
HRRR score, so the coverage footprint bulges where ducting conditions
are good and cuts off where they aren't, instead of being a perfect
circle. Falls back to score 50 for cells that don't have HRRR data
yet, so the plot is always useful.

Also fixes the prior all-grey-tiles regression by restoring an
explicit setView() at map creation and adding a post-mount
invalidateSize() for when the map is placed inside a flex container.
2026-04-08 13:43:01 -05:00
67e095bd47 Label beacon TX power as EIRP
The power_mw field represents the beacon's effective radiated power,
not just transmitter output. Relabel it as "TX power (EIRP)" on the
form and detail list, "EIRP (mW)" in the index, and drop the now-
meaningless tx_gain_dbi constant in RangeEstimate since the stored
value already includes antenna gain.
2026-04-08 13:32:32 -05:00
a5b3f1f3da Beacon detail map with range estimate and on_the_air flag
- Add on_the_air boolean to beacons (default true); surfaced as a
  checkbox on the form, a badge on the index, and in the detail list.
- Render a Leaflet map on the beacon show page with a marker at the
  beacon's lat/lon tooltipped with callsign + frequency. Marker is
  green when on air, gray when off.
- Compute and draw a reception-range estimate as concentric signal-
  strength rings. New Microwaveprop.Beacons.RangeEstimate solves a
  link budget (FSPL + O2/H2O absorption from BandConfig) at five RX
  thresholds (-100 to -145 dBm), then scales by 0.5 + score/100 from
  the latest Propagation.point_detail at the beacon's grid square, so
  current HRRR conditions shift the rings in or out.
- Re-enable the hourly PropagationGridWorker cron and freshness
  monitor in dev.exs so dev actually has HRRR-backed scores to feed
  the new estimator.
2026-04-08 13:29:58 -05:00
374f9b106a Beacon height in feet, auto-fill lat/lon from grid
- Rename beacons.height_m to height_ft; migration converts existing
  values (m * 3.28084). Form/list/show labels updated.
- Beacon changeset now fills lat/lon from the grid when they're blank
  (previously only the reverse worked). Lat/lon and grid inputs are no
  longer marked required in the form — enter one side and the other
  populates on blur via phx-change.
2026-04-08 12:55:10 -05:00
b656e015bf Beacon power in mW, settings tweaks
- Rename beacons.power_watts to power_mw (migration multiplies existing
  values by 1000); form/list/show all relabeled to "Power (mW)"
- Fix Add-monitor button alignment on /users/settings by rendering the
  label above and putting the input and button in a plain flex row so
  the input wrapper margin no longer offsets the button
- Extend the settings page re-auth window from 10 minutes to 24 hours
2026-04-08 12:45:35 -05:00
ee9275e0b9 Add /beacons CRUD and /users admin page
Beacons:
- Scaffolded with phx.gen.live then reworked so reads are public
  and mutations go through a live_session gated by the new
  :require_admin on_mount hook in UserAuth
- Beacon schema stores frequency (MHz), callsign, grid, lat/lon,
  power (W), and height above ground (m); grid auto-derives from
  lat/lon when left blank via Maidenhead.from_latlon
- Adds Maidenhead.from_latlon/3 so we can compute grids locally
  instead of hitting an external API

Users admin page:
- /users and /users/:id/edit (admin-only) for listing, editing
  (callsign/name/email/is_admin), and deleting other users
- Adds Accounts.list_users, admin_update_user, delete_user, and
  a dedicated admin_changeset on the User schema
- Nav gains a "Users" link for admins and a "Beacons" link for
  everyone
2026-04-08 12:01:45 -05:00
eaef5b0178 Add is_admin flag, auto-grant to graham@mcintire.me
Adds a boolean is_admin column (default false) and has the
registration changeset auto-set it to true when the email matches
graham@mcintire.me (case-insensitive). The migration also backfills
the flag for an existing row with that email.
2026-04-08 11:43:52 -05:00
355bed178d Add beacon monitor registration and fix prod SMTP
Users can now register beacon monitors on the settings page. Each
monitor gets a unique random token the remote program will use to
authenticate its reports. Name is the only user-supplied field for
now; more will be added as the monitor protocol is defined.

Also adds :gen_smtp to deps. Swoosh's SMTP adapter depends on
:mimemail which lives in that package, and production was 500'ing
when trying to send confirmation emails without it.
2026-04-08 11:30:28 -05:00
27170b5139 Merge auth links into main nav and lower password min to 8
- Register/Log in (or callsign/Settings/Log out) now live next to
  the Map/Path/Rover links in the main header instead of a separate
  top menu bar
- Add on_mount in UserAuth to assign current_scope on LiveView mount,
  and pass current_scope through to Layouts.app from every LiveView
- Drop the old top <ul> from root.html.heex
- Password minimum lowered from 12 to 8 characters
2026-04-08 10:39:51 -05:00
1d86e287b2 Add password auth with callsign + email confirmation
Generated Accounts context, User schema, and controllers via
phx.gen.auth. Adapted it for password-only login with required
email confirmation:

- Users have callsign (unique, uppercased), name, email, password
- Registration form fields: callsign, name, email, password, confirm
- Magic-link login path removed; login is email + password only
- After register, a confirmation email is sent and login is blocked
  until the account is confirmed via the token URL
- Confirmation link logs the user in on first use
- SMTP2GO configured as the outgoing mailer in k8s prod
2026-04-08 10:21:40 -05:00
75fd05a6d1 Simplify rover planner: remove grid toggle and coverage calculation 2026-04-08 09:06:09 -05:00
cfaef287e4 Add 902-5760 MHz bands, centralize band options
New BandConfig entries for 902, 1296, 2304, 3456, 5760 MHz:
- All beneficial humidity effect (like 10 GHz)
- Near-zero gaseous absorption and rain attenuation
- Ranges: 902 MHz typical 400 km, 5760 MHz typical 220 km
- Same seasonal curves as 10 GHz (ducting-driven)

BandConfig.band_options/0 generates dropdown options from configs.
All pages (path, rover, submit) use centralized band_options instead
of hardcoded lists. Map page already used BandConfig.all_bands().
10 GHz remains the default on all pages.
2026-04-07 16:39:25 -05:00
caa03fed53 Add grid toggle, use 0.25deg resolution for coverage scoring
- Toggle button for Maidenhead grid overlay (on by default)
- Coverage candidates now at 0.25° resolution (~28 km) instead of
  2°×1° Maidenhead grids, matching HRRR propagation grid density
- Coverage rectangles render as 0.25° cells for finer detail
- Grid overlay is separate from coverage coloring
2026-04-07 16:31:39 -05:00
6aec3eeea4 Extract Coverage module, fix computation, add tests
- Extract scoring logic to Microwaveprop.Rover.Coverage for testability
- Two-phase computation: fast pass (distance + propagation) for all grids,
  then SRTM terrain analysis for top 20 candidates only
- Cap search radius to 300 km to keep candidate count reasonable
- Run terrain analysis in Task with rescue/fallback for resilience
- Background Task doesn't block LiveView process
- 5 tests covering: ranked results, empty inputs, station details,
  band range differences, field presence
2026-04-07 16:08:03 -05:00
8a7719b953 Add /rover planner for microwave contest rovers
Interactive map-based tool for planning rover operating positions.
Rovers enter stationary stations they want to work, then click the
map to evaluate candidate operating grids.

Features:
- Add stationary stations by callsign or grid square
- Click map to add operating positions (snaps to Maidenhead grid)
- Async SRTM terrain analysis: each stop computes terrain profile
  to every station, shows CLEAR/BLOCKED verdict + diffraction loss
- Terrain lines on map: green=workable, red=blocked, dashed=marginal
- Propagation score and 18-hour forecast sparkline per stop
- Route summary: driving distance, unique grid-station pairs
- Band selector affects range limits and propagation scores
- Numbered waypoint markers with score-colored backgrounds

New files:
- Geo module (shared haversine/bearing, used by PathLive too)
- RoverLive with fullscreen map + sidebar
- RoverMap JS hook with grid overlay, station/stop markers, route line
- Nav links in header and map control panel
2026-04-07 14:17:48 -05:00
95351a2358 Add /path calculator page
LiveView page for analyzing microwave propagation between two points.
Features:
- Source/destination input: callsign (via gridmap.org API) or Maidenhead grid
- Band selector for all 8 microwave bands (10-241 GHz)
- Leaflet map showing path line (reuses ContactMap hook)
- Terrain elevation cross-section (reuses ElevationProfile hook)
- Link summary: distance, bearing, terrain verdict
- Loss budget: FSPL + O2 + H2O + rain + diffraction
- Propagation score with 10-factor breakdown
- Current atmospheric conditions from path-integrated HRRR

New CallsignClient for gridmap.org location lookup.
Navigation links added to main navbar and map control panel.
2026-04-07 13:25:04 -05:00
fbf24622fa Fix ERA5 CDS API endpoint for new v1 API
- Submit: /api/retrieve/v1/processes/{dataset}/execution
- Request body wrapped in {"inputs": ...}
- Poll: /api/retrieve/v1/jobs/{jobId}
- Status field: "status" not "state", values: accepted/running/successful/failed
- Download: GET {job_url}/results, follow asset href link
- Add ERA5 credentials to k8s secret manifest
2026-04-07 12:32:21 -05:00
e79e65db43 Add ERA5 to backfill dashboard and enrichment pipeline
- ERA5 checkbox on /backfill page alongside HRRR/weather/terrain/IEMRE
- BackfillEnqueueWorker handles era5 type (targets hrrr_status=unavailable)
- ContactWeatherEnqueueWorker.build_era5_jobs/1 creates ERA5 fetch jobs
  for path points without existing ERA5 profiles
- Progress bar shows ERA5 candidate count (HRRR-unavailable contacts)
- Status panel shows ERA5 candidates and fetched profile count
2026-04-07 12:24:18 -05:00
dea407c8ca Add ERA5 reanalysis and RTMA data sources
ERA5 (Copernicus CDS API):
- Era5Profile schema matching HRRR profile structure for interop
- Era5Client with async job submission, polling, GRIB2 download
- Era5FetchWorker (Oban queue: era5, max_attempts: 5)
- Unified lookup: Weather.best_profile_for_contact/1 tries HRRR
  first, falls back to ERA5 for pre-2014 contacts

RTMA (NOAA S3, 2.5km/15-min):
- RtmaObservation schema for surface-only fields
- RtmaClient with byte-range GRIB2 requests (same pattern as HRRR)
- RtmaFetchWorker (Oban queue: rtma, max_attempts: 10)
- Weather.find_nearest_rtma/3 for surface condition lookup

Both queues added to production Oban config (2 workers each).
2026-04-07 12:04:16 -05:00
7652e41971 Path-integrated HRRR scoring across pos1/mid/pos2
Contact scoring now uses all HRRR profiles along the path instead of
just pos1. Aggregation strategy:
- Best along path for beneficial factors (refractivity, pressure)
- Worst along path for harmful factors (rain, wind)
- Average for neutral factors (temp, dewpoint, PWAT, BL depth)

Scorer.path_integrated_conditions/2 merges multiple profiles into a
single conditions map. Falls back gracefully to single-profile scoring
when only one profile is available.
2026-04-07 12:04:16 -05:00
2eefe64c9b Update seasonal scores from sounding data, document commercial link patterns
Soundings (10,935): October has strongest gradients (-350 N/km) and
65.7% ducting, raised seasonal score 82→88. February ducting (20.2%)
exceeds January (14.8%), raised 32→40.

Commercial links (12,068 samples): 68 GHz shows 3.9 dB diurnal swing,
11 GHz shows 1.7 dB with inverted multipath pattern, 24 GHz remarkably
stable at 0.9 dB. Diurnal sensitivity is non-monotonic with frequency.

RAOB gradients avg -265 N/km vs HRRR -107 (2.5x stronger) — confirms
HRRR misses thin surface ducts that soundings resolve.
2026-04-07 11:45:44 -05:00
3cdd132f24 Data-driven algorithm refinements from full dataset analysis
Analyzed 37,925 HRRR-matched contacts from 58,367 total.

- Remove shallow BL bonus (score 82 for HPBL < 300m): data shows
  medium BL (1000-2000m) produces longest contacts (222 km avg),
  not shallow (210 km). Refractivity fallback now uses default score.

- Refine pressure scoring bins: add <980 mb tier (score 88), steeper
  gradient from low to high. Contacts at <970 mb avg 242.7 km vs
  184.1 km at 990-1000 mb.

- Update algo.md calibration stats (58,367 contacts, 41M HRRR
  profiles, 58,361 terrain profiles) and document dataset bias
  (99.5% Aug-Sep).

- Create updates.md with full binned analysis tables for all factors.
2026-04-07 11:15:49 -05:00
a774e98350 Smooth viewshed boundary to remove SRTM elevation spikes
Apply 5-point circular moving average to reach_km values before
rendering the boundary polygon. Eliminates sharp spikes caused by
isolated terrain anomalies in SRTM data.
2026-04-07 07:42:32 -05:00
e03b9fbc0e Add periodic enrichment backfill cron and terrain reconciliation
Runs BackfillEnqueueWorker every 30 minutes to pick up contacts with
pending/queued/failed enrichment status and enqueue missing jobs.
Before enqueueing, reconciles terrain contacts stuck in "queued" by
checking the terrain_profiles FK directly.
2026-04-07 07:36:57 -05:00
79130a39e8 Mark terrain status complete when profile already exists
TerrainProfileWorker early-exited with :ok when a profile existed
but never updated the contact status, leaving contacts stuck at queued.
2026-04-06 16:54:29 -05:00
646467cf4d Store stub records on empty weather fetches to prevent infinite backfill retries
When ASOS/RAOB/IEMRE APIs return empty data, store a stub record so
dedup checks (has_surface_observations?, has_sounding?, has_iemre_observation?)
see coverage and don't recreate the same jobs on future backfill runs.
2026-04-06 16:10:39 -05:00
4ba7269e57 Prioritize pending contacts over queued in backfill
Queued contacts just need reconciliation (check if data exists),
while pending contacts need actual HRRR fetches. Process pending
first so real work happens before reconciliation.
2026-04-06 11:48:55 -05:00
e99e585b66 Add type filter to backfill enqueue
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
2026-04-06 11:42:26 -05:00
0e885001bc Widen HRRR profile spatial matching from 0.05° to 0.07°
The propagation grid stores profiles at 0.125° intervals, so contacts
up to 0.0625° from a grid point would miss with the old 0.05° tolerance.
This caused has_hrrr_profile? and find_nearest_hrrr to fail matching
against existing data, leaving contacts incorrectly stuck as pending.
2026-04-06 10:40:25 -05:00
eeb92bcc19 Fix HRRR enrichment status stuck at queued/pending
enqueue_for_contact unconditionally marked contacts as :queued even
when no HRRR jobs were needed (profiles already exist). Workers never
updated contact status to :complete after storing profiles. Contacts
got stuck and were never re-processed.

- Mark :complete when no jobs generated (data already exists)
- Include :queued in backfill enrichable states for reconciliation
2026-04-06 10:24:56 -05:00
2de94e2318 Add Erlang clustering via libcluster and expose LiveDashboard
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
2026-04-06 09:46:48 -05:00
8d008ba934
Add logging to WeatherFetchWorker for ASOS and RAOB fetches 2026-04-05 10:26:10 -05:00
7fd8f3165c
Add deterministic tiebreaker to contacts sort order 2026-04-05 09:27:34 -05:00
16883591b4
Database performance fixes and async backfill enqueue
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
  non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
2026-04-04 19:19:18 -05:00
dc66252fdc
Reduce dead tuples by using conditional upserts and skip-on-conflict
- HRRR, IEMRE, terrain: on_conflict: :nothing (immutable data)
- Surface obs, soundings, solar: conditional WHERE (only update when values differ)
- set_enrichment_status!: skip rows already at target status
2026-04-04 17:32:57 -05:00
e107706915
Show dead tuple counts on backfill dashboard, fix table visibility during vacuum
Removed n_live_tup > 0 filter that hid tables when pg_stats reports 0 rows
during VACUUM. Added dead tuple column with warning highlight to monitor
vacuum progress. Also includes Styler reformatting and unused dep cleanup.
2026-04-04 10:45:06 -05:00
23aa2786ab
Skip updating unchanged scores to reduce dead tuples, tune autovacuum
ON CONFLICT now only replaces score/factors when the score actually changed,
avoiding dead tuple generation for the ~80% of grid points that don't change
between consecutive HRRR runs.

Migration sets aggressive autovacuum on propagation_scores: zero cost delay,
2000 cost limit, 1% scale factor. The table was at 72GB with 108M dead rows
because default autovacuum couldn't keep pace with 14M upserts per hour.
2026-04-04 09:53:45 -05:00
4759c65809
Support two-callsign search on contacts page to find contacts between a pair 2026-04-04 09:29:32 -05:00
9cbf8b91f4
Derive upper-air weather layers from HRRR profiles in weather grid queries 2026-04-03 16:52:01 -05:00
489a886b0e
Add WeatherLayers module for deriving upper-air map fields from HRRR profiles 2026-04-03 16:49:17 -05:00
10390f3d07
Broadcast weather:updated PubSub after HRRR profiles stored 2026-04-03 15:10:16 -05:00
0dd695c574
Add weather grid query functions for /weather map page 2026-04-03 15:08:21 -05:00
411951147a
Use ON CONFLICT DO NOTHING for HRRR backfill to prevent table bloat 2026-04-03 13:29:15 -05:00
8d84bd2d18
Prune propagation scores per valid_time to avoid massive single DELETE 2026-04-03 10:46:17 -05:00
33490d37d0
Chunk batch inserts to avoid Postgres parameter limit, add ANALYZE migration 2026-04-03 09:34:02 -05:00
ebeaaa06e4
Use batch insert for HRRR profiles, tune Postgres for 16GB host 2026-04-03 09:12:24 -05:00