Commit graph

109 commits

Author SHA1 Message Date
c12f8cf5ed
Use local solar time for time-of-day scoring, add PWAT factor and pressure refinements
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
2026-04-01 08:58:21 -05:00
b7fc195e82
Ensure stale scores get pruned even when HRRR fetch fails 2026-04-01 08:36:14 -05:00
4bf9d0b4b0
Add mix propagation.analyze task for QSO-HRRR correlation analysis 2026-04-01 08:34:04 -05:00
14e86c4ddb
Filter forecast graph to only show current and future hours 2026-04-01 08:14:29 -05:00
b11e0bac84
Always show at least the most recent valid_time in timeline 2026-03-31 17:49:19 -05:00
4151f7d314
Filter past hours from forecast timeline (30-min grace for current hour) 2026-03-31 17:47:16 -05:00
6dff6372ab
Default forecast timeline selection to closest hour to now 2026-03-31 17:45:54 -05:00
e028de51ae
Tighten HRRR grid profile retention from 48h to 24h 2026-03-31 17:20:46 -05:00
fe5228b306
Prevent duplicate PropagationGridWorker jobs via Oban unique constraint
Worker now uses unique: [period: 3600, states: [:available, :scheduled,
:executing, :retryable]] so Oban atomically prevents duplicates.
FreshnessMonitor simplified — no longer does manual oban_jobs query.
2026-03-31 17:16:40 -05:00
fe150adf77
Change antenna label to 'Antenna height' 2026-03-31 16:55:40 -05:00
6145f80c22
Add forecast sparkline graph to point detail panel
- point_detail response includes forecast array (score per valid_time)
- SVG sparkline shows score trend across all forecast hours
- Trend indicator: Improving/Declining/Steady based on first vs last score
- Add covering index (band_mhz, lat, lon, valid_time) INCLUDE (score) for
  point_forecast queries
2026-03-31 16:47:52 -05:00
e82e631135
HRRR forecast hours (f00-f18) with timeline map UI
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2)
- PropagationGridWorker fetches all 19 forecast hours per run
- Propagation.scores_at/3 queries scores at specific valid_time
- Propagation.available_valid_times/1 returns all forecast times for timeline
- Pruning keeps scores with valid_time >= now - 2h (forecast-aware)
- MapLive: select_time event, timeline data pushed to JS
- JS: forecast timeline bar at bottom of map with clickable hour buttons
- PubSub broadcast sends list of valid_times instead of single time
2026-03-31 16:44:47 -05:00
c775eb2611
Preserve detail panel and range circle across score updates
- Add phx-update="ignore" to detail-panel so LiveView patches don't wipe it
- Re-request point_detail after scores update to refresh with new data
- Range circles persist across updates since they're on a separate layer
2026-03-31 16:28:38 -05:00
8949920b7f
Add Nx/Axon/EXLA ML model skeleton for propagation prediction
13-feature feed-forward network (atmospheric + temporal + frequency).
Includes build, init, predict, encode_features, save/load to disk.
Model weights saved to priv/models/propagation_v1.nx (gitignored).
Not yet trained — scaffolding only.
2026-03-31 16:26:34 -05:00
77db483a27
Add timing instrumentation to PropagationGridWorker
Logs duration for each phase: HRRR fetch, profile storage, score
computation, and upsert, plus total elapsed time.
2026-03-31 16:18:10 -05:00
c9112b9280
Recalibrate refractivity thresholds for HRRR gradient distribution
Previous thresholds (-500 to -60) were calibrated for radiosonde data.
HRRR profiles have coarser vertical resolution, with gradients clustering
between -40 and -130 N/km (median -70). Nearly all grid points were
falling through to the default score of 42, wasting the refractivity
factor. New thresholds (-200 to -40) spread across HRRR percentiles.
2026-03-31 16:08:15 -05:00
49cbe6789c
Implement ITU-R P.526-16 terrain diffraction model
- Replace piecewise knife-edge loss with P.526-16 Eq. 31 single formula
- Fix diffraction parameter ν to use standard formula instead of ad-hoc approximation
- Implement Deygout 3-edge method for multiple obstacle diffraction
- Add dynamic k-factor from HRRR refractivity gradient (falls back to 4/3)
- Terrain worker now looks up nearest HRRR profile for atmospheric correction
- Update algo.md with P.526-16 methods and k-factor table
- Fix pre-existing map_live_test antenna height default (33 ft, not 8)
2026-03-31 16:04:23 -05:00
6d103d7cc4
Prune HRRR grid profiles older than 48h after each scoring run
Deletes grid-aligned profiles (0.125 degree) older than 48 hours
while preserving QSO-linked profiles at arbitrary positions.
Called after each PropagationGridWorker run.
2026-03-31 16:03:14 -05:00
ee6975bc39
Fix ASOS upsert timeout: infinite tx timeout, bump prod pool to 20
458K-record upsert held a connection for the entire transaction,
exceeding the 15s Postgrex timeout on prod. Set transaction timeout
to infinity and increase prod pool from 10 to 20.
2026-03-31 15:54:19 -05:00
ffae4b52e7
Fix HRRR fetch timeouts: 60s receive_timeout, reduce queue to 5
Req default 15s receive_timeout too short for GRIB2 byte-range
downloads on prod. Also reduce HRRR queue concurrency from 20 to 5
to avoid NOAA rate limiting (was causing burst-then-stall pattern).
2026-03-31 15:37:21 -05:00
ff3a879dff
Rework viewshed: score-aware NLOS range, sidebar detail panel, factor explanations
- Viewshed uses propagation score to set max range (ducting/NLOS distance)
- Terrain checked within radio horizon; clear paths get full atmospheric range
- Detail panel moves from map popup to sidebar for better visibility
- Panel shows instantly on click with loading state
- Dark opaque background for readability over map
- Factor analysis section explains each score with contextual descriptions
2026-03-31 13:49:42 -05:00
6e5409bcd9
Update esbuild 0.27.4 and tailwind 4.2.2, add viewshed logging 2026-03-31 13:15:56 -05:00
5eaa55448e
Add terrain-aware viewshed on map click
Replace generic range circles with actual LOS coverage polygon computed
from SRTM elevation data. Casts 180 rays (every 2 degrees) from the
clicked point, runs Fresnel/diffraction analysis on each, and renders
the reachable area as a Leaflet polygon.

- Viewshed module with haversine forward, terrain sweep, async compute
- Antenna height control (default 8 ft) in map panel
- LiveView start_async/handle_async for non-blocking computation
- Remove signal icon from band selector
2026-03-31 13:06:01 -05:00
166f95e29d
Embed algo.md at compile time to fix prod 500
algo.md was read at runtime via File.read! but isn't included in
the release. Use @external_resource + compile-time module attribute
to bake the HTML into the module.
2026-03-31 12:51:40 -05:00
40d1fa03aa
QSO submission triggers enrichment directly, fix prod badarith crash
- Add enqueue_for_qso/1 to directly enqueue weather/HRRR/terrain/IEMRE
  jobs for a single user-submitted QSO (no cron, no bulk processing)
- Submit flow calls enqueue_for_qso instead of generic enqueue worker
- Add enrichment queues to prod config for on-demand processing
- Guard against HRRR fill values in store_hrrr_profiles (fixes badarith)
- Filter QSOs without pos2 in build_terrain_jobs
2026-03-31 12:29:39 -05:00
105298fc3f
Add FreshnessMonitor GenServer to detect and fill stale propagation data
Replaces the one-shot startup Task with a GenServer that checks every
5 minutes whether propagation scores are older than 2 hours. If stale,
enqueues a PropagationGridWorker job (with dedup check to avoid double
queuing). Disabled in test env to avoid SQL Sandbox conflicts.
2026-03-31 12:08:51 -05:00
45fb9736fa
Log client IP from X-Forwarded-For in request logs
Add RemoteIp plug that extracts real client IP from X-Forwarded-For
header (set by nginx/dokku proxy) and adds it to Logger metadata.
2026-03-31 12:05:05 -05:00
c709fb5c32
Enqueue propagation grid worker on startup if scores are stale
On app start, check if latest scores are older than 2 hours. If so,
immediately enqueue a PropagationGridWorker job. Covers app restarts,
deploys, and missed cron ticks.
2026-03-31 11:56:41 -05:00
4301f48d27
Guard against HRRR fill values in grid scoring
Skip grid points where surface_temp_c or surface_dewpoint_c are
physically impossible (< -80°C or > 60°C). HRRR returns -273.15
(absolute zero) for ocean/missing points which caused division by
zero in absolute_humidity calculation.
2026-03-31 11:07:26 -05:00
a18cf48493
Add regional analysis to algo.md, update site title to NTMS
- Document Finding 11: regional QSO performance by Maidenhead field
- Explain why regional weight adjustments are not justified
- Fix algo.md table formatting (blank lines before tables for Earmark)
- Change page title from Phoenix Framework to North Texas Microwave Society
2026-03-31 10:47:39 -05:00
a6b4ac1965
Increase weather queue to 20 and iemre to 10 for faster backfill 2026-03-31 10:41:49 -05:00
3874fc173d
Atomic score upserts and auto-prune old propagation data
- Wrap upsert_scores in Repo.transaction for all-or-nothing visibility
- Prune scores older than the 2 most recent valid_times after each upsert
- Add band-specific latest_valid_time/1 to eliminate N+1 query
- Add require Logger to Propagation module
2026-03-31 10:35:59 -05:00
1aebd7f1cc
Optimize map performance: reduce payload 75%, faster rendering
- Remove factors from heatmap scores, fetch on-demand via point_detail
- Pre-encode initial scores JSON once instead of per-render
- Don't store scores in socket assigns (reduce memory)
- Pre-calculate RGBA color lookup table for canvas rendering
- Batch fillStyle changes, hoist math out of inner loop
- Show topbar immediately on data requests (no 300ms delay)
2026-03-31 10:26:20 -05:00
2f2f3388cd
Rename 'How this works' to 'Scoring Algorithm' 2026-03-31 10:24:04 -05:00
ccac7a5020
Add NTMS title to map panel, make range circles non-interactive 2026-03-31 10:17:16 -05:00
c3a5cf4b1a
Add Maidenhead grid overlay, consolidate map controls, improve UX
- Port grid square rendering from gridmap-web with toggle control
- Consolidate band selector, grid toggle, data timestamp, and links
  into unified control panel
- Add "How this works" (/algo) and "Submit a QSO" (/submit) links
- Add back-to-map link on algo page
- Darken range circles for visibility over propagation overlay
- Fix popup blocking double-click zoom with pointer-events passthrough
- Prevent control panel from intercepting map click/scroll events
2026-03-31 10:15:21 -05:00
14e1078bf9
Fall back to 1000mb level data when surface fields are missing (fills coastal gaps) 2026-03-31 09:58:16 -05:00
2e216d4244
Make point detail popup instant with client-side lookup and range circles
- Scores now include factors and valid_time in the viewport query,
  eliminating the server round-trip for popups
- Click shows two range circles: solid inner (typical range) and
  dashed outer (max estimated range), colored by score tier
- Circles disappear when popup closes
- Band info pushed to client on band switch for accurate range estimates
2026-03-31 09:50:48 -05:00
0c295c1558
Add click-to-inspect factor breakdown popup on map
Click anywhere on the propagation map to see a detailed popup with:
- Overall score and tier label with color
- Estimated range for the selected band (CW mode)
- All 9 scoring factors with visual bar charts, individual scores,
  and weight percentages
- Grid point coordinates and data timestamp

Factors are displayed in weight order so users can immediately see
which atmospheric conditions are driving the prediction.
2026-03-31 09:45:50 -05:00
d5140267b1
Show topbar loading indicator while scores are loading 2026-03-31 09:30:20 -05:00
24840c48d4
Show 'Latest data: date time (Xm ago)' in map badge 2026-03-31 09:12:32 -05:00
e0d9900608
Add real-time ASOS adjustments between hourly HRRR updates
New AsosAdjustmentWorker runs every 10 minutes:
- Fetches latest ASOS observations from all ~2900 US stations via
  IEM bulk currents API (parallel fetch across 51 state networks)
- For each grid point within 75km of a reporting station, re-scores
  using fresh ASOS data (temp, dewpoint, wind, sky, pressure, precip)
  with HRRR refractivity gradient from the last hourly computation
- Pushes updated scores to the map via PubSub

Also stores HRRR profiles in the database during grid computation
so the data persists for reference and ASOS blending.
2026-03-31 09:10:12 -05:00
b87572aa81
Close band dropdown after selection by shifting focus to map 2026-03-31 09:04:31 -05:00
97d4bd9372
Only send visible map scores based on viewport bounds
The JS hook sends map bounds on load and on pan/zoom. The server
queries only scores within those bounds, dramatically reducing the
payload for band switches and map updates. At zoom 7 (DFW area)
this sends ~2k scores instead of ~95k.
2026-03-31 09:01:48 -05:00
321133c3cb
Push map updates via PubSub when propagation scores are computed 2026-03-31 08:57:51 -05:00
4e2d74e21c
Build wgrib2 in Docker with cached build stage
Add a separate wgrib2-builder stage that compiles wgrib2 from source.
Docker layer caching means this only builds once — subsequent deploys
reuse the cached layer. The binary is copied into the final runtime
image. Also fixed wgrib2 path lookup to be runtime instead of
compile-time so it works in the Docker build pipeline.
2026-03-31 08:55:09 -05:00
66639e3717
Increase grid to 0.125 degrees with smooth canvas overlay
Replace circle markers with a canvas tile layer that renders smooth,
flowing colored regions using bilinear interpolation between grid
points. Colors interpolate between tiers for gradients. ~95k grid
points at 0.125 degree resolution with wgrib2 extraction.
2026-03-31 08:52:07 -05:00
becdf89978
Truncate valid_time microseconds for Ecto utc_datetime 2026-03-31 08:48:32 -05:00
22cb055269
Skip grid points with missing surface data (ocean/outside-domain) 2026-03-31 08:36:17 -05:00
b7446e83f5
Use wgrib2 for HRRR grid extraction (7s vs 80+ minutes)
Add Wgrib2 module that shells out to wgrib2 binary for fast GRIB2
grid extraction using -lola (nearest-neighbor to regular lat-lon grid).
Falls back to pure-Elixir decoder if wgrib2 is not installed.

Also: parallel GRIB2 range downloads, merge adjacent byte ranges,
skip corrupt messages instead of failing, pressure fetch is optional.
2026-03-31 08:33:29 -05:00