The ML model undervalues conditions outside Aug/Sep training data
(e.g. April with excellent factors scored 37/100). Algorithm's
physics-based factors handle unseen seasons correctly.
- Algorithm is primary scorer, ML infrastructure kept for iteration
- Remove unused ML grid worker code path
- Add client-side propagation reach: BFS flood-fill from clicked point
through contiguous cells with score >= 50, drawn as convex hull polygon
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
normalized distance — reduces noise from operator/equipment variation
Features: HRRR conditions averaged at both QSO endpoints + solar time
Target: distance_km normalized per-band (distance / p99_range, capped at 1.0)
This trains on actual propagation outcomes from 57K+ QSOs, not the
hand-tuned algorithm output.
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
- point_detail response includes forecast array (score per valid_time)
- SVG sparkline shows score trend across all forecast hours
- Trend indicator: Improving/Declining/Steady based on first vs last score
- Add covering index (band_mhz, lat, lon, valid_time) INCLUDE (score) for
point_forecast queries
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2)
- PropagationGridWorker fetches all 19 forecast hours per run
- Propagation.scores_at/3 queries scores at specific valid_time
- Propagation.available_valid_times/1 returns all forecast times for timeline
- Pruning keeps scores with valid_time >= now - 2h (forecast-aware)
- MapLive: select_time event, timeline data pushed to JS
- JS: forecast timeline bar at bottom of map with clickable hour buttons
- PubSub broadcast sends list of valid_times instead of single time
- Add phx-update="ignore" to detail-panel so LiveView patches don't wipe it
- Re-request point_detail after scores update to refresh with new data
- Range circles persist across updates since they're on a separate layer
Previous thresholds (-500 to -60) were calibrated for radiosonde data.
HRRR profiles have coarser vertical resolution, with gradients clustering
between -40 and -130 N/km (median -70). Nearly all grid points were
falling through to the default score of 42, wasting the refractivity
factor. New thresholds (-200 to -40) spread across HRRR percentiles.
- Replace piecewise knife-edge loss with P.526-16 Eq. 31 single formula
- Fix diffraction parameter ν to use standard formula instead of ad-hoc approximation
- Implement Deygout 3-edge method for multiple obstacle diffraction
- Add dynamic k-factor from HRRR refractivity gradient (falls back to 4/3)
- Terrain worker now looks up nearest HRRR profile for atmospheric correction
- Update algo.md with P.526-16 methods and k-factor table
- Fix pre-existing map_live_test antenna height default (33 ft, not 8)
Deletes grid-aligned profiles (0.125 degree) older than 48 hours
while preserving QSO-linked profiles at arbitrary positions.
Called after each PropagationGridWorker run.
458K-record upsert held a connection for the entire transaction,
exceeding the 15s Postgrex timeout on prod. Set transaction timeout
to infinity and increase prod pool from 10 to 20.
Req default 15s receive_timeout too short for GRIB2 byte-range
downloads on prod. Also reduce HRRR queue concurrency from 20 to 5
to avoid NOAA rate limiting (was causing burst-then-stall pattern).
- Viewshed uses propagation score to set max range (ducting/NLOS distance)
- Terrain checked within radio horizon; clear paths get full atmospheric range
- Detail panel moves from map popup to sidebar for better visibility
- Panel shows instantly on click with loading state
- Dark opaque background for readability over map
- Factor analysis section explains each score with contextual descriptions
Replace generic range circles with actual LOS coverage polygon computed
from SRTM elevation data. Casts 180 rays (every 2 degrees) from the
clicked point, runs Fresnel/diffraction analysis on each, and renders
the reachable area as a Leaflet polygon.
- Viewshed module with haversine forward, terrain sweep, async compute
- Antenna height control (default 8 ft) in map panel
- LiveView start_async/handle_async for non-blocking computation
- Remove signal icon from band selector
algo.md was read at runtime via File.read! but isn't included in
the release. Use @external_resource + compile-time module attribute
to bake the HTML into the module.
- Add enqueue_for_qso/1 to directly enqueue weather/HRRR/terrain/IEMRE
jobs for a single user-submitted QSO (no cron, no bulk processing)
- Submit flow calls enqueue_for_qso instead of generic enqueue worker
- Add enrichment queues to prod config for on-demand processing
- Guard against HRRR fill values in store_hrrr_profiles (fixes badarith)
- Filter QSOs without pos2 in build_terrain_jobs
Replaces the one-shot startup Task with a GenServer that checks every
5 minutes whether propagation scores are older than 2 hours. If stale,
enqueues a PropagationGridWorker job (with dedup check to avoid double
queuing). Disabled in test env to avoid SQL Sandbox conflicts.
On app start, check if latest scores are older than 2 hours. If so,
immediately enqueue a PropagationGridWorker job. Covers app restarts,
deploys, and missed cron ticks.
Skip grid points where surface_temp_c or surface_dewpoint_c are
physically impossible (< -80°C or > 60°C). HRRR returns -273.15
(absolute zero) for ocean/missing points which caused division by
zero in absolute_humidity calculation.
- Document Finding 11: regional QSO performance by Maidenhead field
- Explain why regional weight adjustments are not justified
- Fix algo.md table formatting (blank lines before tables for Earmark)
- Change page title from Phoenix Framework to North Texas Microwave Society
- Wrap upsert_scores in Repo.transaction for all-or-nothing visibility
- Prune scores older than the 2 most recent valid_times after each upsert
- Add band-specific latest_valid_time/1 to eliminate N+1 query
- Add require Logger to Propagation module
- Remove factors from heatmap scores, fetch on-demand via point_detail
- Pre-encode initial scores JSON once instead of per-render
- Don't store scores in socket assigns (reduce memory)
- Pre-calculate RGBA color lookup table for canvas rendering
- Batch fillStyle changes, hoist math out of inner loop
- Show topbar immediately on data requests (no 300ms delay)
- Port grid square rendering from gridmap-web with toggle control
- Consolidate band selector, grid toggle, data timestamp, and links
into unified control panel
- Add "How this works" (/algo) and "Submit a QSO" (/submit) links
- Add back-to-map link on algo page
- Darken range circles for visibility over propagation overlay
- Fix popup blocking double-click zoom with pointer-events passthrough
- Prevent control panel from intercepting map click/scroll events
- Scores now include factors and valid_time in the viewport query,
eliminating the server round-trip for popups
- Click shows two range circles: solid inner (typical range) and
dashed outer (max estimated range), colored by score tier
- Circles disappear when popup closes
- Band info pushed to client on band switch for accurate range estimates
Click anywhere on the propagation map to see a detailed popup with:
- Overall score and tier label with color
- Estimated range for the selected band (CW mode)
- All 9 scoring factors with visual bar charts, individual scores,
and weight percentages
- Grid point coordinates and data timestamp
Factors are displayed in weight order so users can immediately see
which atmospheric conditions are driving the prediction.
