- Cache raw GRIB2 byte ranges to ~/hrrr in dev (never deleted)
- mix hrrr_backfill re-fetches QSO-linked HRRR profiles with 13 levels
- Supports --all (re-fetch everything) and --limit N flags
- Groups by HRRR hour to batch requests, 500ms rate limit
- AWS archive has full HRRR history, same URL pattern as live feed
Fetch every 25mb from 1000-700mb (13 levels, up from 8). Gives ~80m
vertical spacing near the surface, enough to resolve ducting layers
that were previously invisible between the coarse 250m level spacing.
- Add timex ~> 3.7, downgrade gettext to ~> 0.26 for compatibility
- Parse ISO 8601, US dates (M/D/YYYY), AM/PM, 24h, compact formats
- 22 tests covering all accepted timestamp formats and edge cases
- Integration tests for CSV import with US date and space-separated formats
Parse US dates (6/15/2024 2:30 PM), 24h (06/15/2024 14:30),
dashes (06-15-2024 14:30), compact (20240615 1430), date-only
(2024-06-15), and ISO 8601 — all normalized to UTC before insert.
Adds a tabbed UI with single-contact form and CSV upload. Users can
download a sample CSV, upload their file, and get partial import with
per-row error reporting. Enrichment jobs enqueue for each imported contact.
- Text summary explaining likely propagation mechanism (ducting,
enhanced refraction, diffraction, troposcatter) based on terrain,
HRRR, and sounding data
- 10-factor scoring table with per-factor notes, weights, contributions
- Composite score with tier badge
- Grid squares added back to path info line
- Analysis positioned between elevation chart and terrain profile
- Mark duct closest to endpoint elevations as likely signal path
- Likely duct drawn with solid borders, bold label, stronger fill
- Other ducts shown dimmed with dashed borders
- Fix path info layout: table columns prevent El wrapping
- Surface observations as compact table, limited to 5 nearest stations
- Use all HRRR profiles along path (start, mid, end) for duct detection
- Fall back to sounding ducts when HRRR too coarse to detect layers
- Merge overlapping ducts, show top 3 strongest with source labels
- Limit surface observations to 5 closest stations along path
- Compact solar conditions as inline row
- Compact band/mode/distance info line
- Fix button text to "Flag as Invalid"
- Overlay HRRR duct layers on elevation profile chart (green bands)
- Add flagged_invalid field to contacts with toggle button on detail page
- Fix prune_old_grid_profiles to only delete 0.125° grid profiles,
preserving QSO-linked HRRR data at arbitrary positions
- Auto-enqueue HRRR fetch when viewing contact without HRRR data (<48h)
- Compact solar conditions and contact details layout
- Add band/mode/distance info line to elevation profile section
- Leaflet map on contact detail pages showing both station locations
- SRTM elevation profile with Chart.js: terrain, LOS, Fresnel zone
- Path info: distance, azimuth, elevation angle in both directions
- Compact 4-column grid layout for contact details
- Contact map page with all contacts plotted
- Rename QSOs to Contacts in page header
- Wrap algo page in standard layout
- All menus and UI text now say Contacts instead of QSOs
- Add 68, 122, 134, 241 GHz bands to submit form and validation
- Add info box on submit page explaining why contacts matter
- Larger submit button with icon
- Make HRRR partition migration idempotent for partial re-runs
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
Rename all modules, functions, variables, routes, and UI text from
qso/qsos to contact/contacts. Database table stays as "qsos" to avoid
migration. Add /qsos -> /contacts redirects for old URLs.
The ML model undervalues conditions outside Aug/Sep training data
(e.g. April with excellent factors scored 37/100). Algorithm's
physics-based factors handle unseen seasons correctly.
- Algorithm is primary scorer, ML infrastructure kept for iteration
- Remove unused ML grid worker code path
- Add client-side propagation reach: BFS flood-fill from clicked point
through contiguous cells with score >= 50, drawn as convex hull polygon
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
normalized distance — reduces noise from operator/equipment variation
Features: HRRR conditions averaged at both QSO endpoints + solar time
Target: distance_km normalized per-band (distance / p99_range, capped at 1.0)
This trains on actual propagation outcomes from 57K+ QSOs, not the
hand-tuned algorithm output.
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
