Runs BackfillEnqueueWorker every 30 minutes to pick up contacts with
pending/queued/failed enrichment status and enqueue missing jobs.
Before enqueueing, reconciles terrain contacts stuck in "queued" by
checking the terrain_profiles FK directly.
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
Per-pod queues divided by 3 to maintain sane cluster totals.
Lifeline rescue reduced to 10 min to recover OOM orphans faster.
Memory limit 2Gi → 4Gi for HRRR wgrib2 processing.
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
- Unified queue_info helper shows "(N jobs in queue)" on all spinners
- Solar index auto-enqueues when missing on contact page
- Freshness monitor disabled in dev
- Fix test stubs for solar client
- New fields: hrrr_status, weather_status, terrain_status, iemre_status
with values: pending, queued, processing, complete, failed, unavailable
- EnrichmentStatus module defines valid state transitions
- Migration backfills: queued=true → complete, false → pending
- Workers set status on transitions (queued on enqueue, complete on finish)
- Show page reads status from contact struct, not computed dynamically
- Backfill dashboard queries status fields for accurate progress counts
- Remove cron-scheduled ContactWeatherEnqueueWorker (enrichment only on
submission, page view, or manual backfill)
- Partial indexes on status != 'complete' for fast unprocessed lookups
- Cache raw GRIB2 byte ranges to ~/hrrr in dev (never deleted)
- mix hrrr_backfill re-fetches QSO-linked HRRR profiles with 13 levels
- Supports --all (re-fetch everything) and --limit N flags
- Groups by HRRR hour to batch requests, 500ms rate limit
- AWS archive has full HRRR history, same URL pattern as live feed
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
Rename all modules, functions, variables, routes, and UI text from
qso/qsos to contact/contacts. Database table stays as "qsos" to avoid
migration. Add /qsos -> /contacts redirects for old URLs.
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
ASOS re-scoring was upserting 458K records every 10 min while losing
the refractivity gradient (the key ducting indicator). HRRR hourly
provides all surface variables plus vertical profile data.
458K-record upsert held a connection for the entire transaction,
exceeding the 15s Postgrex timeout on prod. Set transaction timeout
to infinity and increase prod pool from 10 to 20.
Req default 15s receive_timeout too short for GRIB2 byte-range
downloads on prod. Also reduce HRRR queue concurrency from 20 to 5
to avoid NOAA rate limiting (was causing burst-then-stall pattern).
- Add enqueue_for_qso/1 to directly enqueue weather/HRRR/terrain/IEMRE
jobs for a single user-submitted QSO (no cron, no bulk processing)
- Submit flow calls enqueue_for_qso instead of generic enqueue worker
- Add enrichment queues to prod config for on-demand processing
- Guard against HRRR fill values in store_hrrr_profiles (fixes badarith)
- Filter QSOs without pos2 in build_terrain_jobs
Replaces the one-shot startup Task with a GenServer that checks every
5 minutes whether propagation scores are older than 2 hours. If stale,
enqueues a PropagationGridWorker job (with dedup check to avoid double
queuing). Disabled in test env to avoid SQL Sandbox conflicts.
Only run live scoring (propagation, commercial, solar) in prod.
Backfill queues (weather, hrrr, iemre, terrain, enqueue) and the
QsoWeatherEnqueueWorker cron are dev-only for historical enrichment.
New AsosAdjustmentWorker runs every 10 minutes:
- Fetches latest ASOS observations from all ~2900 US stations via
IEM bulk currents API (parallel fetch across 51 state networks)
- For each grid point within 75km of a reporting station, re-scores
using fresh ASOS data (temp, dewpoint, wind, sky, pressure, precip)
with HRRR refractivity gradient from the last hourly computation
- Pushes updated scores to the map via PubSub
Also stores HRRR profiles in the database during grid computation
so the data persists for reference and ASOS blending.
- Add precip_1h_in and wx_codes fields to ASOS surface observations
- Add IEMRE reanalysis schema for radar-derived hourly precipitation
- Add IemreFetchWorker with exponential backoff and idempotency
- Integrate IEMRE enqueue into cron weather backfill pipeline
- All existing QSOs marked iemre_queued=false for automatic backfill
Poll UBNT AirFiber radios (AF11X + AF60-LR) every 5 minutes via
net-snmp CLI, storing signal metrics in commercial_samples. Fetches
ASOS weather alongside each cycle for propagation correlation.
Includes 7 seeded link definitions, Oban cron worker, and net-snmp
in the Docker image.
