Commit graph

49 commits

Author SHA1 Message Date
70e65ba034
refactor(map): drop the 7-day outlook strip, fix missing Analysis, trim timeline
- Remove the 7-day outlook strip and daily_outlook_at/3. It was
  derived from point_forecast, which is capped at HRRR's 18-hour
  horizon, so it never had 7 days of data to show.
- factors_for: fall back to the nearest persisted analysis profile
  at or before the requested time. Only f00 hours persist profiles,
  so clicking any other hour used to yield an empty Analysis and
  factor table. After the recent cursor fix lands 'Now' on a
  forecast hour more often, this regressed further.
- available_valid_times/1: cap the forward horizon to 18h from now.
  Leftover valid_times from prior cycles used to pile on the
  timeline without adding information.
2026-04-19 08:17:47 -05:00
4e6c87eca2
feat(telemetry): broaden Instrument span coverage
Adds spans to 15 previously-unmeasured hot paths so every question we
might ask while tuning has a histogram to answer it:

External I/O:
- iem.fetch_iemre (gridded weather reanalysis)
- mrms.list_latest / mrms.download (precip radar)
- rtma.fetch_observation
- ncei.fetch_metar (historical 5-min METAR backfill)
- solar.fetch_indices (GFZ solar indices)
- swpc.fetch (SWPC Kp/F10.7/X-ray)
- giro.fetch (ionosonde)
- qrz.request, geocoder.geocode (callsign enrichment)
- srtm.download_tile (terrain tile download + gunzip)
- hrrr.download_grib_ranges (parallel byte-range fetch phase)

Subprocess:
- wgrib2.extract_grid / extract_grid_from_file / extract_grid_from_file_mapped

LiveView hot paths:
- propagation.scores_at (map score fetch + cache hit/miss counter)
- propagation.point_forecast (sparkline)
- propagation.point_detail (click-to-inspect)
- propagation.daily_outlook_at (/map outlook strip)

Worker-level end-to-end:
- worker.terrain_profile
- worker.mechanism_classify
- worker.mrms_fetch

Each event is registered in Microwaveprop.PromEx.InstrumentPlugin as
a Prometheus histogram (default / long buckets as appropriate) plus
a counter for the scores_at cache hit/miss ratio. Prometheus at
10.0.15.25 will start seeing the new series on the next scrape after
deploy.
2026-04-18 17:25:33 -05:00
2da74c5cd8
feat(telemetry): wide instrumentation + bump hrrr to 2 per pod
Config:
- runtime.exs hrrr queue 1 → 2 (6 concurrent HRRR jobs across 3 pods)

New helper Microwaveprop.Instrument.span/3 wraps :telemetry.span with
a [:microwaveprop | event_suffix] prefix so metrics can key off it.

Spans added (each emits duration + result tag):
- HrrrClient.fetch_grid / fetch_profile / fetch_idx
- NexradClient.fetch_frame + decode_png
- IemClient.fetch_asos / fetch_raob
- GefsClient.fetch_grid_profiles
- NarrClient.fetch_profile_at
- UwyoSoundingClient.fetch_sounding
- ElevationClient.fetch_elevation_profile
- Weather.upsert_hrrr_profiles_batch / upsert_gefs_profiles_batch
- Propagation.replace_scores
- PropagationGridWorker.compute_scores_algorithm
- TerrainAnalysis.analyse
- CommonVolumeRadarWorker.aggregate_stats

Telemetry catalog (MicrowavepropWeb.Telemetry):
- Oban job duration / queue_time / count / exception by (worker, queue, state)
- Per-span summary metrics for every instrumented phase above
- Periodic (10s) poller emits oban queue depth by (queue, state) — drops
  into the /admin/dashboard Metrics tab immediately

Also drops the now-redundant "fetching n0q frame" and "fetching <url>"
info lines from CommonVolumeRadarWorker / NexradClient; the span events
cover that and the worker's "ingested" line stays for per-job signal.
2026-04-18 16:33:34 -05:00
a6af6b115a
feat(map): 7-day outlook strip
Adds a compact seven-card horizontal strip to both mobile and desktop
map sidebars showing the per-day peak score at the viewport center
for the selected band. Cards are color-coded (emerald for 80+,
rose for bad) so the weekend verdict reads at a glance.

- Propagation.daily_outlook_at/3: groups point_forecast entries by
  UTC date, picks each day's peak, returns ascending order
- MapLive mounts with today's outlook for the initial center; the
  select_band handler refreshes it from the viewport midpoint so the
  strip tracks the user's current band and rough location
- Empty-state message covers the case where no extended-horizon
  scores have landed yet (fresh deploy, or before the first GEFS
  run completes)

This is the consumer of the GEFS pipeline — once the cron starts
running and f024-f168 scores accumulate, days 2-7 of the strip
populate automatically.
2026-04-18 14:46:44 -05:00
d3b0420457
feat(scoring): Richardson-gated native-duct boost
Added 5-arity Scorer.score_refractivity/5 that takes bulk_richardson
alongside best_duct_band_ghz. The existing 1.15× boost now only
applies when Richardson is in the stable regime (< 25) — native-
profile duct cells average 8.7-18.4 for ducting vs 38.3 for
non-ducting, so a duct-band reading under turbulent conditions is
more likely to be torn up by mechanical mixing than to carry a
beyond-LOS signal.

Wiring:
- Weather.nearest_native_duct_info/3 returns {ghz, richardson}; the
  bare nearest_native_duct_ghz/3 now delegates.
- PathLive and Propagation.score_with_algorithm/7 both pull the
  Richardson value into the conditions map alongside best_duct_band.
- Scorer.composite_score/2 reads conditions[:bulk_richardson] and
  passes it into score_refractivity/5.
- Backward compat: 4-arity variant and nil Richardson keep the old
  unconditional-boost behaviour so existing callers don't regress.
2026-04-18 12:40:57 -05:00
fc91f204fd
feat(propagation): record per-forecast-hour chain step timings
Add a propagation_run_timings table so the wall-clock duration of each
(run_time, forecast_hour) step is queryable long after the run is over.
Keyed by (run_time, forecast_hour) with a status column that captures
whether the step succeeded or bailed out, and an error string on
failure.

PropagationGridWorker stamps every step (ok and failed) via
Propagation.record_run_timing/1. Timing inserts are wrapped in rescue +
changeset-error handling so the instrumentation can never brick the
chain.
2026-04-18 09:36:52 -05:00
e227978f2a
Let the /path forecast chart start at the current analysis hour
point_forecast used a strict \`>= now\` cutoff, so the leftmost
"now" sample was always dropped — the HRRR publishing lag means the
newest analysis file is typically 30–60 min behind the wall clock,
which made the filter evict the very hour the chart most wanted to
anchor on. If the chain ever fell behind by even one step the whole
forecast went empty and the chart disappeared.

Use the same window as available_valid_times: keep everything from
one hour before now onward, and fall back to the single newest file
when nothing is fresh enough. Both the cache-path and store-path
flows go through a shared forecast_window helper so the behavior
stays in lockstep with the map timeline.
2026-04-15 10:46:42 -05:00
32fdb2583d
Shrink PropagationGridWorker per-forecast-hour memory footprint
Two related optimizations in the propagation chain hot path. Both
land on the f01..f18 step that was previously OOM-killing prod
pods after the HRRR pressure-level footprint halved wasn't enough
to fit inside 4 Gi.

1. Skip the GridCache broadcast on forecast hours.

   /weather only ever renders the analysis hour (latest_grid_valid_time
   feeds the map). Building 92k rows, serializing them through PubSub,
   and rebuilding the {lat,lon}→row map on all three replicas was
   pure waste for f01..f18 — no consumer was reading that data. Only
   f00 now calls build_grid_cache_rows + broadcast_put. Point lookups
   for non-analysis hours still work through ProfilesFile on disk
   (weather_point_detail_from_profiles/3) exactly as before.

2. Fold replace_scores into a single streaming pass.

   The old path did `Enum.to_list/1` on the ~460k-entry score stream
   followed by `Enum.group_by/2`, holding two full copies of the grid
   before any file was written. A single `Enum.reduce/3` that folds
   each score into a per-band accumulator keeps only one copy and
   eliminates the group_by intermediate entirely. The public
   signature — an Enumerable in, {:ok, count} out — is unchanged.
2026-04-15 08:36:53 -05:00
2d9e5a2d1f
Make the map timeline read .ntms files authoritatively
Two related fixes so the main map reliably picks up new binary
propagation score files as soon as PropagationGridWorker writes them.

1. Propagation.available_valid_times/1 previously preferred ScoreCache
   over ScoresFile, using the cache as an index of what was available.
   The cache is a lazy ETS of whatever hours have been fetched or
   broadcast, which is a strict subset of what's on disk. A new
   forecast hour landing on disk while the cache was warm with older
   entries was invisible to the timeline until the cache happened to
   catch up. Read directly from ScoresFile so the disk store is the
   source of truth.

2. Add Propagation.scores_at_fresh/3 that always reads the .ntms file
   and overwrites the cache entry, and use it from MapLive's
   propagation_updated handler. PropagationGridWorker publishes the
   cache_refresh on `propagation:cache` and the timeline ping on
   `propagation:updated` as separate PubSub broadcasts, so by the time
   MapLive runs through scores_at the ScoreCache GenServer may not
   have processed the refresh yet — fetch_bounds then returns the
   previous chain's bytes. scores_at_fresh takes disk as the source
   of truth for the refresh path and warms the cache as a side effect
   so subsequent readers see the new data.
2026-04-14 17:42:30 -05:00
8af790bd87
Restore point_detail factor breakdown via persisted f00 profiles
The propagation_scores → binary files cutover dropped the factors
JSONB column, which left point_detail returning factors: %{} and
broke the analysis breakdown popup on map clicks. Persist the
enriched f00 grid_data to /data/scores/profiles/{iso}.etf.gz and
rescore on demand at click time so the factor block renders again.
2026-04-14 16:14:55 -05:00
811251dd15
Drop propagation_scores table, ScoresFile is the only store
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:

1. New migration drops the propagation_scores table and its indexes
   (the earlier tuning migrations for it were already applied and
   are now no-ops against a missing table, which is fine — Ecto
   just runs them on fresh environments).

2. Propagation context is gutted of every GridScore reference.
   replace_scores/2 writes files only. upsert_scores/2 is deleted.
   load_scores_from_db, available_valid_times_from_db,
   point_detail_from_db, point_forecast_from_db, fetch_factors,
   coalesce_factors, the Postgres side of prune_old_scores, and
   the Postgres fallbacks in latest/earliest_valid_time are all
   removed. point_detail always returns an empty factors map now
   since factor breakdowns were retired with the table.

3. Deleted modules:
   - Propagation.GridScore (the schema)
   - Propagation.ScorerDiff (read factors from the table)
   - Propagation.AsosNudge (helper for AsosAdjustmentWorker)
   - Workers.AsosAdjustmentWorker (its cron was already disabled)
   - Mix.Tasks.ScorerDiff (wrapper around the deleted module)
   And their tests. AdminTaskWorker's scorer_diff task is a
   logging no-op so any queued Oban rows drain cleanly.
   Release.scorer_diff stays as a stub that tells the operator.

propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
2026-04-14 15:13:01 -05:00
07ffcf52d7
Flip map read path to ScoresFile, stop writing grid HRRR profiles
Read-side cutover for the binary scores store and a companion
cleanup that removes the biggest remaining DB write from the hot
path.

Propagation.scores_at/3, available_valid_times/1, latest_valid_time/0,
latest_valid_time/1, earliest_valid_time/1, point_detail/4, and
point_forecast/3 all now prefer ScoresFile and fall back to the
propagation_scores table when a file is missing. The map render
path reads from /data/scores first; Postgres stays as a safety
net while dual-write is on. point_detail still pulls factors from
Postgres (analysis-hour rows only) and coalesces nil to an empty
map so the JS popup iterates cleanly.

replace_scores/2 is now gated by a postgres_writes_enabled? flag
(runtime env MICROWAVEPROP_SCORES_POSTGRES=false, or the
:propagation_scores_postgres app env key) so the binary-only path
can be benchmarked locally without the DB insert. Default stays
true.

PropagationGridWorker no longer calls store_hrrr_profiles —
persisting 92k grid rows × 19 forecast hours of JSONB profiles
was ~12 min of wall time per chain for a table only
AsosAdjustmentWorker read from. Per-contact HRRR enrichment
through HrrrFetchWorker still writes its own (is_grid_point:
false) rows. AsosAdjustmentWorker is disabled in all three cron
configs since its data source is gone.

DataCase resets the scores tree between tests so per-test
ScoresFile writes don't leak across cases, and ScoresFileTest
switches to async: false because it mutates the global
:propagation_scores_dir env.
2026-04-14 14:50:43 -05:00
690a3523e2
Dual-write propagation scores to binary files on /data
First step of the disk-backed scores migration from the DuckDB plan
doc. Ended up shipping the raw-binary variant instead of Parquet
because the data is disposable after ~2h — the ecosystem benefits
of Parquet only pay off for long-lived datasets, and the binary
path has zero new dependencies.

Microwaveprop.Propagation.ScoresFile writes one file per
(band_mhz, valid_time) tuple under the configured scores dir,
default /data/scores in prod and priv/dev_scores in dev. Layout is
a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is
the no-data sentinel). The whole CONUS grid serializes to ~93 KB
per band, and writes use the temp-then-rename pattern so NFSv4
concurrent readers never see a partial file.

Propagation.replace_scores/2 now materializes the score stream
once and dual-writes: Postgres on the primary path, then one
ScoresFile per band as a best-effort follow-up (any file error is
logged but doesn't fail the DB write, so we can verify the file
path in prod before cutting readers over).

Propagation.prune_old_scores/0 also clears expired score files so
the existing 15-minute prune cron covers both storage layers.

Dev configuration points at priv/dev_scores/, added to .gitignore.
Test configuration points at a per-run tmp directory.
2026-04-14 14:36:55 -05:00
5ec66df135
Cut propagation_scores write cost with DELETE+COPY, skip-factors, UNLOGGED
The scoring+upsert phase was ~4m40s per forecast hour and dominated
wall time. Three stacked optimizations attack it from different
angles.

replace_scores/2 is a new hot-path writer that does DELETE WHERE
valid_time = $1 followed by a plain insert_all (no ON CONFLICT
resolution). The chain worker rewrites the full (valid_time, all
bands) slice every forecast hour, so conflict detection was pure
waste. AsosAdjustmentWorker still uses upsert_scores because it
only rewrites the subset of cells near a station.

factors is now nullable. Forecast hours f01-f18 pass factors: nil
so the JSONB encode + toast write is skipped entirely — roughly
halves the data volume per run. point_detail/4 coalesces nil to
an empty map so the JS popup renders without a TypeError, and
scorer_diff only pulls the most recent valid_time that still has
factors (the f00 row).

propagation_scores is now UNLOGGED, so inserts bypass WAL entirely.
Durability tradeoff: an unclean shutdown truncates the table, but
PropagationGridWorker rebuilds it from HRRR every 3h so a lost
table is re-populated within one cron cycle.

Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a
speculative plan for a flat-file / DuckDB rewrite with explicit
trigger conditions for when to pick it up (partitioning deferred
too; revisit only if these three don't solve it).
2026-04-14 13:47:35 -05:00
805bbff330
Stop AsosAdjustmentWorker from yanking 120 MB of JSONB per tick
AsosAdjustmentWorker fires every 10 minutes and loads every row of
`hrrr_profiles` on the grid for the latest valid_time. The old query
selected `profile: h.profile` — ~1.3 KB of JSONB × 92k grid points ≈
120 MB of JSONB per tick. Postgrex's Jason.decode! ran inline for
each row and blew past the 15 s pool checkout window, so every tick
was killing connections with:

  DBConnection.ConnectionError: client timed out because it queued
  and checked out the connection for longer than 15000ms

`score_grid_point/4` only touched the profile array to re-derive
`min_refractivity_gradient`, but `hrrr_profiles` already persists
that value as a scalar column at ingestion time. Teach
`derive_from_hrrr/1` to honour the persisted scalar when it's
present and drop `h.profile` from the worker's SELECT list. Net
effect: same score math, ~1% of the JSONB transfer, tick stays
under the pool deadline.

Covered by a new scorer test that feeds a profile map with no
`:profile` list and asserts the refractivity factor still reflects
the persisted gradient instead of the neutral baseline.
2026-04-14 10:37:05 -05:00
86364f43aa
Wire NEXRAD, native ducts, and commercial links into scoring
Three signal sources we already collect but weren't using:

* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
  as max of HRRR-derived and NEXRAD-derived rate so fast convective
  cells between HRRR hourly analyses can still trigger the rain penalty.
  Only active on f00 — forecast hours can't see future radar. New
  Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
  hail-safe ceiling.

* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
  applies a 1.15× boost when the cell's native-resolution duct supports
  the target band's frequency. HRRR pressure-level gradients
  systematically under-read thin trapping layers the native profile can
  resolve. Sub-band ducts do NOT boost — they're evidence that the
  gradient we have is all there is at the target frequency.

* Commercial LOS link rx_power fading → inverse tropo sensor.
  Commercial.link_degradation_at/3 computes the average 7-day-baseline
  vs current delta across enabled links within 75 km, ignoring links
  where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
  to the composite score for 3+ dB of fading. ~150 km radius around
  DFW is the only zone this helps today, but it's the first *measured*
  signal in the algorithm vs the model-derived proxies.

Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.

1,359 tests, 0 failures.
2026-04-13 12:08:15 -05:00
c318c3a932
Nudge HRRR scores with live ASOS obs between runs
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.

The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.

Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.

13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.

Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
2026-04-12 14:27:27 -05:00
250709a1b2 Add more caching to make the map feel instant
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
  point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
  falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
  instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
  keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
  concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
  immediately with a pending marker; push rain_scatter_update when
  NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
  viewport after mount/band change/propagation_updated; client caches
  them and renders timeline scrubs instantly without a server roundtrip.
  Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
  timeline click uses preloaded cache when available
2026-04-12 12:26:25 -05:00
d880201713 Cache propagation scores in ETS, broadcast across cluster
- Add ScoreCache GenServer with node-local ETS table keyed by
  {band, valid_time}, subscribed to "propagation:cache" PubSub topic so
  every pod stays in sync with a single hourly compute
- scores_at/3 checks cache first, falls back to DB and populates on miss
- PropagationGridWorker warms and broadcasts the cache for each band
  after every forecast hour upsert; prunes >2h old entries
- Replace per-pixel string-keyed Map with flat Int8Array over the CONUS
  grid in propagation_map_hook.ts to eliminate allocations in the tile
  rasterization hot loop (interpolateScore / propagationReach)
2026-04-12 12:11:26 -05:00
7fb340bc35 Fix all remaining credo --strict issues (0 issues)
Aliases: add module aliases for 9 nested module references
Apply: replace apply/3 with direct module attribute calls
Line length: break 1 long spec line
Refactoring: extract helpers to reduce complexity and nesting
in show.ex, radio.ex, weather workers, terrain, duct detection,
backfill dashboard, contact map, and mix tasks
2026-04-12 10:26:53 -05:00
51e390959c Add dialyzer specs and types across the codebase
277 @spec/@type annotations added to 58 files covering all public
APIs: contexts (propagation, radio, weather, terrain, beacons,
commercial), GRIB2 decoders, terrain analysis, duct detection,
rain scatter, CSV/ADIF import, weather clients, and all Ecto
schemas. Dialyzer passes with 0 errors.
2026-04-12 08:55:04 -05:00
bcdbf0b53a Show per-duct layer heights in map detail panel
When clicking a grid point with ducting, the panel now shows each
duct layer with base-top height in feet, thickness in meters, and
minimum trapped frequency. Data flows from Duct.analyze through
the scoring factors as a ducts array.
2026-04-11 14:13:24 -05:00
880988b591 Update algo.md weights and data flow for native duct integration
- Replace stale April 2026 manual weights with recalibrated values
- Document native hybrid-sigma data in data flow section
- Note refractivity factor now uses native 10-50m resolution
- Add hourly grid integration section to Part 12
- Expose duct_info (count, freq, thickness) in scoring factors for UI
2026-04-11 13:44:13 -05:00
57578dff4d Add native HRRR duct detection to hourly propagation scoring
The PropagationGridWorker now fetches native hybrid-sigma levels
(TMP, SPFH, HGT, PRES × 50 levels) alongside the standard surface
and pressure products. Native data provides 10-50m vertical spacing
vs 250m from pressure levels, detecting thin surface ducts invisible
to the standard product.

Key design: cell-by-cell reducer in Wgrib2.extract_grid_from_file_mapped
processes each of the 95k CONUS cells through a duct analysis function
inline, keeping only scalar metrics per cell. Peak memory ~86 MB
instead of ~1.8 GB for the full grid map.

Per-cell output: native_min_gradient, best_duct_freq_ghz,
max_duct_thickness_m, duct_count. The scorer prefers the native
gradient over the pressure-level gradient when available.

Native fetch is optional — if it fails, scoring continues with
pressure-level data only.
2026-04-11 13:30:48 -05:00
33fae7b7c9 Reduce memory pressure: Stream large collections, GC between phases
- Stream profile storage and score upsert instead of materializing
  full 20k+ item lists (propagation_grid_worker, propagation.ex)
- GC between forecast hours and store/compute phases to reclaim
  ~400 MB of grid data between steps
- Single-pass field extraction in scorer.ex path_integrated_conditions
  instead of 6 separate Enum traversals
- Eliminate intermediate merged map in fetch_grid by combining
  merge + profile build into one pipe
- Fix UUID bug: bingenerate → generate in native grid worker
  (same issue previously fixed in nexrad_worker)
2026-04-10 16:45:50 -05:00
664f1353db Decouple propagation_scores pruning from the compute worker
Pruning used to only run at the end of a successful PropagationGridWorker
pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM)
stopped prune from running and let the table accumulate ~5h of stale
rows. A dedicated PropagationPruneWorker now runs every 15 minutes on
its own Oban cron, and PropagationGridWorker also calls prune_old_scores
at the start of each run as a second safety net. Bumped the delete
timeout from 2m to 5m so the first catch-up pass has enough headroom.
2026-04-09 12:30:10 -05:00
16883591b4
Database performance fixes and async backfill enqueue
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
  non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
2026-04-04 19:19:18 -05:00
23aa2786ab
Skip updating unchanged scores to reduce dead tuples, tune autovacuum
ON CONFLICT now only replaces score/factors when the score actually changed,
avoiding dead tuple generation for the ~80% of grid points that don't change
between consecutive HRRR runs.

Migration sets aggressive autovacuum on propagation_scores: zero cost delay,
2000 cost limit, 1% scale factor. The table was at 72GB with 108M dead rows
because default autovacuum couldn't keep pace with 14M upserts per hour.
2026-04-04 09:53:45 -05:00
8d84bd2d18
Prune propagation scores per valid_time to avoid massive single DELETE 2026-04-03 10:46:17 -05:00
2c16fd979d
Move ML deps to dev/test only, exclude from production build
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
  elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
  compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
2026-04-01 12:10:31 -05:00
8f4f491a5e
Revert to algorithm scorer as primary, add propagation reach polygon
The ML model undervalues conditions outside Aug/Sep training data
(e.g. April with excellent factors scored 37/100). Algorithm's
physics-based factors handle unseen seasons correctly.

- Algorithm is primary scorer, ML infrastructure kept for iteration
- Remove unused ML grid worker code path
- Add client-side propagation reach: BFS flood-fill from clicked point
  through contiguous cells with score >= 50, drawn as convex hull polygon
2026-04-01 10:21:54 -05:00
02cb4fd67b
Integrate ML model into grid worker, QSO search, and UI improvements
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
  falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point

QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page

Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
2026-04-01 10:14:22 -05:00
c12f8cf5ed
Use local solar time for time-of-day scoring, add PWAT factor and pressure refinements
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
2026-04-01 08:58:21 -05:00
14e86c4ddb
Filter forecast graph to only show current and future hours 2026-04-01 08:14:29 -05:00
b11e0bac84
Always show at least the most recent valid_time in timeline 2026-03-31 17:49:19 -05:00
4151f7d314
Filter past hours from forecast timeline (30-min grace for current hour) 2026-03-31 17:47:16 -05:00
6145f80c22
Add forecast sparkline graph to point detail panel
- point_detail response includes forecast array (score per valid_time)
- SVG sparkline shows score trend across all forecast hours
- Trend indicator: Improving/Declining/Steady based on first vs last score
- Add covering index (band_mhz, lat, lon, valid_time) INCLUDE (score) for
  point_forecast queries
2026-03-31 16:47:52 -05:00
e82e631135
HRRR forecast hours (f00-f18) with timeline map UI
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2)
- PropagationGridWorker fetches all 19 forecast hours per run
- Propagation.scores_at/3 queries scores at specific valid_time
- Propagation.available_valid_times/1 returns all forecast times for timeline
- Pruning keeps scores with valid_time >= now - 2h (forecast-aware)
- MapLive: select_time event, timeline data pushed to JS
- JS: forecast timeline bar at bottom of map with clickable hour buttons
- PubSub broadcast sends list of valid_times instead of single time
2026-03-31 16:44:47 -05:00
ee6975bc39
Fix ASOS upsert timeout: infinite tx timeout, bump prod pool to 20
458K-record upsert held a connection for the entire transaction,
exceeding the 15s Postgrex timeout on prod. Set transaction timeout
to infinity and increase prod pool from 10 to 20.
2026-03-31 15:54:19 -05:00
4301f48d27
Guard against HRRR fill values in grid scoring
Skip grid points where surface_temp_c or surface_dewpoint_c are
physically impossible (< -80°C or > 60°C). HRRR returns -273.15
(absolute zero) for ocean/missing points which caused division by
zero in absolute_humidity calculation.
2026-03-31 11:07:26 -05:00
a6b4ac1965
Increase weather queue to 20 and iemre to 10 for faster backfill 2026-03-31 10:41:49 -05:00
3874fc173d
Atomic score upserts and auto-prune old propagation data
- Wrap upsert_scores in Repo.transaction for all-or-nothing visibility
- Prune scores older than the 2 most recent valid_times after each upsert
- Add band-specific latest_valid_time/1 to eliminate N+1 query
- Add require Logger to Propagation module
2026-03-31 10:35:59 -05:00
2f2f3388cd
Rename 'How this works' to 'Scoring Algorithm' 2026-03-31 10:24:04 -05:00
2e216d4244
Make point detail popup instant with client-side lookup and range circles
- Scores now include factors and valid_time in the viewport query,
  eliminating the server round-trip for popups
- Click shows two range circles: solid inner (typical range) and
  dashed outer (max estimated range), colored by score tier
- Circles disappear when popup closes
- Band info pushed to client on band switch for accurate range estimates
2026-03-31 09:50:48 -05:00
0c295c1558
Add click-to-inspect factor breakdown popup on map
Click anywhere on the propagation map to see a detailed popup with:
- Overall score and tier label with color
- Estimated range for the selected band (CW mode)
- All 9 scoring factors with visual bar charts, individual scores,
  and weight percentages
- Grid point coordinates and data timestamp

Factors are displayed in weight order so users can immediately see
which atmospheric conditions are driving the prediction.
2026-03-31 09:45:50 -05:00
97d4bd9372
Only send visible map scores based on viewport bounds
The JS hook sends map bounds on load and on pan/zoom. The server
queries only scores within those bounds, dramatically reducing the
payload for band switches and map updates. At zoom 7 (DFW area)
this sends ~2k scores instead of ~95k.
2026-03-31 09:01:48 -05:00
22cb055269
Skip grid points with missing surface data (ocean/outside-domain) 2026-03-31 08:36:17 -05:00
1e205cb471
Add propagation context, grid worker, and fix merge issues
- Replace stub propagation.ex with real implementation (score_grid_point,
  upsert_scores, latest_scores, latest_valid_time)
- Add PropagationGridWorker (hourly Oban cron) for CONUS grid scoring
- Add mix propagation_grid task for manual triggering
- Fix duplicate extract_grid/2 from parallel merges
- Fix extract_grid to skip outside-grid points instead of erroring
- Add propagation queue to Oban config
2026-03-30 17:18:05 -05:00
5a0113aac2
Add Leaflet map integration with propagation score display
Vendor Leaflet 1.9.4 (JS, CSS, marker images) and wire it into
the esbuild/Tailwind asset pipeline. Create MapLive with band
selector buttons, auto-refresh, and a colocated JS hook that
renders propagation scores as color-coded circle markers with a
legend. Stub Propagation context and BandConfig modules provide
the data interface for the scoring pipeline.

Add nav bar with links to Map, QSOs, and Submit pages.
2026-03-30 17:11:00 -05:00