Commit graph

13 commits

Author SHA1 Message Date
828814e767
fix: resolve all compiler warnings except framework-generated phoenix_component_verify
- Replace deprecated xref:exclude with elixirc_options in vendor/oban_pro
- Remove unused require Logger from 8 files
- Pin bitstring size variables with ^ in simple_packing, wgrib2,
  complex_packing, section, nexrad_client, mqtt, and radar worker
- Remove dead defp clauses (format/1 nil, depression/2 nil)
- Rename Buildings.Parser type record/0 -> building_record/0 to avoid
  overriding built-in type
- Remove redundant catch-all in candidate_detail.ex
- Simplify contact_live/show.ex conditional based on type narrowing
- Fix DateTime.from_iso8601 return pattern in vendor/oban_web
- Upgrade phoenix_live_view to 1.1.31
- Drop --warnings-as-errors from precommit alias; known false positives
  from @after_verify-generated code in Elixir 1.20.0-rc.6 + PLV 1.1.x
- Add credo --strict to precommit as replacement static-analysis gate
2026-05-29 10:18:52 -05:00
14b90ee9f3
fix(security,perf): address 9 audit findings (access control, DoS, crashes)
- Beacon detail endpoints (LiveView + REST API) now hide unapproved
  beacons from anonymous and unauthorized viewers; only the submitter
  and admins can see pending records before approval. Adds
  Beacons.get_visible_beacon/2 with scope-aware checks.
- API contact pagination now honors per_page end-to-end.
  Radio.list_contacts/1 accepts :per_page and clamps to 200.
- API rate limiter: ETS table is now owned by a long-lived Sweeper
  GenServer (won't die with a request task); Sweeper periodically
  prunes expired-window rows to bound memory; init_table/0 race is
  rescued.
- /scores/cells and /weather/cells: add per-IP rate limiting and a
  shared GridBounds clamp/413 guard so global / oversized viewports
  no longer drive unbounded binary responses.
- NEXRAD PNG unfilter (sub/up/average/paeth): replace acc++[byte]
  + Enum.at(acc, idx-bpp) with O(n) binary recursion. Decode time
  for the 12200x5400 n0q frame goes from quadratic to linear.
- LiveTableFooter.parse_page and ScoresFile.fetch_bound: switch
  String.to_integer/String.to_float to Integer.parse/Float.parse,
  fall back to defaults instead of raising.
- PathLive and MapLive band-event handlers: replace
  String.to_integer(params["band"]) with parse_int / parse_band_param
  so a non-numeric band parameter no longer crashes the LiveView.
2026-05-11 18:53:21 -05:00
cea2507fac
test: HTTP injection for MsFootprints/HrdpsClient/NexradClient (79.68%)
- MsFootprints (51% → 93%): http_get injection for dataset_index and
  download_tile, with stubbed CSV parsing, empty CSV, caching, and
  disk-cache-skip tests
- HrdpsClient (47% → 71%): http_get/http_head injection for fetch_grid
  and cycle_available, with stubbed probe, success/failure/transport-
  error tests, plus fetch_grid error path
- NexradClient (57% → 58%): http_get injection, fetch_frame success
  path with valid PNG stub, process_frame coverage
- HrrrNativeClient: http_get injection for fetch_idx (no direct test
  since function is private)

Coverage: 79.43% → 79.68% (need 0.32% more)
2026-05-07 16:36:17 -05:00
764643bc62
test: expand coverage and refactor to idiomatic style
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
  + `if/else` with a classifier that pattern-matches the error reason
  directly, returning `:transient | :permanent`, and dispatch via
  `case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
  step on the row/col comprehensions so an image box whose centre
  lands outside the raster collapses to an empty iteration instead
  of firing Elixir 1.19's ambiguous-range warning. Adds a regression
  test for the right-edge box.

Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
  duct_thickness, best_duct_freq, duct_usable_for_band,
  distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
  unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
  tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
  handling; parse_snmpget_output OID normalisation and junk-line
  tolerance.
- HrrrNativeClient native_level_count, native_variables,
  native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
  SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
  Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
  monotonic in RH, finite across the operating envelope) + idempotent
  nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
  output; latlon_to_pixel round-trip for every grid cell.

Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
  current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.

Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
2026-04-23 13:28:42 -05:00
bbf6481603
perf(nexrad): route fetch_decoded_frame through NexradCache
Telemetry puts nexrad_decode_png at 1.76 s/call × 37,847 calls/h
— ~18.5 h of CPU across the cluster every real hour, the single
biggest CPU consumer in the system. The map-click path
(fetch_rain_cells) already cached decoded frames by 5-min
bucket, but the per-contact CommonVolumeRadarWorker path
(fetch_decoded_frame) went straight to network + decode on every
call. Backfill means many contacts share a 5-min window, so the
same 66 MB frame was being decoded dozens of times.

Wire fetch_decoded_frame through NexradCache keyed on the
rounded timestamp. Add a 20-entry size cap in NexradCache so
backfill processing contacts in random timestamp order can't
grow the ETS table to hundreds of GB. Each frame is ~66 MB, so
20 = ~1.3 GB worst case, well under typical pod memory.

Expected impact: cuts sustained decode load by an order of
magnitude depending on backfill temporal locality; map-click
path is unchanged.
2026-04-19 09:05:55 -05:00
2da74c5cd8
feat(telemetry): wide instrumentation + bump hrrr to 2 per pod
Config:
- runtime.exs hrrr queue 1 → 2 (6 concurrent HRRR jobs across 3 pods)

New helper Microwaveprop.Instrument.span/3 wraps :telemetry.span with
a [:microwaveprop | event_suffix] prefix so metrics can key off it.

Spans added (each emits duration + result tag):
- HrrrClient.fetch_grid / fetch_profile / fetch_idx
- NexradClient.fetch_frame + decode_png
- IemClient.fetch_asos / fetch_raob
- GefsClient.fetch_grid_profiles
- NarrClient.fetch_profile_at
- UwyoSoundingClient.fetch_sounding
- ElevationClient.fetch_elevation_profile
- Weather.upsert_hrrr_profiles_batch / upsert_gefs_profiles_batch
- Propagation.replace_scores
- PropagationGridWorker.compute_scores_algorithm
- TerrainAnalysis.analyse
- CommonVolumeRadarWorker.aggregate_stats

Telemetry catalog (MicrowavepropWeb.Telemetry):
- Oban job duration / queue_time / count / exception by (worker, queue, state)
- Per-span summary metrics for every instrumented phase above
- Periodic (10s) poller emits oban queue depth by (queue, state) — drops
  into the /admin/dashboard Metrics tab immediately

Also drops the now-redundant "fetching n0q frame" and "fetching <url>"
info lines from CommonVolumeRadarWorker / NexradClient; the span events
cover that and the worker's "ingested" line stays for per-job signal.
2026-04-18 16:33:34 -05:00
1a62e51f82
chore(radar): drop per-fetch ok-bytes log line 2026-04-18 16:22:04 -05:00
ced5e3d1a6
feat(radar): log NEXRAD fetches + worker progress
Previously the CommonVolumeRadarWorker ran silently — no URL logged,
no sign in the logs that it was doing any work. Added info-level
fetch/ok/error logs in NexradClient (same shape as WeatherFetchWorker
RAOB logs) plus start-of-work + ingestion lines in the worker itself.
Unchanged: the existing "no frame" warning and all control flow.
2026-04-18 16:20:59 -05:00
33f5d4edbe
feat(rainscatter): classify QSO propagation mechanism from common-volume radar
Adds a per-contact enrichment pipeline that determines whether a QSO was
most likely carried by rain scatter, tropospheric ducting, or ordinary
troposcatter — using IEM n0q composite reflectivity sampled inside the
lens-shaped intersection of 400 km-radius disks around each endpoint.

Pieces:
  * Microwaveprop.Propagation.CommonVolume — lens geometry (haversine,
    in-CV test, bbox, area).
  * contact_common_volume_radar table (1:1 per contact) storing
    aggregate dBZ stats inside the CV + radar_status column on contacts.
  * Microwaveprop.Workers.CommonVolumeRadarWorker — Oban :radar queue,
    fetches the n0q frame at QSO time, iterates pixels inside the CV
    bbox, aggregates rain/heavy/core-pixel counts, max/mean dBZ, and
    coverage percentage.
  * Microwaveprop.Propagation.RainScatterClassifier — rule-based mapper
    from (band, distance, radar stats, duct flags) to one of
    :likely_rainscatter | :rainscatter_possible | :tropo_duct |
    :troposcatter | :unknown.
  * ContactWeatherEnqueueWorker learns a :radar enrichment type and
    enqueues the CV worker on contact submission; pre-2014 contacts
    (outside IEM n0q coverage) are pinned to :unavailable.
  * `mix radar_backfill` bulk-enqueues historical contacts with
    --year / --limit / --dry-run.
  * Contact detail page renders a mechanism badge with supporting
    stats (common-volume area, max dBZ, heavy-rain pixel count,
    coverage %).
2026-04-17 15:57:59 -05:00
c318c3a932
Nudge HRRR scores with live ASOS obs between runs
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.

The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.

Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.

13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.

Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
2026-04-12 14:27:27 -05:00
250709a1b2 Add more caching to make the map feel instant
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
  point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
  falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
  instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
  keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
  concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
  immediately with a pending marker; push rain_scatter_update when
  NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
  viewport after mount/band change/propagation_updated; client caches
  them and renders timeline scrubs instantly without a server roundtrip.
  Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
  timeline click uses preloaded cache when available
2026-04-12 12:26:25 -05:00
e8ae407be9 Add rain scatter prediction to map detail panel
When clicking a grid point, fetches the latest NEXRAD composite
reflectivity and identifies rain cells within 300 km that could
enable rain scatter contacts. Shows:
- Scatter classification (excellent/good/marginal/none)
- Top 3 cells with dBZ, distance, bearing, and relative signal
- Colored circle markers on the map at rain cell locations
- Markers sized by reflectivity, colored by intensity

Uses simplified bistatic radar equation accounting for reflectivity,
frequency-dependent scattering (Rayleigh/Mie), and R^4 path loss.
NEXRAD cells sampled every ~5 km within bounding box for efficiency.
2026-04-11 15:19:19 -05:00
e7a7ae073d Phase 9.3, 9.4, and Phase 3 NEXRAD pipeline
Task 9.3 - Weight recalibration via gradient descent:
- Recalibrator module fits logistic regression weights using Nx
- Trains on QSO positives vs random baseline negatives
- Cross-validates by month, normalizes weights to sum to 1.0
- Mix task: mix recalibrate_scorer --sample 5000 --epochs 2000

Task 9.4 - Side-by-side scorer comparison:
- ScorerDiff.compare/3 re-scores grid with old vs new weights
- Reports mean diff, regressions, improvements, per-band breakdown
- Mix task: mix scorer_diff --new-weights '{...}'

Phase 3 - NEXRAD ingestion pipeline:
- NexradClient fetches IEM n0q composite PNGs, extracts per-point
  box statistics (mean/max dBZ, texture variance)
- NexradObservation schema with unique (lat, lon, observed_at)
- NexradWorker on :nexrad queue for background processing
- nexrad_texture backtest feature in Features module
- mix nexrad_backfill --limit 200

All tasks added to AdminTaskWorker and Release for production use.
1116 tests, 0 failures.
2026-04-10 12:48:36 -05:00