Commit graph

521 commits

Author SHA1 Message Date
3bdc4b6a11
Fix 4 critical bugs and document remaining findings
- TOCTOU race: add partial unique index + rescue constraint violation on
  concurrent contact insert (radio.ex:737-768)
- Mass assignment: remove :user_id from @optional_fields (contact.ex:85)
- Stale path_live result: clear result when URL params change (path_live.ex:105)
- Blocking migrations: wrap Release.migrate() in Task.start (application.ex:79)
- Rate limiter: fix guard for monitor tokens (not is_nil vs is_binary)
- findings.md: document remaining non-critical bugs and improvements
2026-05-29 15:27:33 -05:00
daafa5a02a
perf(algo): unify recalibration into single Python pipeline with JSON output
Replace the two-script Python pipeline (analysis report + Elixir-source
emitter) with a single `scripts/recalibrate.py` that fits per-band
weights from PSKR spot density (VHF/UHF) and contacts↔HRRR correlations
(microwave), writing `priv/algo/band_weights.json` as a machine-readable
artifact. A new Elixir BandWeights module loads this JSON once via
`:persistent_term` cache; BandConfig.weights/1 consults it before falling
back to in-source overrides or global defaults. The script never touches
Elixir source — recalibration is now `python3 scripts/recalibrate.py`
followed by an app restart.
2026-05-25 14:45:55 -05:00
e3d430f8c4
feat(weather): add duct cutoff band map layer
Show on /weather where the detected duct geometry traps each microwave
band. Overview mode bins cells by their lowest trapped frequency
(Bean & Dutton cutoff); band-picker mode masks cells whose duct doesn't
reach the selected band. Cutoff is pre-computed per cell in both
writers (Elixir derive + Rust derive_row reads best_duct_freq_ghz from
CellValues), persisted to ScalarFile, and shipped to the JS hook via
the existing binary cell pack. Also cleans up six pre-existing
length/1 credo warnings in unrelated test files.
2026-05-15 19:51:24 -05:00
bd731685de
feat(api): ingest endpoint for propmonitor beacon measurements
POST /api/v1/beacon-monitor/measurements accepts one measurement per
integration window from a propmonitor client, authenticated by the
BeaconMonitor token. Records noise floor, signal peak/avg dBFS, SNR,
signal-active-fraction, gain, and frequency for later correlation
against weather and propagation scores.

Beacon UUID must resolve to an approved + on-the-air beacon (404
otherwise — the client drops 404s without retry). Retries with the
same (monitor, beacon, measured_at) are idempotent: a unique index
makes the second insert a 409. Every accepted upload stamps the
monitor's last_seen_at — measurements *are* the heartbeat.

MonitorAuth is a separate plug from the user API-token Auth plug;
monitors are not users. The rate limiter buckets each monitor by id so
retry storms after a 5xx don't burn the shared anon-IP bucket.
2026-05-13 16:07:04 -05:00
c6adc989e9
feat(path): show live stage progress in /path calculator button
PathCompute.compute/5 now accepts an :on_progress callback and emits
9 named stages (resolve → terrain → HRRR grid → atmospheric → sounding
→ scoring → budget → forecast → ionosphere). PathLive runs the compute
in start_async, streams {:compute_progress, step, total, label}
messages from the callback, and renders the current stage + step/total
in the disabled button alongside a daisyUI progress bar — replacing
the static "Computing..." while the multi-second pipeline runs.

handle_async covers :ok, :error, and {:exit, _} (last logs per the
CLAUDE.md async-error rule). Existing PathLive tests that asserted on
result content after live() were switched to render_async(lv) so they
wait for the task; new tests cover the progress callback ordering and
the in-flight label rendering.
2026-05-12 15:59:46 -05:00
1d4530ef21
fix: 6 bugs from bugs.md — ADIF parser, Es MUF, enrichment reset, profile privacy, HRRR OOM risks 2026-05-12 09:37:09 -05:00
0704253af8
fix(security,test): viewer-aware profile queries + create_contact ordering + valkey test isolation
- /u/:callsign profile no longer leaks private contacts or pending beacons;
  Radio.list_contacts_for_user/2 and Beacons.list_beacons_for_user/2 now
  filter by viewer (owner/admin see everything, others see only public).
- Radio.create_contact/2 places user_id on the struct before
  submission_changeset so validate_user_or_email/1 accepts authenticated
  submissions that omit submitter_email.
- valkey_test.exs runs async: false; its setup mutates global Application
  env and Process registry, which otherwise crashed concurrent ConnCase
  GridCache.clear/0 calls (Mox.UnexpectedCallError on SCAN).
2026-05-12 08:49:08 -05:00
a3b97f9e98
fix(security): mass-assignment on owner edit + 3 minor hardening fixes
- Radio.normalize_proposed/1 now Map.take/2 the allowed-edit keys
  before normalization. Owner / admin / pending-edit submit paths all
  funnel through this boundary, so a crafted form can no longer
  mass-assign user_id, flagged_invalid, inserted_at, etc. on a
  contact via String.to_existing_atom -> Ecto.Changeset.change.
- Accounts.revoke_api_token/2 rescues Ecto.Query.CastError and
  returns {:error, :not_found} so a malformed UUID in
  DELETE /api/v1/me/api-tokens/:id renders the API's clean 404
  problem+json instead of a 500.
- ProfilesFile.read_etf decodes with :erlang.binary_to_term(bin, [:safe]).
  Defense-in-depth against tampered on-disk profiles (atom-table
  exhaustion via untrusted ETF).
- Regression tests: contact_edit_test.exs covers the rejected
  mass-assignment fields; accounts_api_token_test.exs covers the
  malformed-UUID path.
2026-05-11 19:04:16 -05:00
14b90ee9f3
fix(security,perf): address 9 audit findings (access control, DoS, crashes)
- Beacon detail endpoints (LiveView + REST API) now hide unapproved
  beacons from anonymous and unauthorized viewers; only the submitter
  and admins can see pending records before approval. Adds
  Beacons.get_visible_beacon/2 with scope-aware checks.
- API contact pagination now honors per_page end-to-end.
  Radio.list_contacts/1 accepts :per_page and clamps to 200.
- API rate limiter: ETS table is now owned by a long-lived Sweeper
  GenServer (won't die with a request task); Sweeper periodically
  prunes expired-window rows to bound memory; init_table/0 race is
  rescued.
- /scores/cells and /weather/cells: add per-IP rate limiting and a
  shared GridBounds clamp/413 guard so global / oversized viewports
  no longer drive unbounded binary responses.
- NEXRAD PNG unfilter (sub/up/average/paeth): replace acc++[byte]
  + Enum.at(acc, idx-bpp) with O(n) binary recursion. Decode time
  for the 12200x5400 n0q frame goes from quadratic to linear.
- LiveTableFooter.parse_page and ScoresFile.fetch_bound: switch
  String.to_integer/String.to_float to Integer.parse/Float.parse,
  fall back to defaults instead of raising.
- PathLive and MapLive band-event handlers: replace
  String.to_integer(params["band"]) with parse_int / parse_band_param
  so a non-numeric band parameter no longer crashes the LiveView.
2026-05-11 18:53:21 -05:00
91c9d98a99
feat(docs/api): protocol-style two-pane API reference layout
Replace the markdown-rendered /docs/api page with a structured
LiveView modelled on the Tailwind UI Protocol template — fixed
left sidebar with grouped section nav, hero with metadata dl,
two-column rows where prose sits next to a sticky code sample,
and endpoint cards with color-coded HTTP method tags.
2026-05-10 11:06:29 -05:00
04f9ed5fe6
feat(settings): manage /api/v1 tokens from user settings UI
Adds a separate API-token section under /users/settings (distinct from
the beacon-monitor token list, since API tokens grant full account
access). The plaintext is surfaced exactly once via flash on creation;
only the SHA-256 hash is persisted, so revocation is the only path back
if the user loses it.

Also fixes the openapi.yaml link on /docs/api: the relative path
resolved to /docs/openapi.yaml from a no-trailing-slash URL and 404'd.
2026-05-09 09:59:56 -05:00
0429c9dbab
fix(docs): render API reference bullets correctly + tighter reading column
Two issues made docs/api hard to read:

1. The site's custom Markdown parser only recognized `- ` bullets,
   so the README's `* ` bullets rendered as a single run-on paragraph
   with literal asterisks. Extended the parser (and tests) to accept
   the full CommonMark bullet set: `-`, `*`, `+`.

2. The shared .markdown-content container is 88rem wide. Comfortable
   for /algo's wide tables but uncomfortable for monospace prose,
   which prefers ~80ch. Added a .api-docs modifier class on the
   /docs/api page that drops max-width to 60rem, allows table
   cells to wrap, and slightly downsizes headings.
2026-05-09 09:46:45 -05:00
c56b55d5af
feat: serve API docs at /docs/api
- /docs/api → LiveView rendering README.md as HTML
- /docs/api/openapi.yaml → raw OpenAPI 3.1 spec
- /docs/api/README.md → raw markdown source
2026-05-09 09:09:45 -05:00
c6d2c48264
feat: secure /api/v1 REST API for regular-user actions
Adds bearer-token authenticated REST API at /api/v1 covering every
action a non-admin user can perform on the website: contact + beacon
submission, beacon-monitor management, propagation queries, profile
read/update, and self-service API token issuance/revocation.

Security: SHA-256-hashed bearer tokens (mwp_ prefix, plaintext shown
once at creation), RFC 9457 problem+json error responses, RFC 9651
RateLimit-* headers backed by an ETS bucket (600/min per token,
60/min per anonymous IP, 30/min on /auth/tokens), private-contact
filtering by viewer.

Docs at docs/api/README.md (prose reference) and docs/api/openapi.yaml
(OpenAPI 3.1 spec covering every endpoint, response, and schema).

Tests: 124 new tests across schema, plug, error renderer, rate
limiter, fallback, and every controller. 16/17 API modules at 100%
line coverage; FallbackController at 87.5% (one defmodule line, an
Erlang-cover artifact for action_fallback-only modules).
2026-05-09 08:59:54 -05:00
fc9d2298ac
test: lift coverage to 85% and pin threshold
Adds tests for previously under-covered modules so the cover-tool
threshold check passes:

  * Mix.Tasks.Prop.Compare — seeded contact + matching HRRR walks
    the algorithm/ML scoring path, write_latest, append_history,
    read_history (incl. the unparseable-line arm), and the per-band
    summary loop.
  * Mix.Tasks.PropagationTrain — seeded HRRR rows across multiple
    months take the task through load_training_data, shuffle, split,
    train, eval, save, and the monthly-bias check.
  * Mix.Tasks.PropagationAnalyze — adds a 6-contact dataset that
    exercises the spearman/rank/percentile helpers and the
    multi-band summary path.
  * Mix.Tasks.Unused — smoke tests run/1 with no flags,
    --skip-external, and --verbose.
  * Mix.Tasks.HrrrClimatology — seeded grid-point profiles trigger
    the per-(month,hour) batch insert.
  * Microwaveprop.Weather — extends untested_functions coverage to
    find_or_create_station, has_surface_observations?,
    station_day_covered?, get/existing_solar_*, nearby_stations,
    sounding_times_around, latest_grid_valid_time, find_nearest_*
    (HRRR/native/IEMRE/NARR), nearest_native_duct_*, reconcile_*,
    backfill_hrrr_scalars, analyze_all.
  * Microwaveprop.Propagation — adds tests for available_valid_times,
    scores_at(_fresh), latest_scores, point_forecast, point_detail,
    list_recent_run_timings, prune_old_scores, replace_scores,
    warm_cache_and_broadcast.
  * Microwaveprop.Backtest.Features — adds duct_usable_*ghz alias
    delegations.
  * Microwaveprop.Weather.IemRateLimiter — covers the is_pid clause
    of registered?/1 with a live PID.
  * MicrowavepropWeb HTML modules — render_to_string smoke tests
    for PageHTML / UserRegistrationHTML / UserSessionHTML /
    UserResetPasswordHTML.

Also pins `test_coverage: [summary: [threshold: 85]]` in mix.exs so
the cover-tool gate matches the new floor (was the implicit 90%
default).

Total goes from 82.77% → 85.06%.
2026-05-08 13:59:56 -05:00
f8dee5e670
fix: pre-compute refractivity in Rust profile entries, fall back in SkewT
Rust cell_to_profile_entry now writes surface_refractivity and
min_refractivity_gradient at the cell level so Elixir consumers have a
guaranteed fallback when SoundingParams.derive hits levels without
dewpoint. The SkewT build_profile_assigns falls back to these pre-computed
cell keys when the derived map has nil for those fields.
2026-05-08 13:02:15 -05:00
680ffa8cf2
test: silence Postgrex sandbox disconnect noise
When a test process exits while still owning a sandboxed Postgrex
connection, Postgrex.Protocol logs `[error] ... disconnected:
DBConnection.ConnectionError client #PID<...> exited`. It's benign
sandbox cleanup but at :error level it bypasses `capture_log: true`
(which only captures the test process's logs) and clutters every
test run.

Pin Postgrex.Protocol + DBConnection.Connection to :critical so the
noise drops without masking genuine DB error reporting (we never log
at :critical from those modules).
2026-05-08 10:43:51 -05:00
b8eea1bd45
prom: wire rust workers into prometheus
prop-grid-rs already exposed prop_grid_rs_chain_step_duration_seconds
+ tasks_in_flight on :9100 but the comment still pointed at the old
prom server (10.0.15.25). hrrr-point-rs had no metrics listener at all.

- metrics.rs: new POINT_BATCH_DURATION / POINT_BATCHES_TOTAL /
  POINTS_PROCESSED_TOTAL series + record_point_batch helper, plus unit
  tests serialized via a static Mutex (the prometheus crate's global
  registry races otherwise)
- hrrr_point_worker.rs: spawn metrics::serve on METRICS_ADDR (default
  0.0.0.0:9100), wrap process_batch in InFlightGuard, record outcome +
  per-point counts on every batch
- deployment-hrrr-point-rs.yaml: prometheus.io/{scrape,port,path,job}
  annotations, METRICS_ADDR env, ports.metrics, /health readiness probe
- deployment-grid-rs.yaml: refresh stale 10.0.15.25 comment, add
  prometheus.io/job for clean dashboard up{} selectors

Also fix Microwaveprop.PromExTest: `assert plugins != []` triggered
Elixir 1.19 type-checker warning ("non_empty_list != empty_list always
true"); replaced with membership assertions for the InstrumentPlugin +
Plugins.Beam.
2026-05-08 10:36:49 -05:00
99103e5cd6
test: PromEx + Canopy.BulkFetch + Maidenhead.valid? coverage
- PromEx: added dashboards/0 + plugins/0 tests
- Canopy.BulkFetch: 0% -> covers run/4 entry path (all tile fetches fail
  cleanly without network); cache/staging dirs created
- Maidenhead.valid?/1: rejects integer/atom/tuple/list/map inputs
- Canopy.tile_filename/2: S/E hemisphere prefix branches
- ScoreCache.max_entries/0 invariant test

3624 tests, 0 failures.
2026-05-08 10:06:36 -05:00
da332d3174
test: 100% coverage push — Valkey.RedixAdapter, Buildings.BulkFetch, AprsRepo, Maidenhead, Canopy
- Valkey.RedixAdapter: 0% -> 100% (delegation lines exercised)
- Buildings.BulkFetch: 0% -> 56% (run/4 with stubbed empty index)
- AprsRepo: 50% -> exercises repo config + Ecto.Repo callbacks
- Maidenhead.valid?/1: covers non-binary non-nil arms
- Canopy.tile_filename/2: S/E hemisphere prefix branches
2026-05-08 10:01:28 -05:00
35caae861e
test: more Weather + RoverLive + GefsFetchWorker coverage tests
- Weather: warm_grid_cache_*, materialize_scalar_file/1,
  build_grid_cache_rows/3 bounding-box filter
- RoverLive: rover_progress info handler, select_candidate Maidenhead
  fallback + malformed grid no-op
- GefsFetchWorker: build_profile_attrs/3 comprehensive 14-field test

Total: 82.74% (up from 82.7%).
2026-05-08 09:46:16 -05:00
2c6458957b
test: SkewtLive HRRR + sounding fixtures bring coverage to 82.7%
Added profile/sounding setup blocks to SkewtLive tests so the
list_valid_times_near + nearest_sounding_to paths run with real
data. SkewtLive: 33% -> 46%.

Total project coverage: 82.7%.
2026-05-08 09:38:41 -05:00
363646f34f
test: enable :slow tests by default + add coverage tests across modules
Removed the `exclude: [:slow]` from test_helper.exs — the slow-tagged
wgrib2 fixture tests pass with the binary on PATH and pull
Microwaveprop.Weather.Grib2.Wgrib2 from 23% to 90%+ in coverage runs.

Additional tests:
- Pskr.Client: tcp/tcp_closed/tcp_error stale-socket fall-throughs
- Weather context: weather_point_detail/3, latest_grid_valid_time/0,
  available_weather_valid_times/0, available_hrdps_valid_times/0
- PathCompute.compute/4: with HRRR profile near the path
- RoverPathProfileWorker: real Path row + skip-when-complete
- RoverLive: URL stations encoding, add_station error paths
- PathLive: rover_path_id branches (404, malformed UUID)

3599 tests, 0 failures.
2026-05-08 09:28:51 -05:00
888701e627
test: bump coverage 80% -> 81.3% via more LiveView + module tests
- Pskr.Client: 0% -> 18% (init/standby/handle_info/terminate)
- RoverPlanningLive.PathShow: 0% -> 82% (mount/load_path)
- Rover.CandidateDetail: 0% -> partial (summarize/5 with stations)
- Mix.Tasks.Weather.RebatchAsos: 0% -> covered (wrapper run/1)
- HrrrNativeClient: 57% -> 67% (fetch_native_duct_grid error paths)
- RoverLive: 40% -> 48% (toggle/delete/update_station_grid logged-in)
- PathLive: 62% -> 63% (path_forecast_detail + rover_path_id branches)
- PathCompute: 64% -> 64% (compute/4 full pipeline + resolve_location)
- SkewtLive: 29% -> 31% (search/select_time event no-ops)
- SkewtLocationResolver: 38% -> 60%+ (callsign cache + error paths)

3557 tests passing.
2026-05-08 09:10:34 -05:00
f644d1796f
test: push coverage 79.64% -> 80.04% across multiple modules
New test files: RoverPlanning.Station, LiveStashGuard, RebatchAsos,
RoverMissionReconcileWorker, RoverPathProfileWorker. Extended existing
suites for PathCompute (resolve_location, build_ionosphere_readout),
HrdpsClient (DEPR fallback, missing-level skip), NexradClient
(fetch_rain_cells HTTP error paths), UserHomeQthLookupWorker (error +
Maidenhead-fallback paths), RoverPlanning.Path (statuses/0 + schema).

3512 tests passing.
2026-05-08 08:42:03 -05:00
06a54075f8
test: http_get injection for GefsClient + HrrrClient
- GefsClient (73%): http_get injection for fetch_idx + grib download,
  tests for timeout + HTTP error paths, dewpoint_from_rh edge cases
- HrrrClient: http_get injection for do_fetch_idx + range downloads
- GefsClient fetch_grid_profiles error-path tests with stubbed HTTP

Coverage: 79.68%
2026-05-07 16:44:40 -05:00
cea2507fac
test: HTTP injection for MsFootprints/HrdpsClient/NexradClient (79.68%)
- MsFootprints (51% → 93%): http_get injection for dataset_index and
  download_tile, with stubbed CSV parsing, empty CSV, caching, and
  disk-cache-skip tests
- HrdpsClient (47% → 71%): http_get/http_head injection for fetch_grid
  and cycle_available, with stubbed probe, success/failure/transport-
  error tests, plus fetch_grid error path
- NexradClient (57% → 58%): http_get injection, fetch_frame success
  path with valid PNG stub, process_frame coverage
- HrrrNativeClient: http_get injection for fetch_idx (no direct test
  since function is private)

Coverage: 79.43% → 79.68% (need 0.32% more)
2026-05-07 16:36:17 -05:00
f82dfb4d00
fix(test): GridCache Valkey fetch_point bug + data-path tests (GridCache 49%→84%)
- Fix production bug in valkey_fetch_point: Map.get(chunk_map, {lat, lon})
  was looking up lat/lon in the outer map keyed by chunk tuple, not the
  inner cells map. Added chunk_key unwrap before lat/lon lookup.
- DataStubAdapter returns real chunk data for fetch/fetch_bounds/fetch_point
- Tests cover: fetch with decoded rows, fetch_bounds filtering, fetch_point
  with chunk unwrap, put with large row sets, claim_fill/release_fill

Coverage: 79.30% → 79.43% (GridCache 74% → 84%)
2026-05-07 16:15:02 -05:00
5d90d52a45
test: format all test files + GridCache Valkey backend (49%->74%)
Coverage: 79.09% -> 79.30% (GridCache 49% -> 74%)
2026-05-07 16:04:32 -05:00
6028e8a838
test: add GridCache prune_older_than ETS test
Coverage: 79.07%
2026-05-07 13:59:42 -05:00
3a2a28437d
test: property tests for PathCompute loss/power budget + Viewshed destination
- PathCompute: compute_loss_budget/5 and compute_power_budget/2 made
  public with @doc false; property tests covering fspl, o2, h2o, rain,
  diffraction across all 13 known bands + varying distances
- Viewshed: property tests for destination_point back-bearing round-trip
  and east-west bearing at equator (2 new properties)

Coverage: 79.07% → 79.06% (PathCompute 60→63%)
2026-05-07 13:56:28 -05:00
89e8bb2267
test: use Mox for Valkey tests without real Redis (55%→88%)
- Create Microwaveprop.Valkey.Adapter behaviour with command/3 + pipeline/3
- Create Microwaveprop.Valkey.RedixAdapter as the production impl
- Inject adapter via Application config; mock with Mox in tests
- Cover all Valkey operations: get, mget, set, mset_with_ttl, zadd,
  zrevrange, zrangebyscore, zrem, del, scan_match (multi-cursor)
- Full round-trip tests for encode/decode, error paths, edge cases

Coverage: 78.93% → 79.07%
2026-05-07 13:48:08 -05:00
58f4a7714b
test: add Mox deps + SnmpClient poll/3 tests, RoadProximity, Buildings.Loader
- Add {:mox, "~> 1.0", only: :test} to deps
- Commercial.SnmpClient: injectable snmp_runner for poll/3 tests (62→92%)
- Rover.RoadProximity: pure road_distances/3 tests with injectable fetch (37→82%)
- Buildings.Loader: ensure_loaded/1 with real tile file + quadkey tests (69→100%)
- Buildings.Index: records_near, mark_loaded, loaded? tests (72→96%)

Coverage: 77.96% → 78.93% (+0.97%)
2026-05-07 13:36:22 -05:00
b764034e46
test: push coverage to 78.6% — MapLayers 100%, LinkMargin 100%, Elevation 94%
- Weather.MapLayers: unit tests for all/0, default_id/0, valid_id?/1 (25→100%)
- Rover.LinkMargin: link_margin_db ScoresFile round-trip tests (57→100%)
- Rover.Elevation: SRTM tile file read path + void sentinel (44→94%)
- Pskr.Aggregator: pending_count, empty flush tests (65→74%)
- Buildings.MsFootprints: quadkey/bbox pure function tests (51→51%)
- Weather.Grib2.Wgrib2: property tests for parse_lon_val_segment (23→23%)
- test/support/data_case: File.rm_rf -> rm_rf to fix concurrent race
2026-05-07 13:17:40 -05:00
75e074fb66
fix(test): use non-bang File.rm_rf to avoid race in concurrent score-dir cleanup 2026-05-07 13:03:41 -05:00
e982dc4175
fix(test): make rover calculate test resilient to empty ScoresFile 2026-05-07 12:58:56 -05:00
9ef10f5aae
test: expand coverage across rover, pskr, valkey, skewt, scores
- Rover.Compute: road/prominence/clearance/clutter/canopy tests (90%)
- Rover.Prominence: full unit test suite with mock elev_lookup (100%)
- Rover.Elevation: dedup + multi-tile tests (44%)
- Rover.LinkMargin: edge cases, negative scores, all modes (57%)
- Rover.Location: changeset validation + statuses/0 (100%)
- RoverPlanning.Path: changeset validation + statuses/0 (67%)
- Pskr.FeatureBin: changeset validation (100%)
- Pskr.Mqtt: QoS reject, unknown type, varint overflow, ping/disconnect (97%)
- Valkey: not-configured error paths, empty guards (55%)
- ScoresController: bad params, time format, missing band tests (81%)
- SkewtLive: info/no-profiles state, loading states (29%)
2026-05-07 12:51:58 -05:00
a0065e2f44
refactor: deduplicate hrrr download, simplify radio/mqtt, fix credo/perf issues
- hrrr_client: extract shared fetch_grib_ranges/2 to eliminate ~80% duplicated code
- mqtt: encode_varint/1 uses IO list accumulator instead of O(n^2) binary concat
- commercial: single-pass reduce for aggregate_degradation, remove unused average/1
- scorer: single-pass reduce for safe_avg and build_path_conditions
- terrain_analysis: split 76-line do_analyse/4 into 3 focused functions
- radio: build_position_changes/3 simplified from then-chains to Enum.reduce
- recalibrator_test: fix credo warning (length > 0 -> != [])
2026-05-07 11:50:29 -05:00
fe2ad2df15
fix(pskr): self-healing follow-up sampler runs for hours with missing HRRR
The producer-only path left older hours stuck at 0% HRRR coverage
forever once their fetch tasks drained: the rolling-window cron only
re-passes the prior hour, so manual lookback runs (or any
out-of-window hour) wrote NULL samples + enqueued tasks, then never
re-ran to UPSERT the now-landed HRRR data.

Fix: split the responsibility cleanly.

* CalibrationSampler.build_for_hour/1 now returns
  %{upserted: int, missing_hrrr_cells: int} so callers know whether
  the producer enqueued anything (i.e. whether a follow-up makes
  sense). Spec-tightened with a new @type result.
* PskrCalibrationWorker, after each hour's sampler pass, schedules a
  follow-up of itself for that exact hour 10 min later when missing
  > 0. The follow-up carries args %{"hour_utc" => ..., "_follow_up"
  => true}; the sentinel prevents follow-ups from chaining further
  follow-ups (rolling-window cron is the safety net).

Tests:
* Updated 4 existing sampler tests to the new map return shape.
* Moved the follow-up assertions from the sampler suite into the
  worker suite where they belong (sampler is now scheduling-free).
* Added 3 worker tests: schedules-when-missing, no-schedule-when-
  covered, follow-up-doesn't-chain.

3313 tests + 228 properties, 0 failures.
2026-05-05 18:00:10 -05:00
7ee19c5b6a
perf+test: cache nearest_hrrr per cell, batch pg_inherits, +property tests
Two performance fixes the property tests pinned down:

* CalibrationSampler.build_for_hour/1 walked every cell through
  nearest_hrrr/3 twice — once in the producer and again in
  build_sample. Now computed once into a (band, lat, lon) → hrrr_or_nil
  map, halving the O(cells × profiles) scan (~10M comparisons saved
  per fire at typical sizes).
* PartitionManager.ensure_quarterly_partitions/2 issued one
  pg_inherits scan per (parent × lookahead). Refactored to one scan
  per parent with in-memory coverage check across all quarters.

Property tests for both modules:

* PartitionManager: tile-coverage invariant (no gaps, no overlaps),
  exact 3-month bounds, name format, multi-call idempotence,
  multi-parent independence — caught a real NaiveDateTime sort bug
  in the original test helpers. Default Enum.sort_by/2 falls back to
  term comparison on NaiveDateTime; now uses NaiveDateTime as the
  comparator module.
* CalibrationSampler: enqueued points = unique missing midpoints,
  empty enqueue when fully covered, sample row count invariant under
  HRRR availability.

Saved operational gotchas to CLAUDE.md (Oban leader election spans
all app=prop pods, kubectl exec deploy/prop ambiguity, half-year
partition coexistence, HRRR f000 publish lag).

3310 tests + 228 properties, 0 failures.
2026-05-05 17:40:13 -05:00
f78d4ccb9f
test(pskr): two-pass HRRR pipeline integration test + tighter specs
Adds the test that verifies the full producer→drain→re-pass loop:
sample written with NULL HRRR fields + fetch task enqueued, then once
an HRRR profile lands, the next sampler pass UPSERTs the same
calibration_samples row (id preserved) with non-NULL weather data.

Also tightens dialyzer surface:

* PartitionManager defines a `result()` typedoc and uses it on both
  public functions instead of inlined tuple types.
* PartitionMaintenanceWorker.perform/1 + the lookahead/1 helper get
  explicit specs.
* CalibrationSampler.enqueue_missing_hrrr/3 gets a spec covering its
  group_by-shape input map and HRRR profile list.

`mix precommit` clean (3310 tests, 0 failures). Local `mix dialyzer`
hits an OTP 28.4 vs 28.5 toolchain mismatch unrelated to this change.
2026-05-05 17:31:17 -05:00
413cc92cab
feat(infra): auto-extend quarterly partitions for hrrr/hrdps tables
Partition list for hrrr_profiles + hrdps_profiles was hand-edited in
priv/repo/structure.sql; if no human extended it, writes for the next
quarter would silently fail at the worker level (Rust hrrr-point-worker
marks tasks failed; the calibration pipeline silently joins NULLs).

Adds:

* Microwaveprop.PartitionManager — runtime DDL helper. Quarter math via
  a single integer index (year*4 + quarter) for rollover safety.
  Coverage check via pg_inherits before CREATE so a quarterly target
  inside an existing wider partition (some 2027 ones are half-year)
  is treated as :exists rather than colliding.
* Microwaveprop.Workers.PartitionMaintenanceWorker — daily cron at
  03:15 UTC, 4-quarter lookahead. Idempotent; cheap when nothing's
  missing. lookback_quarters arg lets ops widen for catch-up.

Tests cover the create + idempotent re-run + name format + year
rollover paths via a synthetic parent created inside the sandbox
transaction (no DDL leaks into the shared test DB).
2026-05-05 16:22:16 -05:00
f668b457cb
feat(pskr): enqueue hrrr_fetch_tasks for cells missing HRRR data
The CalibrationSampler joined PSKR midpoints against hrrr_profiles via
nearest-within-window, but post-Phase-3 Stream C nothing bulk-populates
the grid — hrrr_profiles only sees per-QSO point fetches from the Rust
hrrr-point-worker. Result: 0 / 16,042 prod samples had HRRR data.

Producer pattern: when a cell has no profile in the lookahead window,
enqueue a row into hrrr_fetch_tasks at the cell's HRRR-grid-rounded
midpoint. The Rust worker drains it; the next 5-min rolling-window
pass UPSERTs the same sample with non-NULL weather fields. Idempotent
with the existing on-conflict design — no extra worker, no churn on
cells already covered, multiple bands sharing a midpoint dedupe to a
single fetch via Enum.uniq + jsonb point-union in HrrrPointEnqueuer.
2026-05-05 16:14:43 -05:00
56e9755a10
feat(contacts): per-month bar chart on the /contacts page
Aggregates contact counts by calendar month summed across every
year, rendered as an inline SVG bar chart above the table. Months
with no contacts render as a 0-height tick so the x-axis stays
continuous at 12 bars.

Respects the same scope visibility the table uses, so private
contacts a viewer can't see don't leak into the chart counts.
Each <rect> carries data-month / data-month-count for testability
and a <title> for hover.
2026-05-05 09:35:08 -05:00
1a3dc4fa0a
fix(pskr): drop spots with <6-char grids on either end
A 4-char locator covers ~70 km × 100 km, which dwarfs HRRR's 3 km
native cell and any midpoint we'd derive for the calibration corpus.
Storing those samples gives the recalibrator geometry that's
indistinguishable from noise.

fetch_grid/2 now returns {:error, {:short_grid, key, grid}} for any
locator under 6 chars after Maidenhead validation. The spot is
dropped before it reaches the aggregator, so neither pskr_spots_hourly
nor pskr_calibration_samples ever see it.
2026-05-05 09:29:25 -05:00
b08fc5e449
feat(pskr): rolling-window calibration sampling, every 5 min
Old design fired hourly at HH:25 and built one whole hour of cells
in a single batch. During PSKR peaks (50k+ spots/hour) that batch
held the DB pool long enough to look like a backlog, and a pod
restart between fires lost a full hour because the next cron only
caught the most-recent-prev hour.

- Default window is now current hour + 1 prior. Every 5-min fire
  spreads work across the in-flight hour and always covers the
  boundary without waiting for the next cron tick.
- lookback_hours arg widens the window for catch-up
  (e.g. {"lookback_hours": 24} sweeps the last day after an
  outage).
- Unique period 3600 → 240 so consecutive 5-min cron fires actually
  slot in.

UPSERT is already idempotent so the ~12 redundant passes per hour
cost only the read+merge. Latency from spot ingestion to sample
creation drops from up-to-60min to ≤5min.
2026-05-04 16:46:59 -05:00
a68027b00b
feat(scoring): per-band mechanism stack via cell_score/2
Replaces the "tropo composite + post-hoc aurora boost" model with a
max-over-mechanisms dispatcher so each band's score reflects the
dominant propagation channel rather than summing alternative paths.

- BandConfig.mechanisms/1 derives the stack from frequency:
  ≤432 MHz gets [:tropo, :aurora]; >432 MHz stays [:tropo]. Bands
  may override with an explicit :mechanisms field.
- Scorer.mechanism_score/3 returns a standalone integer score for
  one mechanism (:tropo delegates to composite_score; :aurora is a
  clean Kp/freq function, not a boost on top of tropo).
- Scorer.cell_score/2 takes max across the band's mechanisms so a
  Kp-6 storm scores 6m at 85 even when the tropo picture is poor,
  instead of conflating the two channels.

Es and F2 stay in PathCompute since they're path-distance-dependent.
The legacy aurora_boost path is unchanged so no callers shift today.
2026-05-04 16:46:48 -05:00
48000b0dca
feat(pskr): weekly recalibration analysis on the calibration corpus
`Pskr.Recalibrator.run/0` reads `pskr_calibration_samples`, bins
each sample per (band × feature), and writes spot-density stats to
`pskr_feature_bins` so an operator can read whether a feature
actually discriminates propagation at the threshold granularity
the scorer uses.

Bins, not regression: `BandConfig` already encodes scoring as
discrete thresholds, so the bin output matches that shape and an
operator can copy adjusted thresholds directly without translating
from regression coefficients.

Self-healing: corpus too thin ⇒ run row written with status
`skipped_insufficient_data` and the analysis is a no-op until next
fire. `min_total_samples = 1000` (≈ 4-5 days of CONUS PSKR
activity); per-band threshold is 100. Both surface in the run row's
`notes`.

Auto-applies nothing. Weight changes still go through human review
of `BandConfig.@band_configs` and a code commit. The recalibrator
is a read-only analyst that stays out of the production scoring
path.

Features binned (matching the scorer's discriminating fields):
  * pwat_mm — humidity U-shape candidate
  * hpbl_m — boundary layer (mechanism vs scoring re-eval)
  * min_refractivity_gradient — refractivity threshold validation
  * surface_pressure_mb — pressure-front proxy
  * kp_index — aurora boost magnitude tuning

Schema: two tables.
  * `pskr_recalibration_runs` — one row per fire with corpus
    stats, status, notes
  * `pskr_feature_bins` — one row per (run, band, feature, bin)
    with sample_count, spot_count_total/avg/p50/p90

Cron: `0 4 * * 0` (Sundays 04:00 UTC, off-peak, post-climatology).
Manual reruns enqueue with no args.

Tests cover the empty-corpus skip path, sub-threshold totals,
per-band threshold gating, the actual bin emission, nil-feature
handling, spot-count averaging, and the always-records-a-run
audit invariant. 8 new tests, 3282 total passing.

Backfill pipeline untouched.
2026-05-04 16:18:17 -05:00
7a2b1f292c
feat(pskr): hourly calibration sampler joining spots × HRRR × Kp
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:

  * `pskr_spots_hourly` — observed spot density (truth signal)
  * `hrrr_profiles` nearest match at the cell midpoint at hour
    boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
    ducting flag)
  * `geomagnetic_observations` latest Kp at the hour (space weather)

Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.

Components:

  * Migration `20260504210756_create_pskr_calibration_samples` —
    new table + unique index on (hour, band, midpoint_lat, lon),
    plus a midpoint spatial index on `pskr_spots_hourly` to keep
    the join cheap.
  * `Pskr.CalibrationSample` — schema mirror of the table with
    the same `(hour, band, midpoint_lat, lon)` unique constraint.
  * `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
    for the hour, snaps midpoints to the propagation grid (0.125°),
    builds an in-memory HRRR index over a ±0.07°/±60 min window,
    and bulk-upserts samples. Idempotent — re-runs upsert.
  * `Workers.PskrCalibrationWorker` — Oban cron entry on the
    `:backfill_enqueue` queue. Default args target the previous
    full hour; explicit `hour_utc` arg reruns any hour.
  * Cron `25 * * * *` — fires past HRRR analysis publish window
    (~HH:50→HH:05) and PSKR aggregator's 60s flush.

Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.

Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).

Backfill pipeline untouched — none of these changes feed into
contact enrichment.
2026-05-04 16:12:47 -05:00
7702b9e161
feat(scoring): aurora boost on VHF/low-UHF during geomagnetic storms
`Scorer.aurora_boost(score, kp, band_config)` is a terminal boost
applied after `composite_score` in the same shape as the existing
`commercial_link_boost`. It tracks the NOAA G-scale (Kp 5+ opens
auroral E-region paths up to ~2500 km) and is freq-gated to
≤ 432 MHz — auroral propagation is observed almost exclusively on
50/144/222, occasionally on 432, and never at microwave.

Boost magnitudes:
* Kp < 4 (quiet) — pass-through
* Kp 4 — +5
* Kp 5 (G1) — +15
* Kp 6 (G2) — +25
* Kp ≥ 7 (G3+) — +35, clamped to 100

Wired into `Propagation.score_grid_point/4` via the existing
`hrrr_profile` map: callers populate `:kp_index` once per cycle so
the per-grid-point band loop never queries the DB. The UI fallback
path (`factors_from_profile/5`) sources Kp from
`SpaceWeather.latest_kp/0` directly. Other call sites
(GEFS forecast, path-compute, contact-show) can opt in by adding
`:kp_index` to their profile map.

Diagnostic factor `:kp_index` is stashed on VHF/low-UHF results so
the operator-facing factor breakdown can show why a band scored
above its quiet-conditions baseline. Microwave bands don't carry
the diagnostic — they're never affected by the boost.

Refactored `score_with_algorithm/7` to delegate band-level
finalization to `finalize_band_result/5`, keeping the parent
function under credo's cyclomatic-complexity limit.

NOTE: this only affects the Elixir scoring path — the live grid
served by `prop_grid_rs` (Rust) does its own scoring and does not
yet honor Kp. Adding aurora to the Rust side is follow-up work.
2026-05-04 16:02:49 -05:00