Commit graph

6 commits

Author SHA1 Message Date
084e6b6224
refactor(db): use explicit on_conflict set clauses over :replace_all_except
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.

Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.

Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)

Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
2026-04-21 14:22:15 -05:00
1400c38f44
fix(dialyzer): actually fix LiveTable warnings instead of hiding them
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.

Changes:

- Delete .dialyzer_ignore.exs and remove ignore_warnings from
  mix.exs. No more hidden suppressions.

- New MicrowavepropWeb.LiveTableResource shim that wraps
  use LiveTable.LiveResource and overrides:
  * maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
    (previously discarded, tripping :unmatched_return).
  * handle_info/2 — explicit _ = on File.rm, Process.send_after.
  Switch all 4 LiveTable-using LiveViews (contact_live/index,
  beacon_live/index, admin/contact_edit_live, user_management_live/index)
  to use MicrowavepropWeb.LiveTableResource instead.

- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
  LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
  the dep's macro-expanded handle_params/3. The call's return was
  silently discarded, tripping the fourth warning per LiveView.

- New lib/mix/tasks/deps.patch.ex applies every
  priv/dep_patches/*.patch to its target dep idempotently after
  mix deps.get. Wired into the :setup alias + overridden
  mix deps.get so the patch survives cache regeneration in CI.

- Other modified files in this commit are the spec-coverage
  additions from the parallel agent pass (beacons.ex,
  commercial.ex, propagation.ex, weather.ex, narr_client.ex,
  run_timing.ex, 3 workers) — tighter specs for bounds, tuple
  shapes, schema-typed params. Dialyzer remains at 0 after
  the changes.

Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.

mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
2026-04-21 12:58:10 -05:00
8a969e315c
refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).

- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
  'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
  scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
  (was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
  with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
2026-04-17 09:10:32 -05:00
4487fc22e9 Fix NexradWorker UUID type: use generate() not bingenerate() for insert_all 2026-04-10 14:13:38 -05:00
0edb1bd95a Fix NexradWorker crash when minute key is missing from args
Jobs enqueued via backfill only have year/month/day/hour — default
minute to 0 instead of requiring it in the pattern match.
2026-04-10 14:07:55 -05:00
e7a7ae073d Phase 9.3, 9.4, and Phase 3 NEXRAD pipeline
Task 9.3 - Weight recalibration via gradient descent:
- Recalibrator module fits logistic regression weights using Nx
- Trains on QSO positives vs random baseline negatives
- Cross-validates by month, normalizes weights to sum to 1.0
- Mix task: mix recalibrate_scorer --sample 5000 --epochs 2000

Task 9.4 - Side-by-side scorer comparison:
- ScorerDiff.compare/3 re-scores grid with old vs new weights
- Reports mean diff, regressions, improvements, per-band breakdown
- Mix task: mix scorer_diff --new-weights '{...}'

Phase 3 - NEXRAD ingestion pipeline:
- NexradClient fetches IEM n0q composite PNGs, extracts per-point
  box statistics (mean/max dBZ, texture variance)
- NexradObservation schema with unique (lat, lon, observed_at)
- NexradWorker on :nexrad queue for background processing
- nexrad_texture backtest feature in Features module
- mix nexrad_backfill --limit 200

All tasks added to AdminTaskWorker and Release for production use.
1116 tests, 0 failures.
2026-04-10 12:48:36 -05:00