Commit graph

5 commits

Author SHA1 Message Date
63a9417287
fix(admin_task_worker): unknown feature now returns error instead of falling through
Backtest branch's unknown-feature guard was dead code: the if block's
{:error, _} return was discarded and execution fell through, running the
backtest with a just-created atom and writing a report for nonsense
features. Two compounding issues:

- function_exported?/3 returns false for modules not yet loaded in the
  BEAM, so even valid feature names triggered the warning path on a
  cold VM.
- String.to_atom/1 on caller-supplied strings created unbounded atoms.

Fix: Code.ensure_loaded?(Features) before the check, String.to_existing_atom/1
with rescue (Features exports __info__(:functions) so every valid function
atom already exists), and an explicit early return via a with pipeline so
the error tuple actually leaves the worker.
2026-04-16 14:58:55 -05:00
811251dd15
Drop propagation_scores table, ScoresFile is the only store
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:

1. New migration drops the propagation_scores table and its indexes
   (the earlier tuning migrations for it were already applied and
   are now no-ops against a missing table, which is fine — Ecto
   just runs them on fresh environments).

2. Propagation context is gutted of every GridScore reference.
   replace_scores/2 writes files only. upsert_scores/2 is deleted.
   load_scores_from_db, available_valid_times_from_db,
   point_detail_from_db, point_forecast_from_db, fetch_factors,
   coalesce_factors, the Postgres side of prune_old_scores, and
   the Postgres fallbacks in latest/earliest_valid_time are all
   removed. point_detail always returns an empty factors map now
   since factor breakdowns were retired with the table.

3. Deleted modules:
   - Propagation.GridScore (the schema)
   - Propagation.ScorerDiff (read factors from the table)
   - Propagation.AsosNudge (helper for AsosAdjustmentWorker)
   - Workers.AsosAdjustmentWorker (its cron was already disabled)
   - Mix.Tasks.ScorerDiff (wrapper around the deleted module)
   And their tests. AdminTaskWorker's scorer_diff task is a
   logging no-op so any queued Oban rows drain cleanly.
   Release.scorer_diff stays as a stub that tells the operator.

propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
2026-04-14 15:13:01 -05:00
7fb340bc35 Fix all remaining credo --strict issues (0 issues)
Aliases: add module aliases for 9 nested module references
Apply: replace apply/3 with direct module attribute calls
Line length: break 1 long spec line
Refactoring: extract helpers to reduce complexity and nesting
in show.ex, radio.ex, weather workers, terrain, duct detection,
backfill dashboard, contact map, and mix tasks
2026-04-12 10:26:53 -05:00
e7a7ae073d Phase 9.3, 9.4, and Phase 3 NEXRAD pipeline
Task 9.3 - Weight recalibration via gradient descent:
- Recalibrator module fits logistic regression weights using Nx
- Trains on QSO positives vs random baseline negatives
- Cross-validates by month, normalizes weights to sum to 1.0
- Mix task: mix recalibrate_scorer --sample 5000 --epochs 2000

Task 9.4 - Side-by-side scorer comparison:
- ScorerDiff.compare/3 re-scores grid with old vs new weights
- Reports mean diff, regressions, improvements, per-band breakdown
- Mix task: mix scorer_diff --new-weights '{...}'

Phase 3 - NEXRAD ingestion pipeline:
- NexradClient fetches IEM n0q composite PNGs, extracts per-point
  box statistics (mean/max dBZ, texture variance)
- NexradObservation schema with unique (lat, lon, observed_at)
- NexradWorker on :nexrad queue for background processing
- nexrad_texture backtest feature in Features module
- mix nexrad_backfill --limit 200

All tasks added to AdminTaskWorker and Release for production use.
1116 tests, 0 failures.
2026-04-10 12:48:36 -05:00
01909dbe66 Run admin tasks as Oban jobs instead of blocking eval
Release.backtest_all, climatology, native_derive now enqueue an
AdminTaskWorker job on the new :admin queue and return immediately.
Progress visible in Oban Web at /admin/oban.
2026-04-10 12:26:01 -05:00